Oct 30 13:48:08.566302 ip-10-0-140-105 systemd[1]: Starting Kubernetes Kubelet... Oct 30 13:48:08.902349 ip-10-0-140-105 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 13:48:08.902349 ip-10-0-140-105 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 13:48:08.902349 ip-10-0-140-105 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 13:48:08.902349 ip-10-0-140-105 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 13:48:08.909718 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909625 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 13:48:08.909718 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909715 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 13:48:08.909718 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909719 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 13:48:08.909718 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909722 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 13:48:08.909718 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909725 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 13:48:08.909910 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909728 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 13:48:08.909910 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909731 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 13:48:08.909910 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909734 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 13:48:08.909910 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909736 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 13:48:08.909910 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909739 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 13:48:08.909910 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909742 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 13:48:08.909910 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909744 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 13:48:08.909910 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909747 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 13:48:08.909910 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909749 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 13:48:08.909910 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909752 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 13:48:08.909910 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909754 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 13:48:08.909910 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909757 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 13:48:08.909910 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909760 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 13:48:08.909910 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909762 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 13:48:08.909910 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909765 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 13:48:08.909910 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909767 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 13:48:08.909910 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909770 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 13:48:08.909910 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909772 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 13:48:08.909910 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909775 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 13:48:08.909910 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909777 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 13:48:08.910379 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909780 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 13:48:08.910379 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909782 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 13:48:08.910379 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909785 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 13:48:08.910379 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909788 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 13:48:08.910379 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909790 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 13:48:08.910379 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909793 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 30 13:48:08.910379 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909795 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 13:48:08.910379 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909802 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 13:48:08.910379 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909806 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 13:48:08.910379 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909808 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 13:48:08.910379 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909811 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 13:48:08.910379 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909813 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 13:48:08.910379 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909816 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 13:48:08.910379 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909819 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 13:48:08.910379 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909822 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 13:48:08.910379 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909825 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 13:48:08.910379 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909827 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 13:48:08.910379 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909830 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 13:48:08.910379 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909832 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 13:48:08.910825 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909835 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 13:48:08.910825 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909837 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 13:48:08.910825 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909840 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 13:48:08.910825 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909842 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 13:48:08.910825 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909845 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 13:48:08.910825 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909848 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 13:48:08.910825 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909851 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 13:48:08.910825 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909854 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 13:48:08.910825 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909856 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 13:48:08.910825 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909859 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 13:48:08.910825 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909862 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 13:48:08.910825 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909864 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 13:48:08.910825 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909868 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 13:48:08.910825 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909871 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 13:48:08.910825 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909874 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 13:48:08.910825 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909876 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 13:48:08.910825 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909879 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 13:48:08.910825 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909881 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 13:48:08.910825 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909884 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 13:48:08.911267 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909887 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 13:48:08.911267 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909890 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 13:48:08.911267 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909893 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 13:48:08.911267 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909895 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 13:48:08.911267 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909899 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 13:48:08.911267 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909902 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 13:48:08.911267 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909904 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 13:48:08.911267 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909907 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 13:48:08.911267 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909910 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 13:48:08.911267 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909913 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 13:48:08.911267 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909915 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 13:48:08.911267 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909919 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 13:48:08.911267 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909921 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 13:48:08.911267 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909924 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 13:48:08.911267 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909926 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 13:48:08.911267 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909929 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 13:48:08.911267 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909933 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 13:48:08.911267 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909937 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 13:48:08.911267 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909940 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 13:48:08.911701 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909942 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 13:48:08.911701 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909945 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 13:48:08.911701 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909948 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 13:48:08.911701 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909950 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 13:48:08.911701 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909953 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 13:48:08.911701 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909955 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 13:48:08.911701 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909958 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 13:48:08.911701 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909961 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 13:48:08.911701 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909963 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 13:48:08.911701 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909966 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 13:48:08.911701 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909968 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 13:48:08.911701 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909971 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 13:48:08.911701 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909973 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 13:48:08.911701 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909976 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 13:48:08.911701 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909978 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 13:48:08.911701 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909980 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 13:48:08.911701 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909983 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 13:48:08.911701 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909986 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 13:48:08.911701 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909989 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 13:48:08.911701 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909991 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 13:48:08.912168 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909994 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 13:48:08.912168 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.909997 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 13:48:08.912168 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.910002 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 13:48:08.912168 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.910006 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 13:48:08.912168 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.910847 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 13:48:08.912168 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.910856 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 13:48:08.912168 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.910860 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 13:48:08.912168 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.910864 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 13:48:08.912168 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.910867 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 13:48:08.912168 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.910870 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 13:48:08.912168 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.910873 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 13:48:08.912168 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.910875 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 13:48:08.912168 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.910878 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 13:48:08.912168 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.910881 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 13:48:08.912168 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.910884 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 13:48:08.912168 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.910887 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 13:48:08.912168 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.910890 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 13:48:08.912168 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.910892 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 13:48:08.912168 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.910895 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 13:48:08.912613 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.910897 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 13:48:08.912613 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.910900 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 13:48:08.912613 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.910902 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 13:48:08.912613 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.910905 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 13:48:08.912613 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.910907 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 13:48:08.912613 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.910910 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 13:48:08.912613 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.910912 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 13:48:08.912613 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.910915 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 13:48:08.912613 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.910918 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 13:48:08.912613 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.910921 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 13:48:08.912613 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.910923 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 13:48:08.912613 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.910926 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 13:48:08.912613 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.910928 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 13:48:08.912613 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.910931 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 13:48:08.912613 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.910934 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 13:48:08.912613 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.910936 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 13:48:08.912613 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.910939 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 13:48:08.912613 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.910942 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 13:48:08.912613 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.910944 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 13:48:08.912613 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.910946 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 13:48:08.913083 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.910949 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 13:48:08.913083 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.910952 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 13:48:08.913083 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.910954 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 13:48:08.913083 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.910957 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 13:48:08.913083 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.910959 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 13:48:08.913083 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.910961 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 13:48:08.913083 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.910965 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 13:48:08.913083 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.910969 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 13:48:08.913083 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.910973 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 13:48:08.913083 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.910975 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 13:48:08.913083 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.910978 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 13:48:08.913083 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.910981 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 13:48:08.913083 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.910984 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 13:48:08.913083 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.910987 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 13:48:08.913083 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.910990 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 13:48:08.913083 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.910992 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 13:48:08.913083 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.910995 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 13:48:08.913083 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.910998 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 13:48:08.913083 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911000 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 13:48:08.913512 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911002 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 13:48:08.913512 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911005 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 13:48:08.913512 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911008 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 13:48:08.913512 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911011 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 13:48:08.913512 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911013 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 13:48:08.913512 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911015 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 13:48:08.913512 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911018 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 13:48:08.913512 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911021 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 13:48:08.913512 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911023 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 13:48:08.913512 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911026 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 13:48:08.913512 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911029 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 30 13:48:08.913512 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911032 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 13:48:08.913512 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911034 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 13:48:08.913512 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911036 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 13:48:08.913512 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911039 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 13:48:08.913512 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911042 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 13:48:08.913512 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911044 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 13:48:08.913512 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911046 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 13:48:08.913512 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911050 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 13:48:08.913939 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911052 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 13:48:08.913939 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911055 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 13:48:08.913939 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911057 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 13:48:08.913939 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911072 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 13:48:08.913939 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911075 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 13:48:08.913939 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911078 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 13:48:08.913939 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911081 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 13:48:08.913939 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911083 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 13:48:08.913939 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911088 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 13:48:08.913939 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911092 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 13:48:08.913939 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911095 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 13:48:08.913939 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911098 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 13:48:08.913939 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911101 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 13:48:08.913939 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911104 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 13:48:08.913939 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911106 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 13:48:08.913939 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911109 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 13:48:08.913939 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911112 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 13:48:08.913939 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911114 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 13:48:08.913939 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911117 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 13:48:08.914388 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911120 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 13:48:08.914388 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911123 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 13:48:08.914388 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911126 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 13:48:08.914388 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911128 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 13:48:08.914388 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911131 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 13:48:08.914388 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911133 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 13:48:08.914388 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911136 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 13:48:08.914388 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911138 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 13:48:08.914388 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911141 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 13:48:08.914388 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911144 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 13:48:08.914388 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911146 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 13:48:08.914388 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911149 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 13:48:08.914388 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911151 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 13:48:08.914388 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911154 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 13:48:08.914388 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911641 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 13:48:08.914388 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911646 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 13:48:08.914388 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911649 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 13:48:08.914388 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911652 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 13:48:08.914388 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911655 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 13:48:08.914388 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911658 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 13:48:08.914865 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911660 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 13:48:08.914865 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911662 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 13:48:08.914865 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911665 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 13:48:08.914865 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911668 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 13:48:08.914865 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911670 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 13:48:08.914865 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911673 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 13:48:08.914865 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911675 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 13:48:08.914865 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911678 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 13:48:08.914865 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911680 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 13:48:08.914865 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911683 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 13:48:08.914865 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911686 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 13:48:08.914865 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911688 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 13:48:08.914865 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911691 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 13:48:08.914865 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911694 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 13:48:08.914865 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911697 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 13:48:08.914865 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911699 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 13:48:08.914865 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911702 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 13:48:08.914865 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911705 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 13:48:08.914865 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911708 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 13:48:08.914865 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911710 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 13:48:08.915342 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911713 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 13:48:08.915342 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911715 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 13:48:08.915342 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911717 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 13:48:08.915342 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911720 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 13:48:08.915342 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911722 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 13:48:08.915342 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911725 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 13:48:08.915342 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911727 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 13:48:08.915342 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911730 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 13:48:08.915342 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911732 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 13:48:08.915342 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911735 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 13:48:08.915342 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911737 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 13:48:08.915342 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911740 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 13:48:08.915342 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911742 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 13:48:08.915342 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911744 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 13:48:08.915342 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911747 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 13:48:08.915342 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911749 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 13:48:08.915342 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911752 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 13:48:08.915342 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911754 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 13:48:08.915342 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911757 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 13:48:08.915766 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911761 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 13:48:08.915766 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911765 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 13:48:08.915766 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911769 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 13:48:08.915766 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911771 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 13:48:08.915766 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911774 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 13:48:08.915766 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911777 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 13:48:08.915766 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911780 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 13:48:08.915766 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911782 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 13:48:08.915766 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911784 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 13:48:08.915766 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911787 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 13:48:08.915766 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911789 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 13:48:08.915766 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911792 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 13:48:08.915766 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911794 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 13:48:08.915766 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911797 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 13:48:08.915766 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911799 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 13:48:08.915766 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911801 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 13:48:08.915766 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911804 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 13:48:08.915766 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911806 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 13:48:08.915766 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911808 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 13:48:08.916248 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911811 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 13:48:08.916248 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911813 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 13:48:08.916248 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911816 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 13:48:08.916248 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911818 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 13:48:08.916248 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911821 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 13:48:08.916248 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911824 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 13:48:08.916248 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911826 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 13:48:08.916248 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911829 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 13:48:08.916248 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911831 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 13:48:08.916248 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911833 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 13:48:08.916248 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911836 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 13:48:08.916248 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911838 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 13:48:08.916248 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911841 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 13:48:08.916248 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911843 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 13:48:08.916248 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911846 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 13:48:08.916248 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911849 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 13:48:08.916248 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911851 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 13:48:08.916248 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911854 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 13:48:08.916248 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911857 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 13:48:08.916248 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911859 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 13:48:08.916694 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911862 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 13:48:08.916694 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911864 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 13:48:08.916694 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911867 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 13:48:08.916694 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911870 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 13:48:08.916694 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911872 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 13:48:08.916694 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911875 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 13:48:08.916694 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911877 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 30 13:48:08.916694 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911880 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 13:48:08.916694 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911882 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 13:48:08.916694 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911884 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 13:48:08.916694 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911887 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 13:48:08.916694 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911889 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 13:48:08.916694 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911893 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 13:48:08.916694 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911897 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 13:48:08.916694 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911899 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 13:48:08.916694 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911902 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 13:48:08.916694 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911905 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 13:48:08.916694 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911908 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 13:48:08.916694 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911911 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 13:48:08.917151 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911913 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 13:48:08.917151 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911916 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 13:48:08.917151 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.911918 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 13:48:08.917151 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913410 2580 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 13:48:08.917151 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913421 2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 13:48:08.917151 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913427 2580 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 13:48:08.917151 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913433 2580 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 13:48:08.917151 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913438 2580 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 13:48:08.917151 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913442 2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 13:48:08.917151 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913451 2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 13:48:08.917151 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913457 2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 13:48:08.917151 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913462 2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 13:48:08.917151 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913465 2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 13:48:08.917151 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913468 2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 13:48:08.917151 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913471 2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 13:48:08.917151 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913474 2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 13:48:08.917151 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913477 2580 flags.go:64] FLAG: --cgroup-root="" Oct 30 13:48:08.917151 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913480 2580 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 13:48:08.917151 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913483 2580 flags.go:64] FLAG: --client-ca-file="" Oct 30 13:48:08.917151 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913486 2580 flags.go:64] FLAG: --cloud-provider="external" Oct 30 13:48:08.917151 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913489 2580 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 13:48:08.917151 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913494 2580 flags.go:64] FLAG: --cluster-domain="" Oct 30 13:48:08.917151 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913497 2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 13:48:08.917683 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913500 2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 13:48:08.917683 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913503 2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 13:48:08.917683 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913506 2580 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 13:48:08.917683 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913515 2580 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 13:48:08.917683 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913518 2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 13:48:08.917683 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913521 2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 13:48:08.917683 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913524 2580 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 13:48:08.917683 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913528 2580 flags.go:64] FLAG: --contention-profiling="false" Oct 30 13:48:08.917683 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913531 2580 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 13:48:08.917683 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913534 2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 13:48:08.917683 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913537 2580 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 13:48:08.917683 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913540 2580 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 13:48:08.917683 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913545 2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 13:48:08.917683 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913547 2580 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 13:48:08.917683 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913550 2580 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 13:48:08.917683 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913553 2580 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 13:48:08.917683 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913556 2580 flags.go:64] FLAG: --enable-server="true" Oct 30 13:48:08.917683 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913559 2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 13:48:08.917683 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913563 2580 flags.go:64] FLAG: --event-burst="100" Oct 30 13:48:08.917683 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913567 2580 flags.go:64] FLAG: --event-qps="50" Oct 30 13:48:08.917683 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913570 2580 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 13:48:08.917683 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913573 2580 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 13:48:08.917683 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913576 2580 flags.go:64] FLAG: --eviction-hard="" Oct 30 13:48:08.917683 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913580 2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 13:48:08.918289 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913583 2580 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 13:48:08.918289 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913586 2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 13:48:08.918289 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913589 2580 flags.go:64] FLAG: --eviction-soft="" Oct 30 13:48:08.918289 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913592 2580 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 13:48:08.918289 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913595 2580 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 13:48:08.918289 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913598 2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 13:48:08.918289 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913601 2580 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 13:48:08.918289 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913604 2580 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 13:48:08.918289 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913606 2580 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 13:48:08.918289 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913609 2580 flags.go:64] FLAG: --feature-gates="" Oct 30 13:48:08.918289 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913613 2580 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 13:48:08.918289 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913616 2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 13:48:08.918289 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913619 2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 13:48:08.918289 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913622 2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 13:48:08.918289 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913625 2580 flags.go:64] FLAG: --healthz-port="10248" Oct 30 13:48:08.918289 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913628 2580 flags.go:64] FLAG: --help="false" Oct 30 13:48:08.918289 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913631 2580 flags.go:64] FLAG: --hostname-override="ip-10-0-140-105.ec2.internal" Oct 30 13:48:08.918289 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913635 2580 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 13:48:08.918289 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913638 2580 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 13:48:08.918289 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913641 2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 13:48:08.918289 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913644 2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 13:48:08.918289 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913647 2580 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 13:48:08.918289 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913650 2580 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 13:48:08.918289 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913653 2580 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 13:48:08.918847 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913656 2580 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 13:48:08.918847 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913659 2580 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 13:48:08.918847 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913662 2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 13:48:08.918847 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913665 2580 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 13:48:08.918847 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913668 2580 flags.go:64] FLAG: --kube-reserved="" Oct 30 13:48:08.918847 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913671 2580 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 13:48:08.918847 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913673 2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 13:48:08.918847 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913677 2580 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 13:48:08.918847 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913679 2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 13:48:08.918847 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913682 2580 flags.go:64] FLAG: --lock-file="" Oct 30 13:48:08.918847 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913685 2580 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 13:48:08.918847 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913688 2580 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 13:48:08.918847 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913690 2580 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 13:48:08.918847 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913696 2580 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 13:48:08.918847 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913699 2580 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 13:48:08.918847 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913702 2580 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 13:48:08.918847 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913705 2580 flags.go:64] FLAG: --logging-format="text" Oct 30 13:48:08.918847 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913707 2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 13:48:08.918847 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913711 2580 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 13:48:08.918847 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913713 2580 flags.go:64] FLAG: --manifest-url="" Oct 30 13:48:08.918847 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913716 2580 flags.go:64] FLAG: --manifest-url-header="" Oct 30 13:48:08.918847 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913721 2580 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 13:48:08.918847 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913724 2580 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 13:48:08.918847 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913727 2580 flags.go:64] FLAG: --max-pods="110" Oct 30 13:48:08.918847 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913730 2580 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 13:48:08.919433 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913733 2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 13:48:08.919433 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913736 2580 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 13:48:08.919433 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913739 2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 13:48:08.919433 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913742 2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 13:48:08.919433 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913745 2580 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 13:48:08.919433 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913748 2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 13:48:08.919433 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913755 2580 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 13:48:08.919433 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913758 2580 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 13:48:08.919433 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913761 2580 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 13:48:08.919433 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913764 2580 flags.go:64] FLAG: --pod-cidr="" Oct 30 13:48:08.919433 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913767 2580 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 13:48:08.919433 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913769 2580 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 13:48:08.919433 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913772 2580 flags.go:64] FLAG: --pods-per-core="0" Oct 30 13:48:08.919433 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913775 2580 flags.go:64] FLAG: --port="10250" Oct 30 13:48:08.919433 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913778 2580 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 13:48:08.919433 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913781 2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c265bd34ba759816" Oct 30 13:48:08.919433 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913784 2580 flags.go:64] FLAG: --qos-reserved="" Oct 30 13:48:08.919433 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913787 2580 flags.go:64] FLAG: --read-only-port="10255" Oct 30 13:48:08.919433 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913790 2580 flags.go:64] FLAG: --register-node="true" Oct 30 13:48:08.919433 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913793 2580 flags.go:64] FLAG: --register-with-taints="" Oct 30 13:48:08.919433 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913797 2580 flags.go:64] FLAG: --registry-burst="10" Oct 30 13:48:08.919433 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913799 2580 flags.go:64] FLAG: --registry-qps="5" Oct 30 13:48:08.919433 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913802 2580 flags.go:64] FLAG: --reserved-cpus="" Oct 30 13:48:08.919433 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913805 2580 flags.go:64] FLAG: --reserved-memory="" Oct 30 13:48:08.919433 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913809 2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 13:48:08.919433 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913811 2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 13:48:08.920118 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913814 2580 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 13:48:08.920118 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913817 2580 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 13:48:08.920118 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913820 2580 flags.go:64] FLAG: --runonce="false" Oct 30 13:48:08.920118 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913823 2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 13:48:08.920118 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913826 2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 13:48:08.920118 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913828 2580 flags.go:64] FLAG: --seccomp-default="false" Oct 30 13:48:08.920118 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913831 2580 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 13:48:08.920118 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913834 2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 13:48:08.920118 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913840 2580 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 13:48:08.920118 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913845 2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 13:48:08.920118 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913848 2580 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 13:48:08.920118 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913851 2580 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 13:48:08.920118 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913854 2580 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 13:48:08.920118 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913857 2580 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 13:48:08.920118 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913860 2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 13:48:08.920118 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913863 2580 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 13:48:08.920118 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913866 2580 flags.go:64] FLAG: --system-cgroups="" Oct 30 13:48:08.920118 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913869 2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 13:48:08.920118 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913874 2580 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 13:48:08.920118 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913877 2580 flags.go:64] FLAG: --tls-cert-file="" Oct 30 13:48:08.920118 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913880 2580 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 13:48:08.920118 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913884 2580 flags.go:64] FLAG: --tls-min-version="" Oct 30 13:48:08.920118 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913886 2580 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 13:48:08.920118 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913889 2580 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 13:48:08.920118 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913892 2580 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 13:48:08.920725 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913895 2580 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 13:48:08.920725 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913898 2580 flags.go:64] FLAG: --v="2" Oct 30 13:48:08.920725 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913902 2580 flags.go:64] FLAG: --version="false" Oct 30 13:48:08.920725 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913906 2580 flags.go:64] FLAG: --vmodule="" Oct 30 13:48:08.920725 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913910 2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 13:48:08.920725 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.913913 2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 13:48:08.920725 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914045 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 13:48:08.920725 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914048 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 13:48:08.920725 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914051 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 13:48:08.920725 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914054 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 13:48:08.920725 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914056 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 13:48:08.920725 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914070 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 13:48:08.920725 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914073 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 13:48:08.920725 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914076 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 13:48:08.920725 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914078 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 13:48:08.920725 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914081 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 13:48:08.920725 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914085 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 13:48:08.920725 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914088 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 13:48:08.920725 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914090 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 13:48:08.920725 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914093 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 13:48:08.920725 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914096 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 13:48:08.921232 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914099 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 13:48:08.921232 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914101 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 13:48:08.921232 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914106 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 13:48:08.921232 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914109 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 13:48:08.921232 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914111 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 13:48:08.921232 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914114 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 13:48:08.921232 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914116 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 13:48:08.921232 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914119 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 13:48:08.921232 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914121 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 13:48:08.921232 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914124 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 13:48:08.921232 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914126 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 13:48:08.921232 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914129 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 13:48:08.921232 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914132 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 13:48:08.921232 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914134 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 30 13:48:08.921232 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914137 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 13:48:08.921232 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914139 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 13:48:08.921232 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914142 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 13:48:08.921232 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914144 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 13:48:08.921232 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914146 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 13:48:08.921232 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914149 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 13:48:08.921684 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914152 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 13:48:08.921684 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914155 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 13:48:08.921684 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914157 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 13:48:08.921684 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914160 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 13:48:08.921684 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914163 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 13:48:08.921684 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914165 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 13:48:08.921684 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914168 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 13:48:08.921684 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914171 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 13:48:08.921684 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914174 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 13:48:08.921684 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914177 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 13:48:08.921684 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914180 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 13:48:08.921684 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914182 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 13:48:08.921684 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914185 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 13:48:08.921684 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914188 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 13:48:08.921684 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914191 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 13:48:08.921684 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914194 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 13:48:08.921684 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914196 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 13:48:08.921684 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914199 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 13:48:08.921684 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914201 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 13:48:08.922197 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914204 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 13:48:08.922197 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914206 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 13:48:08.922197 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914209 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 13:48:08.922197 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914211 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 13:48:08.922197 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914214 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 13:48:08.922197 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914216 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 13:48:08.922197 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914219 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 13:48:08.922197 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914222 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 13:48:08.922197 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914224 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 13:48:08.922197 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914227 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 13:48:08.922197 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914229 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 13:48:08.922197 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914232 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 13:48:08.922197 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914235 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 13:48:08.922197 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914237 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 13:48:08.922197 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914240 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 13:48:08.922197 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914242 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 13:48:08.922197 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914246 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 13:48:08.922197 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914250 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 13:48:08.922197 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914254 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 13:48:08.922639 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914257 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 13:48:08.922639 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914261 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 13:48:08.922639 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914264 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 13:48:08.922639 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914270 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 13:48:08.922639 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914273 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 13:48:08.922639 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914276 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 13:48:08.922639 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914278 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 13:48:08.922639 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914281 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 13:48:08.922639 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914285 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 13:48:08.922639 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914287 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 13:48:08.922639 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914290 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 13:48:08.922639 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914292 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 13:48:08.922639 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914294 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 13:48:08.922639 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914297 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 13:48:08.922639 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914299 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 13:48:08.922639 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914302 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 13:48:08.922639 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914304 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 13:48:08.922639 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914307 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 13:48:08.922639 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914309 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 13:48:08.922639 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914312 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 13:48:08.923137 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914314 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 13:48:08.923137 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914317 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 13:48:08.923137 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914320 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 13:48:08.923137 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914322 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 13:48:08.923137 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914326 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 13:48:08.923137 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914329 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 13:48:08.923137 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914332 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 13:48:08.923137 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914335 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 13:48:08.923137 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914337 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 13:48:08.923137 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914339 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 13:48:08.923137 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914342 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 13:48:08.923137 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914344 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 13:48:08.923137 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914347 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 13:48:08.923137 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.914375 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 13:48:08.923137 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914410 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 13:48:08.923137 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914414 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 13:48:08.923506 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914418 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 13:48:08.923506 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914421 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 13:48:08.923506 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914423 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 13:48:08.923506 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914426 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 13:48:08.923506 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914430 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 13:48:08.923506 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914433 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 13:48:08.923506 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914435 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 13:48:08.923506 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914438 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 13:48:08.923506 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914440 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 13:48:08.923506 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914443 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 13:48:08.923506 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914445 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 13:48:08.923506 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914448 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 13:48:08.923506 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914451 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 13:48:08.923506 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914454 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 13:48:08.923506 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914456 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 13:48:08.923506 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914458 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 13:48:08.923506 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914462 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 13:48:08.923506 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914466 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 13:48:08.923506 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914470 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 13:48:08.923949 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914473 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 13:48:08.923949 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914476 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 13:48:08.923949 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914479 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 13:48:08.923949 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914481 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 13:48:08.923949 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914484 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 13:48:08.923949 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914486 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 13:48:08.923949 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914489 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 13:48:08.923949 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914491 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 13:48:08.923949 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914494 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 13:48:08.923949 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914496 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 13:48:08.923949 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914499 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 13:48:08.923949 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914503 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 13:48:08.923949 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914506 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 13:48:08.923949 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914509 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 13:48:08.923949 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914512 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 13:48:08.923949 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914515 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 13:48:08.923949 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914517 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 13:48:08.923949 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914521 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 13:48:08.923949 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914523 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 13:48:08.923949 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914526 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 13:48:08.924512 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914529 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 13:48:08.924512 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914531 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 13:48:08.924512 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914534 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 13:48:08.924512 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914536 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 13:48:08.924512 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914541 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 13:48:08.924512 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914545 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 13:48:08.924512 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914547 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 13:48:08.924512 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914550 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 13:48:08.924512 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914553 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 13:48:08.924512 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914555 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 13:48:08.924512 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914557 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 13:48:08.924512 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914560 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 13:48:08.924512 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914563 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 13:48:08.924512 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914565 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 13:48:08.924512 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914568 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 13:48:08.924512 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914570 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 13:48:08.924512 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914573 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 13:48:08.924512 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914575 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 13:48:08.924512 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914578 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 13:48:08.924940 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914580 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 13:48:08.924940 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914583 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 13:48:08.924940 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914585 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 13:48:08.924940 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914588 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 13:48:08.924940 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914592 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 13:48:08.924940 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914594 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 13:48:08.924940 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914597 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 13:48:08.924940 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914599 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 13:48:08.924940 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914608 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 13:48:08.924940 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914611 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 13:48:08.924940 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914615 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 13:48:08.924940 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914617 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 13:48:08.924940 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914620 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 13:48:08.924940 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914623 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 13:48:08.924940 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914625 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 13:48:08.924940 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914627 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 13:48:08.924940 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914630 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 13:48:08.924940 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914632 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 13:48:08.924940 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914635 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 13:48:08.924940 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914638 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 13:48:08.925413 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914640 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 13:48:08.925413 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914643 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 13:48:08.925413 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914645 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 30 13:48:08.925413 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914648 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 13:48:08.925413 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914650 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 13:48:08.925413 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914653 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 13:48:08.925413 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914655 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 13:48:08.925413 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914657 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 13:48:08.925413 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914660 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 13:48:08.925413 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914663 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 13:48:08.925413 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914665 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 13:48:08.925413 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914668 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 13:48:08.925413 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914670 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 13:48:08.925413 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914673 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 13:48:08.925413 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914675 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 13:48:08.925413 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914678 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 13:48:08.925413 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914682 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 13:48:08.925413 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914684 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 13:48:08.925413 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914687 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 13:48:08.925830 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914690 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 13:48:08.925830 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914693 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 13:48:08.925830 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914696 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 13:48:08.925830 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914700 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 13:48:08.925830 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914702 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 13:48:08.925830 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914705 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 13:48:08.925830 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.914708 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 13:48:08.925830 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.921691 2580 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 13:48:08.925830 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.921708 2580 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 13:48:08.925830 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.921724 2580 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 13:48:08.925830 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.921728 2580 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 13:48:08.925830 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921793 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 13:48:08.925830 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921798 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 13:48:08.925830 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921802 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 13:48:08.925830 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921805 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 13:48:08.925830 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921807 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 13:48:08.925830 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921810 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 13:48:08.925830 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921812 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 13:48:08.925830 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921815 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 13:48:08.926304 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921818 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 13:48:08.926304 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921821 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 13:48:08.926304 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921823 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 13:48:08.926304 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921826 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 13:48:08.926304 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921829 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 13:48:08.926304 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921832 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 13:48:08.926304 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921834 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 13:48:08.926304 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921837 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 13:48:08.926304 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921839 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 13:48:08.926304 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921842 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 13:48:08.926304 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921844 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 13:48:08.926304 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921847 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 13:48:08.926304 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921850 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 13:48:08.926304 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921852 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 13:48:08.926304 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921855 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 13:48:08.926304 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921857 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 13:48:08.926304 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921860 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 13:48:08.926304 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921863 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 13:48:08.926304 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921866 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 13:48:08.926749 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921868 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 13:48:08.926749 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921871 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 13:48:08.926749 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921875 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 13:48:08.926749 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921878 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 13:48:08.926749 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921882 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 13:48:08.926749 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921886 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 13:48:08.926749 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921889 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 13:48:08.926749 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921892 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 13:48:08.926749 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921895 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 13:48:08.926749 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921897 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 13:48:08.926749 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921901 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 13:48:08.926749 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921903 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 13:48:08.926749 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921906 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 13:48:08.926749 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921909 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 13:48:08.926749 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921911 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 13:48:08.926749 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921914 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 13:48:08.926749 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921916 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 13:48:08.926749 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921919 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 13:48:08.926749 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921922 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 13:48:08.926749 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921925 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 13:48:08.927285 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921927 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 13:48:08.927285 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921930 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 13:48:08.927285 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921932 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 13:48:08.927285 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921935 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 13:48:08.927285 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921938 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 13:48:08.927285 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921940 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 13:48:08.927285 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921943 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 13:48:08.927285 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921945 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 13:48:08.927285 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921948 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 13:48:08.927285 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921951 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 13:48:08.927285 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921953 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 13:48:08.927285 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921956 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 13:48:08.927285 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921959 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 13:48:08.927285 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921961 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 13:48:08.927285 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921964 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 13:48:08.927285 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921967 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 13:48:08.927285 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921970 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 13:48:08.927285 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921972 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 13:48:08.927285 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921975 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 13:48:08.927711 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921977 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 13:48:08.927711 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921980 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 13:48:08.927711 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921982 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 13:48:08.927711 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921985 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 13:48:08.927711 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921987 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 13:48:08.927711 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921990 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 13:48:08.927711 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921992 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 13:48:08.927711 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921995 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 13:48:08.927711 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.921997 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 13:48:08.927711 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922000 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 13:48:08.927711 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922002 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 13:48:08.927711 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922005 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 13:48:08.927711 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922008 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 13:48:08.927711 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922010 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 13:48:08.927711 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922013 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 13:48:08.927711 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922015 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 13:48:08.927711 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922017 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 13:48:08.927711 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922020 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 13:48:08.927711 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922023 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 13:48:08.927711 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922025 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 13:48:08.928183 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922027 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 13:48:08.928183 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922030 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 13:48:08.928183 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922032 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 13:48:08.928183 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922035 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 13:48:08.928183 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922037 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 13:48:08.928183 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922040 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 13:48:08.928183 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922043 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 13:48:08.928183 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922045 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 13:48:08.928183 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922048 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 13:48:08.928183 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922050 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 13:48:08.928183 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922053 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 13:48:08.928183 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922055 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 13:48:08.928183 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922058 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 13:48:08.928183 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922073 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 13:48:08.928183 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922076 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 13:48:08.928183 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922079 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 13:48:08.928183 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922082 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 30 13:48:08.928183 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922085 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 13:48:08.928183 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922089 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 13:48:08.928183 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922092 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 13:48:08.928735 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.922121 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 13:48:08.928735 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922258 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 13:48:08.928735 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922263 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 13:48:08.928735 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922266 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 13:48:08.928735 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922268 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 13:48:08.928735 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922271 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 13:48:08.928735 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922273 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 13:48:08.928735 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922276 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 13:48:08.928735 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922278 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 13:48:08.928735 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922281 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 13:48:08.928735 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922283 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 13:48:08.928735 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922286 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 13:48:08.928735 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922288 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 13:48:08.928735 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922291 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 13:48:08.928735 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922293 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 13:48:08.928735 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922296 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 13:48:08.929421 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922298 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 13:48:08.929421 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922301 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 13:48:08.929421 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922304 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 13:48:08.929421 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922307 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 13:48:08.929421 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922309 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 13:48:08.929421 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922312 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 13:48:08.929421 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922315 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 13:48:08.929421 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922318 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 13:48:08.929421 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922320 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 13:48:08.929421 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922323 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 13:48:08.929421 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922325 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 13:48:08.929421 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922328 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 13:48:08.929421 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922330 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 13:48:08.929421 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922333 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 13:48:08.929421 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922335 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 13:48:08.929421 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922337 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 13:48:08.929421 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922340 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 13:48:08.929421 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922342 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 13:48:08.929421 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922345 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 13:48:08.929421 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922347 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 13:48:08.930178 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922350 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 13:48:08.930178 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922352 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 13:48:08.930178 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922355 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 13:48:08.930178 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922358 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 13:48:08.930178 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922360 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 13:48:08.930178 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922362 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 13:48:08.930178 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922365 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 13:48:08.930178 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922367 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 13:48:08.930178 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922370 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 13:48:08.930178 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922372 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 13:48:08.930178 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922374 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 13:48:08.930178 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922377 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 13:48:08.930178 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922380 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 13:48:08.930178 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922382 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 13:48:08.930178 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922385 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 13:48:08.930178 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922388 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 13:48:08.930178 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922390 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 13:48:08.930178 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922392 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 13:48:08.930178 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922395 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 13:48:08.930178 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922398 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 13:48:08.930680 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922400 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 13:48:08.930680 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922403 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 13:48:08.930680 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922405 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 13:48:08.930680 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922408 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 13:48:08.930680 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922411 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 30 13:48:08.930680 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922413 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 13:48:08.930680 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922415 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 13:48:08.930680 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922418 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 13:48:08.930680 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922421 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 13:48:08.930680 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922423 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 13:48:08.930680 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922426 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 13:48:08.930680 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922428 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 13:48:08.930680 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922431 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 13:48:08.930680 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922433 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 13:48:08.930680 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922436 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 13:48:08.930680 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922438 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 13:48:08.930680 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922442 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 13:48:08.930680 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922445 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 13:48:08.930680 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922448 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 13:48:08.931283 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922451 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 13:48:08.931283 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922453 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 13:48:08.931283 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922456 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 13:48:08.931283 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922458 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 13:48:08.931283 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922460 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 13:48:08.931283 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922463 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 13:48:08.931283 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922466 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 13:48:08.931283 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922468 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 13:48:08.931283 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922471 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 13:48:08.931283 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922473 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 13:48:08.931283 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922476 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 13:48:08.931283 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922478 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 13:48:08.931283 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922481 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 13:48:08.931283 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922484 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 13:48:08.931283 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922486 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 13:48:08.931283 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922489 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 13:48:08.931283 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922491 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 13:48:08.931283 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922494 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 13:48:08.931283 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922496 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 13:48:08.931283 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922499 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 13:48:08.931794 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922501 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 13:48:08.931794 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922504 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 13:48:08.931794 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922506 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 13:48:08.931794 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922508 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 13:48:08.931794 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922511 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 13:48:08.931794 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922513 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 13:48:08.931794 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922516 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 13:48:08.931794 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922520 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 13:48:08.931794 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922524 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 13:48:08.931794 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922527 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 13:48:08.931794 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922530 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 13:48:08.931794 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922533 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 13:48:08.931794 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.922555 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 13:48:08.931794 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922585 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 13:48:08.931794 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922589 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 13:48:08.932158 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922592 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 13:48:08.932158 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922596 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 13:48:08.932158 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922598 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 13:48:08.932158 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922601 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 13:48:08.932158 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922603 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 13:48:08.932158 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922606 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 13:48:08.932158 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922608 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 13:48:08.932158 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922611 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 13:48:08.932158 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922613 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 13:48:08.932158 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922616 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 13:48:08.932158 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922618 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 13:48:08.932158 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922621 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 13:48:08.932158 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922624 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 13:48:08.932158 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922626 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 13:48:08.932158 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922629 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 13:48:08.932158 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922631 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 13:48:08.932158 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922634 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 13:48:08.932158 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922636 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 13:48:08.932158 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922638 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 13:48:08.932158 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922641 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 13:48:08.932623 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922644 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 13:48:08.932623 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922646 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 13:48:08.932623 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922649 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 13:48:08.932623 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922651 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 13:48:08.932623 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922654 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 13:48:08.932623 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922657 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 13:48:08.932623 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922659 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 13:48:08.932623 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922662 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 13:48:08.932623 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922664 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 13:48:08.932623 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922667 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 13:48:08.932623 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922669 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 13:48:08.932623 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922672 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 13:48:08.932623 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922674 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 13:48:08.932623 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922677 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 13:48:08.932623 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922680 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 13:48:08.932623 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922682 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 13:48:08.932623 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922684 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 13:48:08.932623 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922687 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 13:48:08.932623 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922689 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 13:48:08.932623 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922692 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 13:48:08.933103 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922694 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 13:48:08.933103 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922696 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 13:48:08.933103 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922699 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 30 13:48:08.933103 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922701 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 13:48:08.933103 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922704 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 13:48:08.933103 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922707 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 13:48:08.933103 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922717 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 13:48:08.933103 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922721 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 13:48:08.933103 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922725 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 13:48:08.933103 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922727 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 13:48:08.933103 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922729 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 13:48:08.933103 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922732 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 13:48:08.933103 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922735 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 13:48:08.933103 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922738 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 13:48:08.933103 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922742 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 13:48:08.933103 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922745 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 13:48:08.933103 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922748 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 13:48:08.933103 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922750 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 13:48:08.933103 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922753 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 13:48:08.933575 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922755 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 13:48:08.933575 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922757 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 13:48:08.933575 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922760 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 13:48:08.933575 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922763 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 13:48:08.933575 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922765 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 13:48:08.933575 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922771 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 13:48:08.933575 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922774 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 13:48:08.933575 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922777 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 13:48:08.933575 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922779 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 13:48:08.933575 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922782 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 13:48:08.933575 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922785 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 13:48:08.933575 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922788 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 13:48:08.933575 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922790 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 13:48:08.933575 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922792 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 13:48:08.933575 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922795 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 13:48:08.933575 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922798 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 13:48:08.933575 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922800 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 13:48:08.933575 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922803 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 13:48:08.933575 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922806 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 13:48:08.933575 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922809 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 13:48:08.934027 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922812 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 13:48:08.934027 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922814 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 13:48:08.934027 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922817 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 13:48:08.934027 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922819 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 13:48:08.934027 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922822 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 13:48:08.934027 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922826 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 13:48:08.934027 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922829 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 13:48:08.934027 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922832 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 13:48:08.934027 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922836 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 13:48:08.934027 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922838 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 13:48:08.934027 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922841 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 13:48:08.934027 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922844 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 13:48:08.934027 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922847 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 13:48:08.934027 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922849 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 13:48:08.934027 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922852 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 13:48:08.934027 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922854 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 13:48:08.934027 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922857 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 13:48:08.934027 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922861 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 13:48:08.934027 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922863 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 13:48:08.934470 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922866 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 13:48:08.934470 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922869 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 13:48:08.934470 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922871 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 13:48:08.934470 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922874 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 13:48:08.934470 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922878 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 13:48:08.934470 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:08.922880 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 13:48:08.934470 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.922931 2580 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 13:48:08.934470 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.924964 2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 13:48:08.934470 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.925846 2580 server.go:1014] "Starting client certificate rotation" Oct 30 13:48:08.934470 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.925944 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 13:48:08.934470 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.925971 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 13:48:08.934470 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.927776 2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 13:48:08.934470 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.931654 2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 13:48:08.941360 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.941341 2580 log.go:25] "Validated CRI v1 runtime API" Oct 30 13:48:08.945164 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.945150 2580 log.go:25] "Validated CRI v1 image API" Oct 30 13:48:08.945905 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.945876 2580 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 13:48:08.948755 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.948736 2580 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8a52321c-5064-4169-842a-85c03a43f145:/dev/nvme0n1p3 99ed940f-ac2c-4e29-a416-ab44cbc6aa68:/dev/nvme0n1p4] Oct 30 13:48:08.948824 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.948754 2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 13:48:08.953492 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.953390 2580 manager.go:217] Machine: {Timestamp:2026-10-30 13:48:08.952308078 +0000 UTC m=+0.285407091 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3174252 MemoryCapacity:33164324864 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22720807688383c6a0278a1338e7f7 SystemUUID:ec227208-0768-8383-c6a0-278a1338e7f7 BootID:bdf54b49-b7fe-4723-969d-fc69e66e639f Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582164480 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582160384 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f1:78:7b:9f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f1:78:7b:9f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4e:bd:48:89:d4:82 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164324864 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 13:48:08.953492 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.953492 2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 13:48:08.953641 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.953562 2580 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 13:48:08.956369 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.956344 2580 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 13:48:08.956497 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.956371 2580 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-105.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 13:48:08.956540 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.956498 2580 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 13:48:08.956540 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.956507 2580 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 13:48:08.956540 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.956528 2580 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 13:48:08.957197 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.957187 2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 13:48:08.958152 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.958142 2580 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 13:48:08.962303 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.962291 2580 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 13:48:08.962415 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.962406 2580 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 13:48:08.963154 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.963144 2580 kubelet.go:493] "Attempting to sync node with API server" Oct 30 13:48:08.963188 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.963162 2580 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 13:48:08.963188 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.963174 2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 13:48:08.963188 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.963183 2580 kubelet.go:405] "Adding apiserver pod source" Oct 30 13:48:08.963298 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.963191 2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 13:48:08.963505 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.963494 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 13:48:08.965525 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.965508 2580 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 13:48:08.971554 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.971539 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 13:48:08.972494 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.972482 2580 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 13:48:08.972539 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.972511 2580 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 13:48:08.972652 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.972642 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 13:48:08.972683 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.972656 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 13:48:08.972683 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.972663 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 13:48:08.972683 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.972669 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 13:48:08.972683 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.972677 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 13:48:08.972785 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.972686 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 13:48:08.973225 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.973217 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 13:48:08.973254 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.973226 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 13:48:08.973254 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.973233 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 13:48:08.973254 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.973238 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 13:48:08.973254 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.973246 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 13:48:08.973254 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.973255 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 13:48:08.974523 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.974514 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 13:48:08.974523 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.974523 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 13:48:08.978072 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.978048 2580 server.go:1263] "Started kubelet" Oct 30 13:48:08.978163 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.978131 2580 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 13:48:08.981841 ip-10-0-140-105 systemd[1]: Started Kubernetes Kubelet. Oct 30 13:48:08.983091 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.981281 2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 13:48:08.983091 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.981984 2580 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 13:48:08.984801 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.984696 2580 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 13:48:08.985551 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.985528 2580 server.go:317] "Adding debug handlers to kubelet server" Oct 30 13:48:08.986807 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:08.986787 2580 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 13:48:08.993445 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.993426 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 13:48:08.994409 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.994395 2580 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 13:48:08.994551 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:08.994509 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-105.ec2.internal\" not found" Oct 30 13:48:08.995033 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.995012 2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 13:48:08.995163 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.995019 2580 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 13:48:08.995163 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.995156 2580 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 13:48:08.995291 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.995193 2580 factory.go:55] Registering systemd factory Oct 30 13:48:08.995291 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.995215 2580 factory.go:223] Registration of the systemd container factory successfully Oct 30 13:48:08.995291 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.995237 2580 reconstruct.go:97] "Volume reconstruction finished" Oct 30 13:48:08.995291 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.995247 2580 reconciler.go:29] "Reconciler: start to sync state" Oct 30 13:48:08.996151 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.996138 2580 factory.go:153] Registering CRI-O factory Oct 30 13:48:08.996151 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.996151 2580 factory.go:223] Registration of the crio container factory successfully Oct 30 13:48:08.996237 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.996190 2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 13:48:08.996237 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.996203 2580 factory.go:103] Registering Raw factory Oct 30 13:48:08.996237 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.996214 2580 manager.go:1199] Started watching for new ooms in manager Oct 30 13:48:08.997035 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:08.997024 2580 manager.go:319] Starting recovery of all containers Oct 30 13:48:09.011428 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.011406 2580 manager.go:324] Recovery completed Oct 30 13:48:09.016303 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.016286 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 13:48:09.018327 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.018300 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-105.ec2.internal" event="NodeHasSufficientMemory" Oct 30 13:48:09.018421 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.018338 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-105.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 13:48:09.018421 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.018348 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-105.ec2.internal" event="NodeHasSufficientPID" Oct 30 13:48:09.018942 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.018929 2580 cpu_manager.go:226] "Starting" policy="none" Oct 30 13:48:09.018942 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.018934 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 13:48:09.018942 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.018940 2580 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 13:48:09.019101 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.018958 2580 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 13:48:09.019621 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.019604 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-105.ec2.internal" event="NodeHasSufficientMemory" Oct 30 13:48:09.019670 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.019638 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-105.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 13:48:09.019670 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.019651 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-105.ec2.internal" event="NodeHasSufficientPID" Oct 30 13:48:09.021210 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.021198 2580 policy_none.go:50] "Start" Oct 30 13:48:09.021242 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.021214 2580 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 13:48:09.021242 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.021224 2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 13:48:09.023170 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.023160 2580 policy_none.go:44] "Start" Oct 30 13:48:09.031038 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:09.031010 2580 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-105.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 13:48:09.031138 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.031015 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-105.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 13:48:09.031569 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.031553 2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 13:48:09.031881 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:09.031015 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-105.ec2.internal.18e352c6770e59d4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-105.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-105.ec2.internal,},FirstTimestamp:2026-10-30 13:48:08.97802082 +0000 UTC m=+0.311119832,LastTimestamp:2026-10-30 13:48:08.97802082 +0000 UTC m=+0.311119832,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-105.ec2.internal,}" Oct 30 13:48:09.051709 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:09.051649 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-105.ec2.internal.18e352c679755b99 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-105.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-105.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-105.ec2.internal,},FirstTimestamp:2026-10-30 13:48:09.018325913 +0000 UTC m=+0.351424925,LastTimestamp:2026-10-30 13:48:09.018325913 +0000 UTC m=+0.351424925,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-105.ec2.internal,}" Oct 30 13:48:09.051977 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:09.051950 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 13:48:09.056811 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.056794 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5gjdd" Oct 30 13:48:09.059642 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.059617 2580 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 13:48:09.061930 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.061918 2580 manager.go:341] "Starting Device Plugin manager" Oct 30 13:48:09.061981 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:09.061959 2580 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 13:48:09.061981 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.061969 2580 server.go:87] "Starting device plugin registration server" Oct 30 13:48:09.062255 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.062219 2580 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 13:48:09.062255 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.062234 2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 13:48:09.062388 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.062327 2580 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 13:48:09.062468 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.062427 2580 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 13:48:09.062468 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.062446 2580 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 13:48:09.063580 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:09.063564 2580 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 13:48:09.063651 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:09.063607 2580 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-105.ec2.internal\" not found" Oct 30 13:48:09.065107 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:09.065089 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 13:48:09.065184 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:09.065128 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-105.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 13:48:09.072324 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.072309 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5gjdd" Oct 30 13:48:09.092789 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:09.079479 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-105.ec2.internal.18e352c679759ec4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-105.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-105.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-105.ec2.internal,},FirstTimestamp:2026-10-30 13:48:09.018343108 +0000 UTC m=+0.351442120,LastTimestamp:2026-10-30 13:48:09.018343108 +0000 UTC m=+0.351442120,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-105.ec2.internal,}" Oct 30 13:48:09.124477 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.124455 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 13:48:09.125591 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.125573 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 13:48:09.125591 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.125589 2580 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 13:48:09.125708 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.125606 2580 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 13:48:09.125708 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:09.125638 2580 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 13:48:09.141012 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.140992 2580 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 13:48:09.162945 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.162904 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 13:48:09.163705 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.163690 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-105.ec2.internal" event="NodeHasSufficientMemory" Oct 30 13:48:09.163789 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.163720 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-105.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 13:48:09.163789 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.163733 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-105.ec2.internal" event="NodeHasSufficientPID" Oct 30 13:48:09.163789 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.163760 2580 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-105.ec2.internal" Oct 30 13:48:09.182339 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.182323 2580 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-105.ec2.internal" Oct 30 13:48:09.182410 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:09.182345 2580 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-105.ec2.internal\": node \"ip-10-0-140-105.ec2.internal\" not found" Oct 30 13:48:09.217181 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:09.217158 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-105.ec2.internal\" not found" Oct 30 13:48:09.226271 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.226252 2580 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-105.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-105.ec2.internal"] Oct 30 13:48:09.226319 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.226302 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 13:48:09.227003 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.226988 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-105.ec2.internal" event="NodeHasSufficientMemory" Oct 30 13:48:09.227057 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.227032 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-105.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 13:48:09.227057 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.227045 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-105.ec2.internal" event="NodeHasSufficientPID" Oct 30 13:48:09.229229 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.229217 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 13:48:09.229371 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.229357 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-105.ec2.internal" Oct 30 13:48:09.229428 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.229387 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 13:48:09.230089 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.230055 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-105.ec2.internal" event="NodeHasSufficientMemory" Oct 30 13:48:09.230172 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.230095 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-105.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 13:48:09.230172 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.230106 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-105.ec2.internal" event="NodeHasSufficientPID" Oct 30 13:48:09.230172 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.230121 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-105.ec2.internal" event="NodeHasSufficientMemory" Oct 30 13:48:09.230172 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.230139 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-105.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 13:48:09.230172 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.230150 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-105.ec2.internal" event="NodeHasSufficientPID" Oct 30 13:48:09.232173 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.232156 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-105.ec2.internal" Oct 30 13:48:09.232240 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.232180 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 13:48:09.232874 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.232859 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-105.ec2.internal" event="NodeHasSufficientMemory" Oct 30 13:48:09.232950 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.232887 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-105.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 13:48:09.232950 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.232911 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-105.ec2.internal" event="NodeHasSufficientPID" Oct 30 13:48:09.262504 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:09.262367 2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-105.ec2.internal\" not found" node="ip-10-0-140-105.ec2.internal" Oct 30 13:48:09.264407 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:09.264393 2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-105.ec2.internal\" not found" node="ip-10-0-140-105.ec2.internal" Oct 30 13:48:09.317594 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:09.317576 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-105.ec2.internal\" not found" Oct 30 13:48:09.396804 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.396785 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7f6d989ed16482ea69070d9acfe2e5f0-config\") pod \"kube-apiserver-proxy-ip-10-0-140-105.ec2.internal\" (UID: \"7f6d989ed16482ea69070d9acfe2e5f0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-105.ec2.internal" Oct 30 13:48:09.396879 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.396807 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/18c4e529019bfc1529902a8bff559ebf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-105.ec2.internal\" (UID: \"18c4e529019bfc1529902a8bff559ebf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-105.ec2.internal" Oct 30 13:48:09.396879 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.396823 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/18c4e529019bfc1529902a8bff559ebf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-105.ec2.internal\" (UID: \"18c4e529019bfc1529902a8bff559ebf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-105.ec2.internal" Oct 30 13:48:09.417907 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:09.417876 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-105.ec2.internal\" not found" Oct 30 13:48:09.497258 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.497234 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/18c4e529019bfc1529902a8bff559ebf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-105.ec2.internal\" (UID: \"18c4e529019bfc1529902a8bff559ebf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-105.ec2.internal" Oct 30 13:48:09.497333 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.497269 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/18c4e529019bfc1529902a8bff559ebf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-105.ec2.internal\" (UID: \"18c4e529019bfc1529902a8bff559ebf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-105.ec2.internal" Oct 30 13:48:09.497333 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.497290 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7f6d989ed16482ea69070d9acfe2e5f0-config\") pod \"kube-apiserver-proxy-ip-10-0-140-105.ec2.internal\" (UID: \"7f6d989ed16482ea69070d9acfe2e5f0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-105.ec2.internal" Oct 30 13:48:09.497333 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.497324 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7f6d989ed16482ea69070d9acfe2e5f0-config\") pod \"kube-apiserver-proxy-ip-10-0-140-105.ec2.internal\" (UID: \"7f6d989ed16482ea69070d9acfe2e5f0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-105.ec2.internal" Oct 30 13:48:09.497333 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.497330 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/18c4e529019bfc1529902a8bff559ebf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-105.ec2.internal\" (UID: \"18c4e529019bfc1529902a8bff559ebf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-105.ec2.internal" Oct 30 13:48:09.497470 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.497334 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/18c4e529019bfc1529902a8bff559ebf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-105.ec2.internal\" (UID: \"18c4e529019bfc1529902a8bff559ebf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-105.ec2.internal" Oct 30 13:48:09.518303 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:09.518286 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-105.ec2.internal\" not found" Oct 30 13:48:09.566247 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.566220 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-105.ec2.internal" Oct 30 13:48:09.568330 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.568312 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-105.ec2.internal" Oct 30 13:48:09.619171 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:09.619148 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-105.ec2.internal\" not found" Oct 30 13:48:09.719635 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:09.719593 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-105.ec2.internal\" not found" Oct 30 13:48:09.820102 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:09.820083 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-105.ec2.internal\" not found" Oct 30 13:48:09.920669 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:09.920649 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-105.ec2.internal\" not found" Oct 30 13:48:09.925030 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.925015 2580 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 13:48:09.926098 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.926085 2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 13:48:09.926192 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.926176 2580 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 13:48:09.926248 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:09.926224 2580 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 13:48:10.020979 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:10.020953 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-105.ec2.internal\" not found" Oct 30 13:48:10.077936 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:10.077897 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 13:43:10 +0000 UTC" deadline="2028-06-17 04:28:03.410564448 +0000 UTC" Oct 30 13:48:10.077936 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:10.077932 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14294h39m53.33263695s" Oct 30 13:48:10.121351 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:10.121326 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-105.ec2.internal\" not found" Oct 30 13:48:10.122230 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:10.122197 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18c4e529019bfc1529902a8bff559ebf.slice/crio-20a900a2eab78697079ac3639d0fcc4d077d0e5dcc9fd358b16c4b1215400dc0 WatchSource:0}: Error finding container 20a900a2eab78697079ac3639d0fcc4d077d0e5dcc9fd358b16c4b1215400dc0: Status 404 returned error can't find the container with id 20a900a2eab78697079ac3639d0fcc4d077d0e5dcc9fd358b16c4b1215400dc0 Oct 30 13:48:10.127893 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:10.127877 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 13:48:10.129599 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:10.129552 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-105.ec2.internal" event={"ID":"18c4e529019bfc1529902a8bff559ebf","Type":"ContainerStarted","Data":"20a900a2eab78697079ac3639d0fcc4d077d0e5dcc9fd358b16c4b1215400dc0"} Oct 30 13:48:10.137824 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:10.137806 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 13:48:10.140360 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:10.140345 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-105.ec2.internal" event="NodeHasSufficientMemory" Oct 30 13:48:10.140441 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:10.140373 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-105.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 13:48:10.140441 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:10.140386 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-105.ec2.internal" event="NodeHasSufficientPID" Oct 30 13:48:10.145684 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:10.145659 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f6d989ed16482ea69070d9acfe2e5f0.slice/crio-5e13fec44995927b6b7363560b232626b27499eec9ba59e9958238bd861f3dd1 WatchSource:0}: Error finding container 5e13fec44995927b6b7363560b232626b27499eec9ba59e9958238bd861f3dd1: Status 404 returned error can't find the container with id 5e13fec44995927b6b7363560b232626b27499eec9ba59e9958238bd861f3dd1 Oct 30 13:48:10.222408 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:10.222383 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-105.ec2.internal\" not found" Oct 30 13:48:11.636425 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:11.636380 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-105.ec2.internal\" not found" Oct 30 13:48:11.736899 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:11.736881 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-105.ec2.internal\" not found" Oct 30 13:48:11.815180 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:11.815157 2580 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 13:48:11.900090 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:11.900036 2580 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 13:48:11.909091 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:11.909057 2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-105.ec2.internal" Oct 30 13:48:11.932930 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:11.932909 2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 13:48:11.933954 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:11.933932 2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-105.ec2.internal" Oct 30 13:48:11.961469 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:11.961448 2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 13:48:12.278839 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.278757 2580 apiserver.go:52] "Watching apiserver" Oct 30 13:48:12.309090 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.309058 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 13:48:12.311142 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.311118 2580 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 13:48:12.311451 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.311431 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-pfg7x","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9bxl","openshift-image-registry/node-ca-llx79","openshift-multus/multus-rvt5b","openshift-network-diagnostics/network-check-target-dx94q","kube-system/kube-apiserver-proxy-ip-10-0-140-105.ec2.internal","openshift-network-operator/iptables-alerter-qf6mp","kube-system/konnectivity-agent-b622r","openshift-multus/multus-additional-cni-plugins-94cdn","openshift-ovn-kubernetes/ovnkube-node-bhxjf","kube-system/global-pull-secret-syncer-bdmgn","openshift-cluster-node-tuning-operator/tuned-tqst8","openshift-dns/node-resolver-bjswt","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-105.ec2.internal"] Oct 30 13:48:12.313989 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.313967 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qf6mp" Oct 30 13:48:12.316410 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.316279 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bjswt" Oct 30 13:48:12.318600 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.318509 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-llx79" Oct 30 13:48:12.320769 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.320751 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rvt5b" Oct 30 13:48:12.323304 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.323287 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dx94q" Oct 30 13:48:12.323395 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:12.323349 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dx94q" podUID="bf73153f-1421-44d6-abae-b76c3d90235e" Oct 30 13:48:12.325314 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.325294 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-b622r" Oct 30 13:48:12.327342 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.327319 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-94cdn" Oct 30 13:48:12.329729 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.329710 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfg7x" Oct 30 13:48:12.329826 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:12.329775 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pfg7x" podUID="da694c74-3bce-4d95-bd30-12a1891ad5e6" Oct 30 13:48:12.333548 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.333530 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 13:48:12.333707 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.333692 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2dbfk\"" Oct 30 13:48:12.333843 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.333826 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-s5pv6\"" Oct 30 13:48:12.333938 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.333892 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 13:48:12.334013 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.333999 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 13:48:12.334109 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.334095 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 13:48:12.334350 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.334307 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9bxl" Oct 30 13:48:12.334573 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.334313 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 13:48:12.335247 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.335147 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:12.339265 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.339246 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdmgn" Oct 30 13:48:12.339367 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:12.339327 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdmgn" podUID="6b5e74da-5460-44ed-a599-b01620b64c92" Oct 30 13:48:12.341873 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.341836 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:12.344757 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.344736 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-k4gxw\"" Oct 30 13:48:12.374116 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.374087 2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 13:48:12.374276 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.374167 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-24vlm" Oct 30 13:48:12.378596 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.378572 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-24vlm" Oct 30 13:48:12.409782 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.409754 2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 13:48:12.417116 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.417086 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdt2n\" (UniqueName: \"kubernetes.io/projected/d38c246c-86f8-4b57-89bd-db4cd0d3f6d1-kube-api-access-zdt2n\") pod \"iptables-alerter-qf6mp\" (UID: \"d38c246c-86f8-4b57-89bd-db4cd0d3f6d1\") " pod="openshift-network-operator/iptables-alerter-qf6mp" Oct 30 13:48:12.417274 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.417130 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/06f40dde-8b71-41e0-9acb-cede4f1d4347-registration-dir\") pod \"aws-ebs-csi-driver-node-h9bxl\" (UID: \"06f40dde-8b71-41e0-9acb-cede4f1d4347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9bxl" Oct 30 13:48:12.417274 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.417159 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-etc-openvswitch\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:12.417274 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.417254 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-host-cni-bin\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:12.417446 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.417288 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:12.417446 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.417318 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-etc-kubernetes\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:12.417446 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.417343 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-etc-sysctl-d\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:12.417446 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.417390 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-cnibin\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:12.417446 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.417435 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-host-cni-netd\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:12.417668 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.417479 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6b5e74da-5460-44ed-a599-b01620b64c92-dbus\") pod \"global-pull-secret-syncer-bdmgn\" (UID: \"6b5e74da-5460-44ed-a599-b01620b64c92\") " pod="kube-system/global-pull-secret-syncer-bdmgn" Oct 30 13:48:12.417668 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.417518 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c06b4b47-6141-4263-94b3-803d9f28bb94-host\") pod \"node-ca-llx79\" (UID: \"c06b4b47-6141-4263-94b3-803d9f28bb94\") " pod="openshift-image-registry/node-ca-llx79" Oct 30 13:48:12.417668 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.417543 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-lib-modules\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:12.417668 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.417580 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d38c246c-86f8-4b57-89bd-db4cd0d3f6d1-host-slash\") pod \"iptables-alerter-qf6mp\" (UID: \"d38c246c-86f8-4b57-89bd-db4cd0d3f6d1\") " pod="openshift-network-operator/iptables-alerter-qf6mp" Oct 30 13:48:12.417668 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.417596 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-host-run-ovn-kubernetes\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:12.417668 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.417614 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/aca47fe8-7068-4743-b6ab-283ee0874f66-ovn-node-metrics-cert\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:12.417668 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.417651 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0dda4c8a-827d-428f-a6a2-5e810210662b-system-cni-dir\") pod \"multus-additional-cni-plugins-94cdn\" (UID: \"0dda4c8a-827d-428f-a6a2-5e810210662b\") " pod="openshift-multus/multus-additional-cni-plugins-94cdn" Oct 30 13:48:12.417879 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.417679 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-etc-systemd\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:12.417879 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.417705 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d38c246c-86f8-4b57-89bd-db4cd0d3f6d1-iptables-alerter-script\") pod \"iptables-alerter-qf6mp\" (UID: \"d38c246c-86f8-4b57-89bd-db4cd0d3f6d1\") " pod="openshift-network-operator/iptables-alerter-qf6mp" Oct 30 13:48:12.417879 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.417719 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6b5e74da-5460-44ed-a599-b01620b64c92-kubelet-config\") pod \"global-pull-secret-syncer-bdmgn\" (UID: \"6b5e74da-5460-44ed-a599-b01620b64c92\") " pod="kube-system/global-pull-secret-syncer-bdmgn" Oct 30 13:48:12.417879 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.417739 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb8hg\" (UniqueName: \"kubernetes.io/projected/bf73153f-1421-44d6-abae-b76c3d90235e-kube-api-access-mb8hg\") pod \"network-check-target-dx94q\" (UID: \"bf73153f-1421-44d6-abae-b76c3d90235e\") " pod="openshift-network-diagnostics/network-check-target-dx94q" Oct 30 13:48:12.417879 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.417763 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ad85cf2a-6c44-4e85-8677-8b07cb9f78a4-agent-certs\") pod \"konnectivity-agent-b622r\" (UID: \"ad85cf2a-6c44-4e85-8677-8b07cb9f78a4\") " pod="kube-system/konnectivity-agent-b622r" Oct 30 13:48:12.417879 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.417783 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/06f40dde-8b71-41e0-9acb-cede4f1d4347-sys-fs\") pod \"aws-ebs-csi-driver-node-h9bxl\" (UID: \"06f40dde-8b71-41e0-9acb-cede4f1d4347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9bxl" Oct 30 13:48:12.417879 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.417804 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-host-slash\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:12.417879 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.417826 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-run-openvswitch\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:12.418150 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.417887 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-system-cni-dir\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:12.418150 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.417912 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-host-run-k8s-cni-cncf-io\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:12.418150 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.417938 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-systemd-units\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:12.418150 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.417961 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-run-systemd\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:12.418150 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.417982 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/aca47fe8-7068-4743-b6ab-283ee0874f66-env-overrides\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:12.418150 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.418041 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/aca47fe8-7068-4743-b6ab-283ee0874f66-ovnkube-script-lib\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:12.418150 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.418076 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c06b4b47-6141-4263-94b3-803d9f28bb94-serviceca\") pod \"node-ca-llx79\" (UID: \"c06b4b47-6141-4263-94b3-803d9f28bb94\") " pod="openshift-image-registry/node-ca-llx79" Oct 30 13:48:12.418150 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.418098 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-sys\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:12.418150 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.418123 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn2f4\" (UniqueName: \"kubernetes.io/projected/da694c74-3bce-4d95-bd30-12a1891ad5e6-kube-api-access-bn2f4\") pod \"network-metrics-daemon-pfg7x\" (UID: \"da694c74-3bce-4d95-bd30-12a1891ad5e6\") " pod="openshift-multus/network-metrics-daemon-pfg7x" Oct 30 13:48:12.418439 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.418166 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/06f40dde-8b71-41e0-9acb-cede4f1d4347-device-dir\") pod \"aws-ebs-csi-driver-node-h9bxl\" (UID: \"06f40dde-8b71-41e0-9acb-cede4f1d4347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9bxl" Oct 30 13:48:12.418439 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.418188 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-host-run-netns\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:12.418439 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.418211 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0dda4c8a-827d-428f-a6a2-5e810210662b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-94cdn\" (UID: \"0dda4c8a-827d-428f-a6a2-5e810210662b\") " pod="openshift-multus/multus-additional-cni-plugins-94cdn" Oct 30 13:48:12.418439 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.418233 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-etc-modprobe-d\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:12.418439 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.418256 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/06f40dde-8b71-41e0-9acb-cede4f1d4347-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h9bxl\" (UID: \"06f40dde-8b71-41e0-9acb-cede4f1d4347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9bxl" Oct 30 13:48:12.418439 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.418282 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-var-lib-kubelet\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:12.418439 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.418304 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da694c74-3bce-4d95-bd30-12a1891ad5e6-metrics-certs\") pod \"network-metrics-daemon-pfg7x\" (UID: \"da694c74-3bce-4d95-bd30-12a1891ad5e6\") " pod="openshift-multus/network-metrics-daemon-pfg7x" Oct 30 13:48:12.418439 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.418363 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-etc-sysconfig\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:12.418439 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.418396 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-etc-tuned\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:12.418439 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.418420 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-host-var-lib-kubelet\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:12.418753 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.418442 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-var-lib-openvswitch\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:12.418753 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.418469 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/06f40dde-8b71-41e0-9acb-cede4f1d4347-socket-dir\") pod \"aws-ebs-csi-driver-node-h9bxl\" (UID: \"06f40dde-8b71-41e0-9acb-cede4f1d4347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9bxl" Oct 30 13:48:12.418753 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.418492 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0dda4c8a-827d-428f-a6a2-5e810210662b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-94cdn\" (UID: \"0dda4c8a-827d-428f-a6a2-5e810210662b\") " pod="openshift-multus/multus-additional-cni-plugins-94cdn" Oct 30 13:48:12.418753 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.418515 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-etc-sysctl-conf\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:12.418753 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.418537 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfktw\" (UniqueName: \"kubernetes.io/projected/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-kube-api-access-xfktw\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:12.418753 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.418562 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-host-var-lib-cni-bin\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:12.418753 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.418605 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-host-var-lib-cni-multus\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:12.418753 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.418681 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/aca47fe8-7068-4743-b6ab-283ee0874f66-ovnkube-config\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:12.418753 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.418717 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltkwj\" (UniqueName: \"kubernetes.io/projected/e9a035cc-e78b-4bdb-b0c6-ef13bc07249a-kube-api-access-ltkwj\") pod \"node-resolver-bjswt\" (UID: \"e9a035cc-e78b-4bdb-b0c6-ef13bc07249a\") " pod="openshift-dns/node-resolver-bjswt" Oct 30 13:48:12.418753 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.418748 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-host\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:12.419164 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.418784 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/06f40dde-8b71-41e0-9acb-cede4f1d4347-etc-selinux\") pod \"aws-ebs-csi-driver-node-h9bxl\" (UID: \"06f40dde-8b71-41e0-9acb-cede4f1d4347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9bxl" Oct 30 13:48:12.419164 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.418814 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-log-socket\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:12.419164 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.418830 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6xq2\" (UniqueName: \"kubernetes.io/projected/aca47fe8-7068-4743-b6ab-283ee0874f66-kube-api-access-b6xq2\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:12.419164 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.418894 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brjls\" (UniqueName: \"kubernetes.io/projected/c06b4b47-6141-4263-94b3-803d9f28bb94-kube-api-access-brjls\") pod \"node-ca-llx79\" (UID: \"c06b4b47-6141-4263-94b3-803d9f28bb94\") " pod="openshift-image-registry/node-ca-llx79" Oct 30 13:48:12.419164 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.418932 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0dda4c8a-827d-428f-a6a2-5e810210662b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-94cdn\" (UID: \"0dda4c8a-827d-428f-a6a2-5e810210662b\") " pod="openshift-multus/multus-additional-cni-plugins-94cdn" Oct 30 13:48:12.419164 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.418985 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a85af541-54ed-4d24-b4c5-d69244396157-cni-binary-copy\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:12.419164 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.419012 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-multus-socket-dir-parent\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:12.419164 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.419056 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a85af541-54ed-4d24-b4c5-d69244396157-multus-daemon-config\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:12.419164 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.419081 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-node-log\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:12.419164 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.419126 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0dda4c8a-827d-428f-a6a2-5e810210662b-cni-binary-copy\") pod \"multus-additional-cni-plugins-94cdn\" (UID: \"0dda4c8a-827d-428f-a6a2-5e810210662b\") " pod="openshift-multus/multus-additional-cni-plugins-94cdn" Oct 30 13:48:12.419164 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.419152 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-multus-cni-dir\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:12.419667 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.419187 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-host-run-netns\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:12.419667 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.419214 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-multus-conf-dir\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:12.419667 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.419255 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ad85cf2a-6c44-4e85-8677-8b07cb9f78a4-konnectivity-ca\") pod \"konnectivity-agent-b622r\" (UID: \"ad85cf2a-6c44-4e85-8677-8b07cb9f78a4\") " pod="kube-system/konnectivity-agent-b622r" Oct 30 13:48:12.419667 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.419282 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kdz4\" (UniqueName: \"kubernetes.io/projected/06f40dde-8b71-41e0-9acb-cede4f1d4347-kube-api-access-7kdz4\") pod \"aws-ebs-csi-driver-node-h9bxl\" (UID: \"06f40dde-8b71-41e0-9acb-cede4f1d4347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9bxl" Oct 30 13:48:12.419667 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.419337 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e9a035cc-e78b-4bdb-b0c6-ef13bc07249a-hosts-file\") pod \"node-resolver-bjswt\" (UID: \"e9a035cc-e78b-4bdb-b0c6-ef13bc07249a\") " pod="openshift-dns/node-resolver-bjswt" Oct 30 13:48:12.419667 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.419380 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-host-run-multus-certs\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:12.419667 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.419434 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-etc-kubernetes\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:12.419667 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.419482 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9a035cc-e78b-4bdb-b0c6-ef13bc07249a-tmp-dir\") pod \"node-resolver-bjswt\" (UID: \"e9a035cc-e78b-4bdb-b0c6-ef13bc07249a\") " pod="openshift-dns/node-resolver-bjswt" Oct 30 13:48:12.419667 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.419512 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0dda4c8a-827d-428f-a6a2-5e810210662b-cnibin\") pod \"multus-additional-cni-plugins-94cdn\" (UID: \"0dda4c8a-827d-428f-a6a2-5e810210662b\") " pod="openshift-multus/multus-additional-cni-plugins-94cdn" Oct 30 13:48:12.419667 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.419552 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-hostroot\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:12.419667 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.419578 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp4w5\" (UniqueName: \"kubernetes.io/projected/a85af541-54ed-4d24-b4c5-d69244396157-kube-api-access-gp4w5\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:12.419667 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.419604 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-host-kubelet\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:12.419667 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.419628 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-run-ovn\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:12.420258 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.419686 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b5e74da-5460-44ed-a599-b01620b64c92-original-pull-secret\") pod \"global-pull-secret-syncer-bdmgn\" (UID: \"6b5e74da-5460-44ed-a599-b01620b64c92\") " pod="kube-system/global-pull-secret-syncer-bdmgn" Oct 30 13:48:12.420258 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.419717 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt65k\" (UniqueName: \"kubernetes.io/projected/0dda4c8a-827d-428f-a6a2-5e810210662b-kube-api-access-gt65k\") pod \"multus-additional-cni-plugins-94cdn\" (UID: \"0dda4c8a-827d-428f-a6a2-5e810210662b\") " pod="openshift-multus/multus-additional-cni-plugins-94cdn" Oct 30 13:48:12.420258 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.419744 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-run\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:12.420258 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.419769 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-tmp\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:12.433011 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.432974 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 13:48:12.433011 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.432977 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 13:48:12.433011 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.432981 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 13:48:12.433215 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.433017 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 13:48:12.433215 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.433029 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 13:48:12.433215 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.433188 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 13:48:12.433313 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.433219 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 13:48:12.433313 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.433271 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 13:48:12.433400 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.433337 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jc894\"" Oct 30 13:48:12.433400 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.433190 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 13:48:12.433400 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.433392 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 13:48:12.433533 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.433436 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 13:48:12.433580 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.433530 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5r2q6\"" Oct 30 13:48:12.433627 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.433588 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 13:48:12.433627 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.433596 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 13:48:12.433877 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.433628 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 13:48:12.433877 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.433667 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 13:48:12.433877 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.433693 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7j7wh\"" Oct 30 13:48:12.433877 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.433703 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 13:48:12.433877 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.433711 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 13:48:12.433877 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.433740 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 13:48:12.433877 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.433753 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 13:48:12.433877 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.433760 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 13:48:12.433877 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.433788 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 13:48:12.433877 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.433786 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-697m8\"" Oct 30 13:48:12.433877 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.433794 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xn5x9\"" Oct 30 13:48:12.433877 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.433814 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 13:48:12.433877 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.433870 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-mxshj\"" Oct 30 13:48:12.446959 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.446928 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-105.ec2.internal" event={"ID":"7f6d989ed16482ea69070d9acfe2e5f0","Type":"ContainerStarted","Data":"5e13fec44995927b6b7363560b232626b27499eec9ba59e9958238bd861f3dd1"} Oct 30 13:48:12.520755 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.520720 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a85af541-54ed-4d24-b4c5-d69244396157-multus-daemon-config\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:12.520755 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.520757 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-node-log\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:12.520958 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.520791 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0dda4c8a-827d-428f-a6a2-5e810210662b-cni-binary-copy\") pod \"multus-additional-cni-plugins-94cdn\" (UID: \"0dda4c8a-827d-428f-a6a2-5e810210662b\") " pod="openshift-multus/multus-additional-cni-plugins-94cdn" Oct 30 13:48:12.520958 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.520815 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-multus-cni-dir\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:12.520958 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.520835 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-host-run-netns\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:12.520958 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.520859 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-multus-conf-dir\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:12.520958 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.520879 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ad85cf2a-6c44-4e85-8677-8b07cb9f78a4-konnectivity-ca\") pod \"konnectivity-agent-b622r\" (UID: \"ad85cf2a-6c44-4e85-8677-8b07cb9f78a4\") " pod="kube-system/konnectivity-agent-b622r" Oct 30 13:48:12.520958 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.520871 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-node-log\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:12.520958 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.520904 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kdz4\" (UniqueName: \"kubernetes.io/projected/06f40dde-8b71-41e0-9acb-cede4f1d4347-kube-api-access-7kdz4\") pod \"aws-ebs-csi-driver-node-h9bxl\" (UID: \"06f40dde-8b71-41e0-9acb-cede4f1d4347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9bxl" Oct 30 13:48:12.520958 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.520937 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e9a035cc-e78b-4bdb-b0c6-ef13bc07249a-hosts-file\") pod \"node-resolver-bjswt\" (UID: \"e9a035cc-e78b-4bdb-b0c6-ef13bc07249a\") " pod="openshift-dns/node-resolver-bjswt" Oct 30 13:48:12.521319 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.520978 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-host-run-multus-certs\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:12.521319 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.520999 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-etc-kubernetes\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:12.521319 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.521024 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9a035cc-e78b-4bdb-b0c6-ef13bc07249a-tmp-dir\") pod \"node-resolver-bjswt\" (UID: \"e9a035cc-e78b-4bdb-b0c6-ef13bc07249a\") " pod="openshift-dns/node-resolver-bjswt" Oct 30 13:48:12.521319 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.521044 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0dda4c8a-827d-428f-a6a2-5e810210662b-cnibin\") pod \"multus-additional-cni-plugins-94cdn\" (UID: \"0dda4c8a-827d-428f-a6a2-5e810210662b\") " pod="openshift-multus/multus-additional-cni-plugins-94cdn" Oct 30 13:48:12.521319 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.521061 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-hostroot\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:12.521319 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.521077 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp4w5\" (UniqueName: \"kubernetes.io/projected/a85af541-54ed-4d24-b4c5-d69244396157-kube-api-access-gp4w5\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:12.521319 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.521101 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-host-kubelet\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:12.521319 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.521114 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-run-ovn\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:12.521319 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.521128 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b5e74da-5460-44ed-a599-b01620b64c92-original-pull-secret\") pod \"global-pull-secret-syncer-bdmgn\" (UID: \"6b5e74da-5460-44ed-a599-b01620b64c92\") " pod="kube-system/global-pull-secret-syncer-bdmgn" Oct 30 13:48:12.521319 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.521151 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt65k\" (UniqueName: \"kubernetes.io/projected/0dda4c8a-827d-428f-a6a2-5e810210662b-kube-api-access-gt65k\") pod \"multus-additional-cni-plugins-94cdn\" (UID: \"0dda4c8a-827d-428f-a6a2-5e810210662b\") " pod="openshift-multus/multus-additional-cni-plugins-94cdn" Oct 30 13:48:12.521319 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.521174 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-run\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:12.521319 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.521200 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-tmp\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:12.521319 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.521218 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdt2n\" (UniqueName: \"kubernetes.io/projected/d38c246c-86f8-4b57-89bd-db4cd0d3f6d1-kube-api-access-zdt2n\") pod \"iptables-alerter-qf6mp\" (UID: \"d38c246c-86f8-4b57-89bd-db4cd0d3f6d1\") " pod="openshift-network-operator/iptables-alerter-qf6mp" Oct 30 13:48:12.521319 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.521293 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/06f40dde-8b71-41e0-9acb-cede4f1d4347-registration-dir\") pod \"aws-ebs-csi-driver-node-h9bxl\" (UID: \"06f40dde-8b71-41e0-9acb-cede4f1d4347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9bxl" Oct 30 13:48:12.521319 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.521307 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-etc-openvswitch\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:12.521319 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.521327 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-host-cni-bin\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:12.522044 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.521341 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:12.522044 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.521363 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-etc-kubernetes\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:12.522044 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.521388 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-etc-sysctl-d\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:12.522044 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.521402 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-cnibin\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:12.522044 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.521421 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-host-cni-netd\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:12.522044 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.521448 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6b5e74da-5460-44ed-a599-b01620b64c92-dbus\") pod \"global-pull-secret-syncer-bdmgn\" (UID: \"6b5e74da-5460-44ed-a599-b01620b64c92\") " pod="kube-system/global-pull-secret-syncer-bdmgn" Oct 30 13:48:12.522044 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.521467 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a85af541-54ed-4d24-b4c5-d69244396157-multus-daemon-config\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:12.522044 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.521484 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c06b4b47-6141-4263-94b3-803d9f28bb94-host\") pod \"node-ca-llx79\" (UID: \"c06b4b47-6141-4263-94b3-803d9f28bb94\") " pod="openshift-image-registry/node-ca-llx79" Oct 30 13:48:12.522044 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.521533 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-lib-modules\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:12.522044 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.521563 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d38c246c-86f8-4b57-89bd-db4cd0d3f6d1-host-slash\") pod \"iptables-alerter-qf6mp\" (UID: \"d38c246c-86f8-4b57-89bd-db4cd0d3f6d1\") " pod="openshift-network-operator/iptables-alerter-qf6mp" Oct 30 13:48:12.522044 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.521590 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-host-run-ovn-kubernetes\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:12.522044 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.521604 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/aca47fe8-7068-4743-b6ab-283ee0874f66-ovn-node-metrics-cert\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:12.522044 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.521603 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-run\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:12.522044 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.521625 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0dda4c8a-827d-428f-a6a2-5e810210662b-system-cni-dir\") pod \"multus-additional-cni-plugins-94cdn\" (UID: \"0dda4c8a-827d-428f-a6a2-5e810210662b\") " pod="openshift-multus/multus-additional-cni-plugins-94cdn" Oct 30 13:48:12.522044 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.521657 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-etc-systemd\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:12.522044 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.521680 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e9a035cc-e78b-4bdb-b0c6-ef13bc07249a-hosts-file\") pod \"node-resolver-bjswt\" (UID: \"e9a035cc-e78b-4bdb-b0c6-ef13bc07249a\") " pod="openshift-dns/node-resolver-bjswt" Oct 30 13:48:12.522044 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.521738 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d38c246c-86f8-4b57-89bd-db4cd0d3f6d1-iptables-alerter-script\") pod \"iptables-alerter-qf6mp\" (UID: \"d38c246c-86f8-4b57-89bd-db4cd0d3f6d1\") " pod="openshift-network-operator/iptables-alerter-qf6mp" Oct 30 13:48:12.522827 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.521747 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-host-run-multus-certs\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:12.522827 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.521753 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6b5e74da-5460-44ed-a599-b01620b64c92-kubelet-config\") pod \"global-pull-secret-syncer-bdmgn\" (UID: \"6b5e74da-5460-44ed-a599-b01620b64c92\") " pod="kube-system/global-pull-secret-syncer-bdmgn" Oct 30 13:48:12.522827 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.521793 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb8hg\" (UniqueName: \"kubernetes.io/projected/bf73153f-1421-44d6-abae-b76c3d90235e-kube-api-access-mb8hg\") pod \"network-check-target-dx94q\" (UID: \"bf73153f-1421-44d6-abae-b76c3d90235e\") " pod="openshift-network-diagnostics/network-check-target-dx94q" Oct 30 13:48:12.522827 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.521808 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6b5e74da-5460-44ed-a599-b01620b64c92-kubelet-config\") pod \"global-pull-secret-syncer-bdmgn\" (UID: \"6b5e74da-5460-44ed-a599-b01620b64c92\") " pod="kube-system/global-pull-secret-syncer-bdmgn" Oct 30 13:48:12.522827 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.521822 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ad85cf2a-6c44-4e85-8677-8b07cb9f78a4-agent-certs\") pod \"konnectivity-agent-b622r\" (UID: \"ad85cf2a-6c44-4e85-8677-8b07cb9f78a4\") " pod="kube-system/konnectivity-agent-b622r" Oct 30 13:48:12.522827 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.521838 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0dda4c8a-827d-428f-a6a2-5e810210662b-cni-binary-copy\") pod \"multus-additional-cni-plugins-94cdn\" (UID: \"0dda4c8a-827d-428f-a6a2-5e810210662b\") " pod="openshift-multus/multus-additional-cni-plugins-94cdn" Oct 30 13:48:12.522827 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.521863 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/06f40dde-8b71-41e0-9acb-cede4f1d4347-sys-fs\") pod \"aws-ebs-csi-driver-node-h9bxl\" (UID: \"06f40dde-8b71-41e0-9acb-cede4f1d4347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9bxl" Oct 30 13:48:12.522827 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.521889 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-host-slash\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:12.522827 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.521917 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-host-run-netns\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:12.522827 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.521928 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-run-openvswitch\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:12.522827 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.521954 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-system-cni-dir\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:12.522827 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.521977 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-host-run-k8s-cni-cncf-io\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:12.522827 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.522000 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-systemd-units\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:12.522827 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.522032 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-run-systemd\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:12.522827 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.522080 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/aca47fe8-7068-4743-b6ab-283ee0874f66-env-overrides\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:12.522827 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.522101 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/aca47fe8-7068-4743-b6ab-283ee0874f66-ovnkube-script-lib\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:12.522827 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.522118 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c06b4b47-6141-4263-94b3-803d9f28bb94-serviceca\") pod \"node-ca-llx79\" (UID: \"c06b4b47-6141-4263-94b3-803d9f28bb94\") " pod="openshift-image-registry/node-ca-llx79" Oct 30 13:48:12.522827 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.522131 2580 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 13:48:12.523589 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.522154 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-sys\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:12.523589 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.522188 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn2f4\" (UniqueName: \"kubernetes.io/projected/da694c74-3bce-4d95-bd30-12a1891ad5e6-kube-api-access-bn2f4\") pod \"network-metrics-daemon-pfg7x\" (UID: \"da694c74-3bce-4d95-bd30-12a1891ad5e6\") " pod="openshift-multus/network-metrics-daemon-pfg7x" Oct 30 13:48:12.523589 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.522204 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/06f40dde-8b71-41e0-9acb-cede4f1d4347-device-dir\") pod \"aws-ebs-csi-driver-node-h9bxl\" (UID: \"06f40dde-8b71-41e0-9acb-cede4f1d4347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9bxl" Oct 30 13:48:12.523589 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.522229 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-host-run-netns\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:12.523589 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.522252 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0dda4c8a-827d-428f-a6a2-5e810210662b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-94cdn\" (UID: \"0dda4c8a-827d-428f-a6a2-5e810210662b\") " pod="openshift-multus/multus-additional-cni-plugins-94cdn" Oct 30 13:48:12.523589 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.522280 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-etc-modprobe-d\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:12.523589 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.522307 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/06f40dde-8b71-41e0-9acb-cede4f1d4347-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h9bxl\" (UID: \"06f40dde-8b71-41e0-9acb-cede4f1d4347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9bxl" Oct 30 13:48:12.523589 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.522332 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-var-lib-kubelet\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:12.523589 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.522399 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da694c74-3bce-4d95-bd30-12a1891ad5e6-metrics-certs\") pod \"network-metrics-daemon-pfg7x\" (UID: \"da694c74-3bce-4d95-bd30-12a1891ad5e6\") " pod="openshift-multus/network-metrics-daemon-pfg7x" Oct 30 13:48:12.523589 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.522422 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-etc-sysconfig\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:12.523589 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.522424 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ad85cf2a-6c44-4e85-8677-8b07cb9f78a4-konnectivity-ca\") pod \"konnectivity-agent-b622r\" (UID: \"ad85cf2a-6c44-4e85-8677-8b07cb9f78a4\") " pod="kube-system/konnectivity-agent-b622r" Oct 30 13:48:12.523589 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.522445 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-etc-tuned\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:12.523589 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.522496 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-host-var-lib-kubelet\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:12.523589 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.522525 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-var-lib-openvswitch\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:12.523589 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.522544 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/06f40dde-8b71-41e0-9acb-cede4f1d4347-socket-dir\") pod \"aws-ebs-csi-driver-node-h9bxl\" (UID: \"06f40dde-8b71-41e0-9acb-cede4f1d4347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9bxl" Oct 30 13:48:12.523589 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.522558 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-system-cni-dir\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:12.523589 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.522572 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0dda4c8a-827d-428f-a6a2-5e810210662b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-94cdn\" (UID: \"0dda4c8a-827d-428f-a6a2-5e810210662b\") " pod="openshift-multus/multus-additional-cni-plugins-94cdn" Oct 30 13:48:12.524355 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.522655 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-etc-sysctl-conf\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:12.524355 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.522673 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-hostroot\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:12.524355 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.522710 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfktw\" (UniqueName: \"kubernetes.io/projected/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-kube-api-access-xfktw\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:12.524355 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.522751 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-host-var-lib-cni-bin\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:12.524355 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.522769 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6b5e74da-5460-44ed-a599-b01620b64c92-dbus\") pod \"global-pull-secret-syncer-bdmgn\" (UID: \"6b5e74da-5460-44ed-a599-b01620b64c92\") " pod="kube-system/global-pull-secret-syncer-bdmgn" Oct 30 13:48:12.524355 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.522783 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-host-var-lib-cni-multus\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:12.524355 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.522822 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0dda4c8a-827d-428f-a6a2-5e810210662b-cnibin\") pod \"multus-additional-cni-plugins-94cdn\" (UID: \"0dda4c8a-827d-428f-a6a2-5e810210662b\") " pod="openshift-multus/multus-additional-cni-plugins-94cdn" Oct 30 13:48:12.524355 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.522828 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-host-var-lib-cni-multus\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:12.524355 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.522866 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/aca47fe8-7068-4743-b6ab-283ee0874f66-ovnkube-config\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:12.524355 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.522884 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-etc-sysctl-conf\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:12.524355 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.522922 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltkwj\" (UniqueName: \"kubernetes.io/projected/e9a035cc-e78b-4bdb-b0c6-ef13bc07249a-kube-api-access-ltkwj\") pod \"node-resolver-bjswt\" (UID: \"e9a035cc-e78b-4bdb-b0c6-ef13bc07249a\") " pod="openshift-dns/node-resolver-bjswt" Oct 30 13:48:12.524355 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.522933 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-host-run-k8s-cni-cncf-io\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:12.524355 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.522953 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-host\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:12.524355 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.522979 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/06f40dde-8b71-41e0-9acb-cede4f1d4347-etc-selinux\") pod \"aws-ebs-csi-driver-node-h9bxl\" (UID: \"06f40dde-8b71-41e0-9acb-cede4f1d4347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9bxl" Oct 30 13:48:12.524355 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.522987 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-systemd-units\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:12.524355 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.523003 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-log-socket\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:12.524355 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.523024 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6xq2\" (UniqueName: \"kubernetes.io/projected/aca47fe8-7068-4743-b6ab-283ee0874f66-kube-api-access-b6xq2\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:12.524355 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.523038 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-run-systemd\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:12.525135 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.523048 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brjls\" (UniqueName: \"kubernetes.io/projected/c06b4b47-6141-4263-94b3-803d9f28bb94-kube-api-access-brjls\") pod \"node-ca-llx79\" (UID: \"c06b4b47-6141-4263-94b3-803d9f28bb94\") " pod="openshift-image-registry/node-ca-llx79" Oct 30 13:48:12.525135 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.523071 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0dda4c8a-827d-428f-a6a2-5e810210662b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-94cdn\" (UID: \"0dda4c8a-827d-428f-a6a2-5e810210662b\") " pod="openshift-multus/multus-additional-cni-plugins-94cdn" Oct 30 13:48:12.525135 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.523093 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a85af541-54ed-4d24-b4c5-d69244396157-cni-binary-copy\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:12.525135 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.523095 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-host-kubelet\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:12.525135 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.523112 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-multus-socket-dir-parent\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:12.525135 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.523152 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-run-ovn\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:12.525135 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.523430 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/aca47fe8-7068-4743-b6ab-283ee0874f66-env-overrides\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:12.525135 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.523689 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/06f40dde-8b71-41e0-9acb-cede4f1d4347-registration-dir\") pod \"aws-ebs-csi-driver-node-h9bxl\" (UID: \"06f40dde-8b71-41e0-9acb-cede4f1d4347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9bxl" Oct 30 13:48:12.525135 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.523763 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-etc-openvswitch\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:12.525135 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.523808 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-host-cni-bin\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:12.525135 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.523849 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:12.525135 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.523917 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-etc-kubernetes\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:12.525135 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.523998 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-etc-sysctl-d\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:12.525135 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.524034 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/aca47fe8-7068-4743-b6ab-283ee0874f66-ovnkube-script-lib\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:12.525135 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.524048 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-cnibin\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:12.525135 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.524348 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9a035cc-e78b-4bdb-b0c6-ef13bc07249a-tmp-dir\") pod \"node-resolver-bjswt\" (UID: \"e9a035cc-e78b-4bdb-b0c6-ef13bc07249a\") " pod="openshift-dns/node-resolver-bjswt" Oct 30 13:48:12.525135 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.524537 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c06b4b47-6141-4263-94b3-803d9f28bb94-serviceca\") pod \"node-ca-llx79\" (UID: \"c06b4b47-6141-4263-94b3-803d9f28bb94\") " pod="openshift-image-registry/node-ca-llx79" Oct 30 13:48:12.525135 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.524851 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-etc-kubernetes\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:12.528075 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.524881 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/aca47fe8-7068-4743-b6ab-283ee0874f66-ovnkube-config\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:12.528075 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.525167 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-etc-sysconfig\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:12.528075 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.525370 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0dda4c8a-827d-428f-a6a2-5e810210662b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-94cdn\" (UID: \"0dda4c8a-827d-428f-a6a2-5e810210662b\") " pod="openshift-multus/multus-additional-cni-plugins-94cdn" Oct 30 13:48:12.528075 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.525700 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-run-openvswitch\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:12.528075 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.525757 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/06f40dde-8b71-41e0-9acb-cede4f1d4347-sys-fs\") pod \"aws-ebs-csi-driver-node-h9bxl\" (UID: \"06f40dde-8b71-41e0-9acb-cede4f1d4347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9bxl" Oct 30 13:48:12.528075 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.525808 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-multus-cni-dir\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:12.528075 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.525847 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-host-slash\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:12.528075 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.525879 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-multus-conf-dir\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:12.528075 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.525931 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-host-var-lib-kubelet\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:12.528075 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.525978 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-var-lib-openvswitch\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:12.528075 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.526804 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0dda4c8a-827d-428f-a6a2-5e810210662b-system-cni-dir\") pod \"multus-additional-cni-plugins-94cdn\" (UID: \"0dda4c8a-827d-428f-a6a2-5e810210662b\") " pod="openshift-multus/multus-additional-cni-plugins-94cdn" Oct 30 13:48:12.528075 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.526884 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-etc-systemd\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:12.528075 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.527298 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a85af541-54ed-4d24-b4c5-d69244396157-cni-binary-copy\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:12.528075 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.527517 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d38c246c-86f8-4b57-89bd-db4cd0d3f6d1-iptables-alerter-script\") pod \"iptables-alerter-qf6mp\" (UID: \"d38c246c-86f8-4b57-89bd-db4cd0d3f6d1\") " pod="openshift-network-operator/iptables-alerter-qf6mp" Oct 30 13:48:12.528659 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.528126 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-etc-tuned\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:12.528659 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.528198 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-host-var-lib-cni-bin\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:12.528659 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.525958 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-lib-modules\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:12.528659 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.528340 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-multus-socket-dir-parent\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:12.528659 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.528385 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-host\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:12.528659 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:12.528455 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 13:48:12.528659 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.528475 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-log-socket\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:12.528659 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:12.528577 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b5e74da-5460-44ed-a599-b01620b64c92-original-pull-secret podName:6b5e74da-5460-44ed-a599-b01620b64c92 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:13.028520486 +0000 UTC m=+3.048041878 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6b5e74da-5460-44ed-a599-b01620b64c92-original-pull-secret") pod "global-pull-secret-syncer-bdmgn" (UID: "6b5e74da-5460-44ed-a599-b01620b64c92") : object "kube-system"/"original-pull-secret" not registered Oct 30 13:48:12.529001 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.528889 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0dda4c8a-827d-428f-a6a2-5e810210662b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-94cdn\" (UID: \"0dda4c8a-827d-428f-a6a2-5e810210662b\") " pod="openshift-multus/multus-additional-cni-plugins-94cdn" Oct 30 13:48:12.529055 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.528457 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/06f40dde-8b71-41e0-9acb-cede4f1d4347-etc-selinux\") pod \"aws-ebs-csi-driver-node-h9bxl\" (UID: \"06f40dde-8b71-41e0-9acb-cede4f1d4347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9bxl" Oct 30 13:48:12.531573 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.529309 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/06f40dde-8b71-41e0-9acb-cede4f1d4347-socket-dir\") pod \"aws-ebs-csi-driver-node-h9bxl\" (UID: \"06f40dde-8b71-41e0-9acb-cede4f1d4347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9bxl" Oct 30 13:48:12.531573 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.529359 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-host-run-netns\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:12.531573 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.529499 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0dda4c8a-827d-428f-a6a2-5e810210662b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-94cdn\" (UID: \"0dda4c8a-827d-428f-a6a2-5e810210662b\") " pod="openshift-multus/multus-additional-cni-plugins-94cdn" Oct 30 13:48:12.531573 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.529692 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-etc-modprobe-d\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:12.531573 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.529750 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/06f40dde-8b71-41e0-9acb-cede4f1d4347-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h9bxl\" (UID: \"06f40dde-8b71-41e0-9acb-cede4f1d4347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9bxl" Oct 30 13:48:12.531573 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.529798 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-var-lib-kubelet\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:12.531573 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:12.529890 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 13:48:12.531573 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:12.529946 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da694c74-3bce-4d95-bd30-12a1891ad5e6-metrics-certs podName:da694c74-3bce-4d95-bd30-12a1891ad5e6 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:13.029928974 +0000 UTC m=+3.049450369 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/da694c74-3bce-4d95-bd30-12a1891ad5e6-metrics-certs") pod "network-metrics-daemon-pfg7x" (UID: "da694c74-3bce-4d95-bd30-12a1891ad5e6") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 13:48:12.531573 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.530001 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-sys\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:12.531573 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.530059 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-host-cni-netd\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:12.531573 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.530135 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c06b4b47-6141-4263-94b3-803d9f28bb94-host\") pod \"node-ca-llx79\" (UID: \"c06b4b47-6141-4263-94b3-803d9f28bb94\") " pod="openshift-image-registry/node-ca-llx79" Oct 30 13:48:12.531573 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.530189 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/06f40dde-8b71-41e0-9acb-cede4f1d4347-device-dir\") pod \"aws-ebs-csi-driver-node-h9bxl\" (UID: \"06f40dde-8b71-41e0-9acb-cede4f1d4347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9bxl" Oct 30 13:48:12.531573 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.530380 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-host-run-ovn-kubernetes\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:12.531573 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.530524 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d38c246c-86f8-4b57-89bd-db4cd0d3f6d1-host-slash\") pod \"iptables-alerter-qf6mp\" (UID: \"d38c246c-86f8-4b57-89bd-db4cd0d3f6d1\") " pod="openshift-network-operator/iptables-alerter-qf6mp" Oct 30 13:48:12.531573 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.530597 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ad85cf2a-6c44-4e85-8677-8b07cb9f78a4-agent-certs\") pod \"konnectivity-agent-b622r\" (UID: \"ad85cf2a-6c44-4e85-8677-8b07cb9f78a4\") " pod="kube-system/konnectivity-agent-b622r" Oct 30 13:48:12.533551 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.533526 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/aca47fe8-7068-4743-b6ab-283ee0874f66-ovn-node-metrics-cert\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:12.533708 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.533551 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-tmp\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:12.547186 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:12.547164 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 13:48:12.547305 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:12.547189 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 13:48:12.547305 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:12.547204 2580 projected.go:196] Error preparing data for projected volume kube-api-access-mb8hg for pod openshift-network-diagnostics/network-check-target-dx94q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 13:48:12.547305 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:12.547271 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf73153f-1421-44d6-abae-b76c3d90235e-kube-api-access-mb8hg podName:bf73153f-1421-44d6-abae-b76c3d90235e nodeName:}" failed. No retries permitted until 2026-10-30 13:48:13.047253378 +0000 UTC m=+3.066774777 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mb8hg" (UniqueName: "kubernetes.io/projected/bf73153f-1421-44d6-abae-b76c3d90235e-kube-api-access-mb8hg") pod "network-check-target-dx94q" (UID: "bf73153f-1421-44d6-abae-b76c3d90235e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 13:48:12.548022 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.547948 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kdz4\" (UniqueName: \"kubernetes.io/projected/06f40dde-8b71-41e0-9acb-cede4f1d4347-kube-api-access-7kdz4\") pod \"aws-ebs-csi-driver-node-h9bxl\" (UID: \"06f40dde-8b71-41e0-9acb-cede4f1d4347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9bxl" Oct 30 13:48:12.549273 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.549083 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp4w5\" (UniqueName: \"kubernetes.io/projected/a85af541-54ed-4d24-b4c5-d69244396157-kube-api-access-gp4w5\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:12.550066 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.550010 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdt2n\" (UniqueName: \"kubernetes.io/projected/d38c246c-86f8-4b57-89bd-db4cd0d3f6d1-kube-api-access-zdt2n\") pod \"iptables-alerter-qf6mp\" (UID: \"d38c246c-86f8-4b57-89bd-db4cd0d3f6d1\") " pod="openshift-network-operator/iptables-alerter-qf6mp" Oct 30 13:48:12.550887 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.550840 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6xq2\" (UniqueName: \"kubernetes.io/projected/aca47fe8-7068-4743-b6ab-283ee0874f66-kube-api-access-b6xq2\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:12.552259 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.552239 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfktw\" (UniqueName: \"kubernetes.io/projected/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-kube-api-access-xfktw\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:12.557338 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.557290 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltkwj\" (UniqueName: \"kubernetes.io/projected/e9a035cc-e78b-4bdb-b0c6-ef13bc07249a-kube-api-access-ltkwj\") pod \"node-resolver-bjswt\" (UID: \"e9a035cc-e78b-4bdb-b0c6-ef13bc07249a\") " pod="openshift-dns/node-resolver-bjswt" Oct 30 13:48:12.557589 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.557567 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brjls\" (UniqueName: \"kubernetes.io/projected/c06b4b47-6141-4263-94b3-803d9f28bb94-kube-api-access-brjls\") pod \"node-ca-llx79\" (UID: \"c06b4b47-6141-4263-94b3-803d9f28bb94\") " pod="openshift-image-registry/node-ca-llx79" Oct 30 13:48:12.558764 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.558743 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt65k\" (UniqueName: \"kubernetes.io/projected/0dda4c8a-827d-428f-a6a2-5e810210662b-kube-api-access-gt65k\") pod \"multus-additional-cni-plugins-94cdn\" (UID: \"0dda4c8a-827d-428f-a6a2-5e810210662b\") " pod="openshift-multus/multus-additional-cni-plugins-94cdn" Oct 30 13:48:12.559389 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.559356 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn2f4\" (UniqueName: \"kubernetes.io/projected/da694c74-3bce-4d95-bd30-12a1891ad5e6-kube-api-access-bn2f4\") pod \"network-metrics-daemon-pfg7x\" (UID: \"da694c74-3bce-4d95-bd30-12a1891ad5e6\") " pod="openshift-multus/network-metrics-daemon-pfg7x" Oct 30 13:48:12.629117 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.629085 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qf6mp" Oct 30 13:48:12.638255 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.638234 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bjswt" Oct 30 13:48:12.647989 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.647968 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-llx79" Oct 30 13:48:12.654887 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.654871 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rvt5b" Oct 30 13:48:12.661216 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:12.661187 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd38c246c_86f8_4b57_89bd_db4cd0d3f6d1.slice/crio-acfa03e8884898362b46b545016cc54a3e8c6767802f494b08ccd9e8d9a7b45e WatchSource:0}: Error finding container acfa03e8884898362b46b545016cc54a3e8c6767802f494b08ccd9e8d9a7b45e: Status 404 returned error can't find the container with id acfa03e8884898362b46b545016cc54a3e8c6767802f494b08ccd9e8d9a7b45e Oct 30 13:48:12.662098 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.662053 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-b622r" Oct 30 13:48:12.674742 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:12.674714 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc06b4b47_6141_4263_94b3_803d9f28bb94.slice/crio-119282a48a7da81396d040950eab5be9eed0d50580fde28f1b5714cf9ac5d7f4 WatchSource:0}: Error finding container 119282a48a7da81396d040950eab5be9eed0d50580fde28f1b5714cf9ac5d7f4: Status 404 returned error can't find the container with id 119282a48a7da81396d040950eab5be9eed0d50580fde28f1b5714cf9ac5d7f4 Oct 30 13:48:12.675532 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:12.675506 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda85af541_54ed_4d24_b4c5_d69244396157.slice/crio-1c3374fc75c25c90d2209a22f831ad738912d8e0e51227dafebb8818b4a6ea8d WatchSource:0}: Error finding container 1c3374fc75c25c90d2209a22f831ad738912d8e0e51227dafebb8818b4a6ea8d: Status 404 returned error can't find the container with id 1c3374fc75c25c90d2209a22f831ad738912d8e0e51227dafebb8818b4a6ea8d Oct 30 13:48:12.676196 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:12.676165 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad85cf2a_6c44_4e85_8677_8b07cb9f78a4.slice/crio-05342334dd6693d001bf65221626386848cbec0f12347b577c8df568ccf4ef04 WatchSource:0}: Error finding container 05342334dd6693d001bf65221626386848cbec0f12347b577c8df568ccf4ef04: Status 404 returned error can't find the container with id 05342334dd6693d001bf65221626386848cbec0f12347b577c8df568ccf4ef04 Oct 30 13:48:12.688572 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.688554 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-94cdn" Oct 30 13:48:12.695949 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.695926 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9bxl" Oct 30 13:48:12.698103 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.698086 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:12.707545 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:12.707500 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06f40dde_8b71_41e0_9acb_cede4f1d4347.slice/crio-346de427586bad84a8f4fb0455636dda4d7688ea0d7137117cfecc01d588bf21 WatchSource:0}: Error finding container 346de427586bad84a8f4fb0455636dda4d7688ea0d7137117cfecc01d588bf21: Status 404 returned error can't find the container with id 346de427586bad84a8f4fb0455636dda4d7688ea0d7137117cfecc01d588bf21 Oct 30 13:48:12.710821 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:12.710801 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a5f2fbe_141f_4beb_a5b5_0e2ba4686203.slice/crio-887b88c8ae1bce1a3eec5e3dcaa7467828bda7dbf78bab1e48452092bdc05f6c WatchSource:0}: Error finding container 887b88c8ae1bce1a3eec5e3dcaa7467828bda7dbf78bab1e48452092bdc05f6c: Status 404 returned error can't find the container with id 887b88c8ae1bce1a3eec5e3dcaa7467828bda7dbf78bab1e48452092bdc05f6c Oct 30 13:48:12.710898 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.710861 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:12.720498 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:12.720479 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaca47fe8_7068_4743_b6ab_283ee0874f66.slice/crio-08647e20db8ce95dd4fb26e2abb25c8b6667307bd738215b3b50be2d36b4e463 WatchSource:0}: Error finding container 08647e20db8ce95dd4fb26e2abb25c8b6667307bd738215b3b50be2d36b4e463: Status 404 returned error can't find the container with id 08647e20db8ce95dd4fb26e2abb25c8b6667307bd738215b3b50be2d36b4e463 Oct 30 13:48:12.791779 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:12.791719 2580 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 13:48:13.127748 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:13.127075 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b5e74da-5460-44ed-a599-b01620b64c92-original-pull-secret\") pod \"global-pull-secret-syncer-bdmgn\" (UID: \"6b5e74da-5460-44ed-a599-b01620b64c92\") " pod="kube-system/global-pull-secret-syncer-bdmgn" Oct 30 13:48:13.127748 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:13.127348 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb8hg\" (UniqueName: \"kubernetes.io/projected/bf73153f-1421-44d6-abae-b76c3d90235e-kube-api-access-mb8hg\") pod \"network-check-target-dx94q\" (UID: \"bf73153f-1421-44d6-abae-b76c3d90235e\") " pod="openshift-network-diagnostics/network-check-target-dx94q" Oct 30 13:48:13.127748 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:13.127247 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 13:48:13.127748 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:13.127389 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da694c74-3bce-4d95-bd30-12a1891ad5e6-metrics-certs\") pod \"network-metrics-daemon-pfg7x\" (UID: \"da694c74-3bce-4d95-bd30-12a1891ad5e6\") " pod="openshift-multus/network-metrics-daemon-pfg7x" Oct 30 13:48:13.127748 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:13.127447 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b5e74da-5460-44ed-a599-b01620b64c92-original-pull-secret podName:6b5e74da-5460-44ed-a599-b01620b64c92 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:14.127424027 +0000 UTC m=+4.146945417 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6b5e74da-5460-44ed-a599-b01620b64c92-original-pull-secret") pod "global-pull-secret-syncer-bdmgn" (UID: "6b5e74da-5460-44ed-a599-b01620b64c92") : object "kube-system"/"original-pull-secret" not registered Oct 30 13:48:13.127748 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:13.127514 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 13:48:13.127748 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:13.127561 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da694c74-3bce-4d95-bd30-12a1891ad5e6-metrics-certs podName:da694c74-3bce-4d95-bd30-12a1891ad5e6 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:14.127546094 +0000 UTC m=+4.147067477 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/da694c74-3bce-4d95-bd30-12a1891ad5e6-metrics-certs") pod "network-metrics-daemon-pfg7x" (UID: "da694c74-3bce-4d95-bd30-12a1891ad5e6") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 13:48:13.127748 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:13.127657 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 13:48:13.127748 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:13.127671 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 13:48:13.127748 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:13.127683 2580 projected.go:196] Error preparing data for projected volume kube-api-access-mb8hg for pod openshift-network-diagnostics/network-check-target-dx94q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 13:48:13.127748 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:13.127717 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf73153f-1421-44d6-abae-b76c3d90235e-kube-api-access-mb8hg podName:bf73153f-1421-44d6-abae-b76c3d90235e nodeName:}" failed. No retries permitted until 2026-10-30 13:48:14.127706022 +0000 UTC m=+4.147227414 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mb8hg" (UniqueName: "kubernetes.io/projected/bf73153f-1421-44d6-abae-b76c3d90235e-kube-api-access-mb8hg") pod "network-check-target-dx94q" (UID: "bf73153f-1421-44d6-abae-b76c3d90235e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 13:48:13.382332 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:13.382208 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 13:43:12 +0000 UTC" deadline="2028-04-04 15:49:06.327888809 +0000 UTC" Oct 30 13:48:13.382332 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:13.382265 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12530h0m52.945627111s" Oct 30 13:48:13.440034 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:13.440003 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dx94q" Oct 30 13:48:13.440164 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:13.440113 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dx94q" podUID="bf73153f-1421-44d6-abae-b76c3d90235e" Oct 30 13:48:13.440520 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:13.440501 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfg7x" Oct 30 13:48:13.440631 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:13.440612 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pfg7x" podUID="da694c74-3bce-4d95-bd30-12a1891ad5e6" Oct 30 13:48:13.476587 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:13.476506 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-94cdn" event={"ID":"0dda4c8a-827d-428f-a6a2-5e810210662b","Type":"ContainerStarted","Data":"347f2dc700ce6b660126fb5349d4f0be1ff9e5b6c7f0371edad033f971a5ee6e"} Oct 30 13:48:13.483557 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:13.483530 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-llx79" event={"ID":"c06b4b47-6141-4263-94b3-803d9f28bb94","Type":"ContainerStarted","Data":"119282a48a7da81396d040950eab5be9eed0d50580fde28f1b5714cf9ac5d7f4"} Oct 30 13:48:13.497704 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:13.497609 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qf6mp" event={"ID":"d38c246c-86f8-4b57-89bd-db4cd0d3f6d1","Type":"ContainerStarted","Data":"acfa03e8884898362b46b545016cc54a3e8c6767802f494b08ccd9e8d9a7b45e"} Oct 30 13:48:13.503075 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:13.502989 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" event={"ID":"aca47fe8-7068-4743-b6ab-283ee0874f66","Type":"ContainerStarted","Data":"08647e20db8ce95dd4fb26e2abb25c8b6667307bd738215b3b50be2d36b4e463"} Oct 30 13:48:13.512439 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:13.512388 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-b622r" event={"ID":"ad85cf2a-6c44-4e85-8677-8b07cb9f78a4","Type":"ContainerStarted","Data":"05342334dd6693d001bf65221626386848cbec0f12347b577c8df568ccf4ef04"} Oct 30 13:48:13.518045 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:13.518019 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rvt5b" event={"ID":"a85af541-54ed-4d24-b4c5-d69244396157","Type":"ContainerStarted","Data":"1c3374fc75c25c90d2209a22f831ad738912d8e0e51227dafebb8818b4a6ea8d"} Oct 30 13:48:13.535223 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:13.535196 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bjswt" event={"ID":"e9a035cc-e78b-4bdb-b0c6-ef13bc07249a","Type":"ContainerStarted","Data":"d59463f528816d4b59dd570cc059bb929537bcfab0c33399f92624e5bb8edc20"} Oct 30 13:48:13.549855 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:13.549830 2580 generic.go:356] "Generic (PLEG): container finished" podID="18c4e529019bfc1529902a8bff559ebf" containerID="9b348ccc05a60fc27fd2d38cb5b50d635360ecbf5b23aef06c8e01b4cf5f3a98" exitCode=0 Oct 30 13:48:13.549960 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:13.549911 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-105.ec2.internal" event={"ID":"18c4e529019bfc1529902a8bff559ebf","Type":"ContainerDied","Data":"9b348ccc05a60fc27fd2d38cb5b50d635360ecbf5b23aef06c8e01b4cf5f3a98"} Oct 30 13:48:13.561977 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:13.561534 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-105.ec2.internal" event={"ID":"7f6d989ed16482ea69070d9acfe2e5f0","Type":"ContainerStarted","Data":"373a818ad40af0f606177e9696274dc40302e998ad58f78f704925d34417e144"} Oct 30 13:48:13.568845 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:13.568814 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tqst8" event={"ID":"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203","Type":"ContainerStarted","Data":"887b88c8ae1bce1a3eec5e3dcaa7467828bda7dbf78bab1e48452092bdc05f6c"} Oct 30 13:48:13.572267 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:13.572247 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9bxl" event={"ID":"06f40dde-8b71-41e0-9acb-cede4f1d4347","Type":"ContainerStarted","Data":"346de427586bad84a8f4fb0455636dda4d7688ea0d7137117cfecc01d588bf21"} Oct 30 13:48:13.884195 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:13.884152 2580 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 13:48:14.136700 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:14.136666 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da694c74-3bce-4d95-bd30-12a1891ad5e6-metrics-certs\") pod \"network-metrics-daemon-pfg7x\" (UID: \"da694c74-3bce-4d95-bd30-12a1891ad5e6\") " pod="openshift-multus/network-metrics-daemon-pfg7x" Oct 30 13:48:14.136847 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:14.136741 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b5e74da-5460-44ed-a599-b01620b64c92-original-pull-secret\") pod \"global-pull-secret-syncer-bdmgn\" (UID: \"6b5e74da-5460-44ed-a599-b01620b64c92\") " pod="kube-system/global-pull-secret-syncer-bdmgn" Oct 30 13:48:14.136847 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:14.136784 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb8hg\" (UniqueName: \"kubernetes.io/projected/bf73153f-1421-44d6-abae-b76c3d90235e-kube-api-access-mb8hg\") pod \"network-check-target-dx94q\" (UID: \"bf73153f-1421-44d6-abae-b76c3d90235e\") " pod="openshift-network-diagnostics/network-check-target-dx94q" Oct 30 13:48:14.136964 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:14.136954 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 13:48:14.137231 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:14.136976 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 13:48:14.137231 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:14.136988 2580 projected.go:196] Error preparing data for projected volume kube-api-access-mb8hg for pod openshift-network-diagnostics/network-check-target-dx94q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 13:48:14.137231 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:14.137037 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf73153f-1421-44d6-abae-b76c3d90235e-kube-api-access-mb8hg podName:bf73153f-1421-44d6-abae-b76c3d90235e nodeName:}" failed. No retries permitted until 2026-10-30 13:48:16.137020956 +0000 UTC m=+6.156542342 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mb8hg" (UniqueName: "kubernetes.io/projected/bf73153f-1421-44d6-abae-b76c3d90235e-kube-api-access-mb8hg") pod "network-check-target-dx94q" (UID: "bf73153f-1421-44d6-abae-b76c3d90235e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 13:48:14.137459 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:14.137436 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 13:48:14.137516 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:14.137495 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da694c74-3bce-4d95-bd30-12a1891ad5e6-metrics-certs podName:da694c74-3bce-4d95-bd30-12a1891ad5e6 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:16.137479838 +0000 UTC m=+6.157001221 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/da694c74-3bce-4d95-bd30-12a1891ad5e6-metrics-certs") pod "network-metrics-daemon-pfg7x" (UID: "da694c74-3bce-4d95-bd30-12a1891ad5e6") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 13:48:14.137607 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:14.137569 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 13:48:14.137607 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:14.137605 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b5e74da-5460-44ed-a599-b01620b64c92-original-pull-secret podName:6b5e74da-5460-44ed-a599-b01620b64c92 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:16.137594469 +0000 UTC m=+6.157115863 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6b5e74da-5460-44ed-a599-b01620b64c92-original-pull-secret") pod "global-pull-secret-syncer-bdmgn" (UID: "6b5e74da-5460-44ed-a599-b01620b64c92") : object "kube-system"/"original-pull-secret" not registered Oct 30 13:48:14.383067 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:14.383026 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 13:43:12 +0000 UTC" deadline="2028-08-11 14:05:58.392534422 +0000 UTC" Oct 30 13:48:14.383067 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:14.383065 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15624h17m44.009473241s" Oct 30 13:48:14.440652 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:14.440220 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdmgn" Oct 30 13:48:14.440652 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:14.440318 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdmgn" podUID="6b5e74da-5460-44ed-a599-b01620b64c92" Oct 30 13:48:14.585692 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:14.585610 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-105.ec2.internal" event={"ID":"18c4e529019bfc1529902a8bff559ebf","Type":"ContainerStarted","Data":"9685a3b604fe44912c985b9919798a50746a4f2d5cfd957adb49525a10338ff4"} Oct 30 13:48:14.627593 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:14.626885 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-105.ec2.internal" podStartSLOduration=3.626872243 podStartE2EDuration="3.626872243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:48:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:13.638338417 +0000 UTC m=+3.657859818" watchObservedRunningTime="2026-10-30 13:48:14.626872243 +0000 UTC m=+4.646393642" Oct 30 13:48:15.440186 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:15.440135 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfg7x" Oct 30 13:48:15.440608 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:15.440265 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pfg7x" podUID="da694c74-3bce-4d95-bd30-12a1891ad5e6" Oct 30 13:48:15.440767 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:15.440653 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dx94q" Oct 30 13:48:15.440767 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:15.440744 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dx94q" podUID="bf73153f-1421-44d6-abae-b76c3d90235e" Oct 30 13:48:15.597845 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:15.597811 2580 generic.go:356] "Generic (PLEG): container finished" podID="0dda4c8a-827d-428f-a6a2-5e810210662b" containerID="2fdc6ec6c6a17d45c7745fd442c8f3c89921da01a9c4577bc4d57b92d1ba5194" exitCode=0 Oct 30 13:48:15.598840 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:15.598813 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-94cdn" event={"ID":"0dda4c8a-827d-428f-a6a2-5e810210662b","Type":"ContainerDied","Data":"2fdc6ec6c6a17d45c7745fd442c8f3c89921da01a9c4577bc4d57b92d1ba5194"} Oct 30 13:48:15.671739 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:15.671427 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-105.ec2.internal" podStartSLOduration=4.671407129 podStartE2EDuration="4.671407129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:48:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:14.627542656 +0000 UTC m=+4.647064057" watchObservedRunningTime="2026-10-30 13:48:15.671407129 +0000 UTC m=+5.690928532" Oct 30 13:48:16.155252 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:16.154262 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb8hg\" (UniqueName: \"kubernetes.io/projected/bf73153f-1421-44d6-abae-b76c3d90235e-kube-api-access-mb8hg\") pod \"network-check-target-dx94q\" (UID: \"bf73153f-1421-44d6-abae-b76c3d90235e\") " pod="openshift-network-diagnostics/network-check-target-dx94q" Oct 30 13:48:16.155252 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:16.154390 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da694c74-3bce-4d95-bd30-12a1891ad5e6-metrics-certs\") pod \"network-metrics-daemon-pfg7x\" (UID: \"da694c74-3bce-4d95-bd30-12a1891ad5e6\") " pod="openshift-multus/network-metrics-daemon-pfg7x" Oct 30 13:48:16.155252 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:16.154439 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b5e74da-5460-44ed-a599-b01620b64c92-original-pull-secret\") pod \"global-pull-secret-syncer-bdmgn\" (UID: \"6b5e74da-5460-44ed-a599-b01620b64c92\") " pod="kube-system/global-pull-secret-syncer-bdmgn" Oct 30 13:48:16.155252 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:16.154562 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 13:48:16.155252 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:16.154621 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b5e74da-5460-44ed-a599-b01620b64c92-original-pull-secret podName:6b5e74da-5460-44ed-a599-b01620b64c92 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:20.15460236 +0000 UTC m=+10.174123752 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6b5e74da-5460-44ed-a599-b01620b64c92-original-pull-secret") pod "global-pull-secret-syncer-bdmgn" (UID: "6b5e74da-5460-44ed-a599-b01620b64c92") : object "kube-system"/"original-pull-secret" not registered Oct 30 13:48:16.155252 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:16.155057 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 13:48:16.155252 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:16.155075 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 13:48:16.155252 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:16.155086 2580 projected.go:196] Error preparing data for projected volume kube-api-access-mb8hg for pod openshift-network-diagnostics/network-check-target-dx94q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 13:48:16.155252 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:16.155130 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf73153f-1421-44d6-abae-b76c3d90235e-kube-api-access-mb8hg podName:bf73153f-1421-44d6-abae-b76c3d90235e nodeName:}" failed. No retries permitted until 2026-10-30 13:48:20.155115023 +0000 UTC m=+10.174636415 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mb8hg" (UniqueName: "kubernetes.io/projected/bf73153f-1421-44d6-abae-b76c3d90235e-kube-api-access-mb8hg") pod "network-check-target-dx94q" (UID: "bf73153f-1421-44d6-abae-b76c3d90235e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 13:48:16.155252 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:16.155187 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 13:48:16.155252 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:16.155220 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da694c74-3bce-4d95-bd30-12a1891ad5e6-metrics-certs podName:da694c74-3bce-4d95-bd30-12a1891ad5e6 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:20.155209063 +0000 UTC m=+10.174730454 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/da694c74-3bce-4d95-bd30-12a1891ad5e6-metrics-certs") pod "network-metrics-daemon-pfg7x" (UID: "da694c74-3bce-4d95-bd30-12a1891ad5e6") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 13:48:16.441208 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:16.440756 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdmgn" Oct 30 13:48:16.441208 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:16.440879 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdmgn" podUID="6b5e74da-5460-44ed-a599-b01620b64c92" Oct 30 13:48:16.604666 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:16.604541 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-b622r" event={"ID":"ad85cf2a-6c44-4e85-8677-8b07cb9f78a4","Type":"ContainerStarted","Data":"b242cded6509da10109eb6b313b3ba0d55a3b9c89bb9d7eeda47f32d57a76535"} Oct 30 13:48:16.648126 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:16.648073 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-b622r" podStartSLOduration=3.503389514 podStartE2EDuration="6.64805477s" totalImagesPullingTime="3.144665256s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:48:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:16.647943967 +0000 UTC m=+6.667465368" watchObservedRunningTime="2026-10-30 13:48:16.64805477 +0000 UTC m=+6.667576170" Oct 30 13:48:17.440035 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:17.440015 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dx94q" Oct 30 13:48:17.440150 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:17.440112 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dx94q" podUID="bf73153f-1421-44d6-abae-b76c3d90235e" Oct 30 13:48:17.440207 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:17.440188 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfg7x" Oct 30 13:48:17.440281 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:17.440264 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pfg7x" podUID="da694c74-3bce-4d95-bd30-12a1891ad5e6" Oct 30 13:48:18.441863 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:18.441755 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdmgn" Oct 30 13:48:18.442322 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:18.441971 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdmgn" podUID="6b5e74da-5460-44ed-a599-b01620b64c92" Oct 30 13:48:18.611209 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:18.611121 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9bxl" event={"ID":"06f40dde-8b71-41e0-9acb-cede4f1d4347","Type":"ContainerStarted","Data":"6ff02a5a3f3dd88f6a3b88e65e2e1d160a0b8ae8aec3c621955aeffd151c71d3"} Oct 30 13:48:18.614094 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:18.613374 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-llx79" event={"ID":"c06b4b47-6141-4263-94b3-803d9f28bb94","Type":"ContainerStarted","Data":"b4df51ac322e18d52c8d523ec877cfb6e4e2094b246613020d0d31ee3f6d33e2"} Oct 30 13:48:19.439909 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:19.439877 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dx94q" Oct 30 13:48:19.440076 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:19.440032 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dx94q" podUID="bf73153f-1421-44d6-abae-b76c3d90235e" Oct 30 13:48:19.440508 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:19.440490 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfg7x" Oct 30 13:48:19.440653 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:19.440622 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pfg7x" podUID="da694c74-3bce-4d95-bd30-12a1891ad5e6" Oct 30 13:48:19.499408 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:19.499217 2580 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 13:48:19.619048 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:19.618985 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9bxl" event={"ID":"06f40dde-8b71-41e0-9acb-cede4f1d4347","Type":"ContainerStarted","Data":"51b4b93eb5492b5f93255d976913d88495ee189a67d7a0e4556d3e459eff9794"} Oct 30 13:48:20.187391 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:20.187350 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b5e74da-5460-44ed-a599-b01620b64c92-original-pull-secret\") pod \"global-pull-secret-syncer-bdmgn\" (UID: \"6b5e74da-5460-44ed-a599-b01620b64c92\") " pod="kube-system/global-pull-secret-syncer-bdmgn" Oct 30 13:48:20.187530 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:20.187407 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb8hg\" (UniqueName: \"kubernetes.io/projected/bf73153f-1421-44d6-abae-b76c3d90235e-kube-api-access-mb8hg\") pod \"network-check-target-dx94q\" (UID: \"bf73153f-1421-44d6-abae-b76c3d90235e\") " pod="openshift-network-diagnostics/network-check-target-dx94q" Oct 30 13:48:20.187530 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:20.187442 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da694c74-3bce-4d95-bd30-12a1891ad5e6-metrics-certs\") pod \"network-metrics-daemon-pfg7x\" (UID: \"da694c74-3bce-4d95-bd30-12a1891ad5e6\") " pod="openshift-multus/network-metrics-daemon-pfg7x" Oct 30 13:48:20.187673 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:20.187580 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 13:48:20.187673 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:20.187652 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da694c74-3bce-4d95-bd30-12a1891ad5e6-metrics-certs podName:da694c74-3bce-4d95-bd30-12a1891ad5e6 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:28.187621424 +0000 UTC m=+18.207142820 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/da694c74-3bce-4d95-bd30-12a1891ad5e6-metrics-certs") pod "network-metrics-daemon-pfg7x" (UID: "da694c74-3bce-4d95-bd30-12a1891ad5e6") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 13:48:20.187795 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:20.187711 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 13:48:20.187795 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:20.187745 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b5e74da-5460-44ed-a599-b01620b64c92-original-pull-secret podName:6b5e74da-5460-44ed-a599-b01620b64c92 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:28.187734491 +0000 UTC m=+18.207255883 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6b5e74da-5460-44ed-a599-b01620b64c92-original-pull-secret") pod "global-pull-secret-syncer-bdmgn" (UID: "6b5e74da-5460-44ed-a599-b01620b64c92") : object "kube-system"/"original-pull-secret" not registered Oct 30 13:48:20.187896 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:20.187825 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 13:48:20.187896 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:20.187839 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 13:48:20.187896 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:20.187851 2580 projected.go:196] Error preparing data for projected volume kube-api-access-mb8hg for pod openshift-network-diagnostics/network-check-target-dx94q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 13:48:20.187896 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:20.187881 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf73153f-1421-44d6-abae-b76c3d90235e-kube-api-access-mb8hg podName:bf73153f-1421-44d6-abae-b76c3d90235e nodeName:}" failed. No retries permitted until 2026-10-30 13:48:28.187871183 +0000 UTC m=+18.207392578 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mb8hg" (UniqueName: "kubernetes.io/projected/bf73153f-1421-44d6-abae-b76c3d90235e-kube-api-access-mb8hg") pod "network-check-target-dx94q" (UID: "bf73153f-1421-44d6-abae-b76c3d90235e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 13:48:20.380810 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:20.380524 2580 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T13:48:19.499236139Z","UUID":"8a0a84b1-10e2-427c-a1db-e24693008b0d","Handler":null,"Name":"","Endpoint":""} Oct 30 13:48:20.382501 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:20.382477 2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 13:48:20.382501 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:20.382502 2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 13:48:20.441345 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:20.440865 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdmgn" Oct 30 13:48:20.441345 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:20.440979 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdmgn" podUID="6b5e74da-5460-44ed-a599-b01620b64c92" Oct 30 13:48:20.622346 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:20.622303 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rvt5b" event={"ID":"a85af541-54ed-4d24-b4c5-d69244396157","Type":"ContainerStarted","Data":"84825d73dd30be0062c48d0196cb94caa75556d660370da4472ba5f0ec82edbe"} Oct 30 13:48:20.624115 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:20.624087 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bjswt" event={"ID":"e9a035cc-e78b-4bdb-b0c6-ef13bc07249a","Type":"ContainerStarted","Data":"b095a429cb0ec3b70471fc58b494aeacb20f61c6e7426c5f75b6b425566f1f44"} Oct 30 13:48:20.626812 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:20.626785 2580 generic.go:356] "Generic (PLEG): container finished" podID="0dda4c8a-827d-428f-a6a2-5e810210662b" containerID="54ed10d44823e3a86c29639af52022900c5ad5e88201a263375aa69610464b3f" exitCode=0 Oct 30 13:48:20.626906 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:20.626842 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-94cdn" event={"ID":"0dda4c8a-827d-428f-a6a2-5e810210662b","Type":"ContainerDied","Data":"54ed10d44823e3a86c29639af52022900c5ad5e88201a263375aa69610464b3f"} Oct 30 13:48:20.669448 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:20.666672 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-llx79" podStartSLOduration=5.920154679 podStartE2EDuration="10.666653422s" totalImagesPullingTime="4.746498743s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:48:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:18.654918419 +0000 UTC m=+8.674439820" watchObservedRunningTime="2026-10-30 13:48:20.666653422 +0000 UTC m=+10.686174820" Oct 30 13:48:20.669448 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:20.666795 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-rvt5b" podStartSLOduration=3.46885236 podStartE2EDuration="10.666789361s" totalImagesPullingTime="7.197937001s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:48:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:20.665921434 +0000 UTC m=+10.685442835" watchObservedRunningTime="2026-10-30 13:48:20.666789361 +0000 UTC m=+10.686310761" Oct 30 13:48:20.751048 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:20.750964 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-bjswt" podStartSLOduration=3.272418642 podStartE2EDuration="10.750946632s" totalImagesPullingTime="7.47852799s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:48:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:20.750387505 +0000 UTC m=+10.769908907" watchObservedRunningTime="2026-10-30 13:48:20.750946632 +0000 UTC m=+10.770468036" Oct 30 13:48:21.432779 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:21.432751 2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-b622r" Oct 30 13:48:21.433189 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:21.433173 2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-b622r" Oct 30 13:48:21.440324 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:21.440301 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dx94q" Oct 30 13:48:21.440480 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:21.440453 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dx94q" podUID="bf73153f-1421-44d6-abae-b76c3d90235e" Oct 30 13:48:21.440768 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:21.440749 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfg7x" Oct 30 13:48:21.440975 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:21.440834 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pfg7x" podUID="da694c74-3bce-4d95-bd30-12a1891ad5e6" Oct 30 13:48:21.632405 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:21.632324 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9bxl" event={"ID":"06f40dde-8b71-41e0-9acb-cede4f1d4347","Type":"ContainerStarted","Data":"023e75be4ae744f0d9f868924c1623193374a71e9f4c14fdb6178fc72fe91a93"} Oct 30 13:48:21.634940 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:21.634912 2580 generic.go:356] "Generic (PLEG): container finished" podID="0dda4c8a-827d-428f-a6a2-5e810210662b" containerID="5c8d71f8e893bf7624b4b799e3b9279d85c5ffdbaa72b569f690327f4cf272c0" exitCode=0 Oct 30 13:48:21.635066 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:21.635006 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-94cdn" event={"ID":"0dda4c8a-827d-428f-a6a2-5e810210662b","Type":"ContainerDied","Data":"5c8d71f8e893bf7624b4b799e3b9279d85c5ffdbaa72b569f690327f4cf272c0"} Oct 30 13:48:21.674395 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:21.674342 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9bxl" podStartSLOduration=4.221443297 podStartE2EDuration="11.674327833s" totalImagesPullingTime="7.452884536s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:48:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:21.674310668 +0000 UTC m=+11.693832081" watchObservedRunningTime="2026-10-30 13:48:21.674327833 +0000 UTC m=+11.693849234" Oct 30 13:48:22.440534 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:22.440113 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdmgn" Oct 30 13:48:22.440534 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:22.440245 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdmgn" podUID="6b5e74da-5460-44ed-a599-b01620b64c92" Oct 30 13:48:22.638977 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:22.638941 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tqst8" event={"ID":"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203","Type":"ContainerStarted","Data":"470c09a5674884cc6bcfcad517087ef4a75f71efefd9ebfc21118d319604a7a2"} Oct 30 13:48:22.640543 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:22.640511 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qf6mp" event={"ID":"d38c246c-86f8-4b57-89bd-db4cd0d3f6d1","Type":"ContainerStarted","Data":"ce4296f270caf84c58189eb41cc34891a806e21a2ce188f0c66cd3e09aa2720d"} Oct 30 13:48:22.681656 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:22.681598 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-tqst8" podStartSLOduration=2.9302728289999997 podStartE2EDuration="12.681584153s" totalImagesPullingTime="9.751311324s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:48:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:22.680936987 +0000 UTC m=+12.700458511" watchObservedRunningTime="2026-10-30 13:48:22.681584153 +0000 UTC m=+12.701105551" Oct 30 13:48:22.729270 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:22.729199 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-qf6mp" podStartSLOduration=5.162255343 podStartE2EDuration="12.729187665s" totalImagesPullingTime="7.566932322s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:48:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:22.728954348 +0000 UTC m=+12.748475749" watchObservedRunningTime="2026-10-30 13:48:22.729187665 +0000 UTC m=+12.748709065" Oct 30 13:48:23.440271 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:23.440240 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfg7x" Oct 30 13:48:23.440436 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:23.440353 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pfg7x" podUID="da694c74-3bce-4d95-bd30-12a1891ad5e6" Oct 30 13:48:23.440436 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:23.440415 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dx94q" Oct 30 13:48:23.440550 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:23.440506 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dx94q" podUID="bf73153f-1421-44d6-abae-b76c3d90235e" Oct 30 13:48:23.615532 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:23.615504 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-b622r" Oct 30 13:48:23.615727 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:23.615629 2580 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 13:48:23.616319 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:23.616289 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-b622r" Oct 30 13:48:23.644839 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:23.644817 2580 generic.go:356] "Generic (PLEG): container finished" podID="0dda4c8a-827d-428f-a6a2-5e810210662b" containerID="44b719f9964236c03a9d2376a362899e228ac68027da40d28da79315f475e998" exitCode=0 Oct 30 13:48:23.645247 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:23.644908 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-94cdn" event={"ID":"0dda4c8a-827d-428f-a6a2-5e810210662b","Type":"ContainerDied","Data":"44b719f9964236c03a9d2376a362899e228ac68027da40d28da79315f475e998"} Oct 30 13:48:24.442315 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:24.440419 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdmgn" Oct 30 13:48:24.442315 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:24.440533 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdmgn" podUID="6b5e74da-5460-44ed-a599-b01620b64c92" Oct 30 13:48:25.440599 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:25.440397 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfg7x" Oct 30 13:48:25.441024 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:25.440441 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dx94q" Oct 30 13:48:25.441024 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:25.440736 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pfg7x" podUID="da694c74-3bce-4d95-bd30-12a1891ad5e6" Oct 30 13:48:25.441024 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:25.440857 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dx94q" podUID="bf73153f-1421-44d6-abae-b76c3d90235e" Oct 30 13:48:25.652549 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:25.652431 2580 generic.go:356] "Generic (PLEG): container finished" podID="0dda4c8a-827d-428f-a6a2-5e810210662b" containerID="c3ec9ef7c73e9f870bf3caa22448c73c3073cc854248a974ffe2e795228e3781" exitCode=0 Oct 30 13:48:25.652549 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:25.652478 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-94cdn" event={"ID":"0dda4c8a-827d-428f-a6a2-5e810210662b","Type":"ContainerDied","Data":"c3ec9ef7c73e9f870bf3caa22448c73c3073cc854248a974ffe2e795228e3781"} Oct 30 13:48:26.440098 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:26.440075 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdmgn" Oct 30 13:48:26.440193 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:26.440177 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdmgn" podUID="6b5e74da-5460-44ed-a599-b01620b64c92" Oct 30 13:48:26.658285 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:26.658082 2580 generic.go:356] "Generic (PLEG): container finished" podID="0dda4c8a-827d-428f-a6a2-5e810210662b" containerID="85e5d87cab66a35b7dbf08ab96bc34e3f943ff7f6e478882ba29c0a44f7b0fb8" exitCode=0 Oct 30 13:48:26.658676 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:26.658165 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-94cdn" event={"ID":"0dda4c8a-827d-428f-a6a2-5e810210662b","Type":"ContainerDied","Data":"85e5d87cab66a35b7dbf08ab96bc34e3f943ff7f6e478882ba29c0a44f7b0fb8"} Oct 30 13:48:26.661023 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:26.661003 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhxjf_aca47fe8-7068-4743-b6ab-283ee0874f66/ovn-acl-logging/0.log" Oct 30 13:48:26.661354 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:26.661333 2580 generic.go:356] "Generic (PLEG): container finished" podID="aca47fe8-7068-4743-b6ab-283ee0874f66" containerID="8ae5eba6d805209a9813e5b8cb249e019de8c8ff2a821150567b953c36724d38" exitCode=1 Oct 30 13:48:26.661419 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:26.661373 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" event={"ID":"aca47fe8-7068-4743-b6ab-283ee0874f66","Type":"ContainerStarted","Data":"06054c8cbc21480e6e44ce8831209ee9f0d93e432f65acbc57360fe09095bf9d"} Oct 30 13:48:26.661455 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:26.661439 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" event={"ID":"aca47fe8-7068-4743-b6ab-283ee0874f66","Type":"ContainerStarted","Data":"2512585dbf6f87c9b427a90627898be182edd46d10e3af6a5ef0a540cc5bffa8"} Oct 30 13:48:26.661490 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:26.661482 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" event={"ID":"aca47fe8-7068-4743-b6ab-283ee0874f66","Type":"ContainerStarted","Data":"7c55effe456b580d337434994fbb2fab291bf63d2e2e7c2acb6449dc5ec3706e"} Oct 30 13:48:26.661538 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:26.661526 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" event={"ID":"aca47fe8-7068-4743-b6ab-283ee0874f66","Type":"ContainerStarted","Data":"0d4dff6f00fae5aa1a9d150bd43c93a2701d6065a2d4436e36ac77c2d2f2903f"} Oct 30 13:48:26.661584 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:26.661573 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" event={"ID":"aca47fe8-7068-4743-b6ab-283ee0874f66","Type":"ContainerDied","Data":"8ae5eba6d805209a9813e5b8cb249e019de8c8ff2a821150567b953c36724d38"} Oct 30 13:48:26.661632 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:26.661620 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" event={"ID":"aca47fe8-7068-4743-b6ab-283ee0874f66","Type":"ContainerStarted","Data":"6a6032fed815a0715b4991180a309591c8e0d21e7fb97b148feadc2cbb01ef3e"} Oct 30 13:48:27.439422 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:27.439395 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfg7x" Oct 30 13:48:27.439527 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:27.439511 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pfg7x" podUID="da694c74-3bce-4d95-bd30-12a1891ad5e6" Oct 30 13:48:27.439577 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:27.439567 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dx94q" Oct 30 13:48:27.439660 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:27.439631 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dx94q" podUID="bf73153f-1421-44d6-abae-b76c3d90235e" Oct 30 13:48:27.666144 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:27.666121 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-94cdn" event={"ID":"0dda4c8a-827d-428f-a6a2-5e810210662b","Type":"ContainerStarted","Data":"09da0f405b1b563ef66d9728bd628d51f438c0a0483b12bb7004e1d9fb4ad8e3"} Oct 30 13:48:27.743508 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:27.743447 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-94cdn" podStartSLOduration=7.50010162 podStartE2EDuration="17.743435749s" totalImagesPullingTime="10.243334129s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:48:10 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:27.743092113 +0000 UTC m=+17.762613513" watchObservedRunningTime="2026-10-30 13:48:27.743435749 +0000 UTC m=+17.762957148" Oct 30 13:48:28.245488 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:28.245453 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b5e74da-5460-44ed-a599-b01620b64c92-original-pull-secret\") pod \"global-pull-secret-syncer-bdmgn\" (UID: \"6b5e74da-5460-44ed-a599-b01620b64c92\") " pod="kube-system/global-pull-secret-syncer-bdmgn" Oct 30 13:48:28.245488 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:28.245495 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb8hg\" (UniqueName: \"kubernetes.io/projected/bf73153f-1421-44d6-abae-b76c3d90235e-kube-api-access-mb8hg\") pod \"network-check-target-dx94q\" (UID: \"bf73153f-1421-44d6-abae-b76c3d90235e\") " pod="openshift-network-diagnostics/network-check-target-dx94q" Oct 30 13:48:28.245707 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:28.245517 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da694c74-3bce-4d95-bd30-12a1891ad5e6-metrics-certs\") pod \"network-metrics-daemon-pfg7x\" (UID: \"da694c74-3bce-4d95-bd30-12a1891ad5e6\") " pod="openshift-multus/network-metrics-daemon-pfg7x" Oct 30 13:48:28.245707 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:28.245584 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 13:48:28.245707 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:28.245597 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 13:48:28.245707 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:28.245613 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 13:48:28.245707 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:28.245622 2580 projected.go:196] Error preparing data for projected volume kube-api-access-mb8hg for pod openshift-network-diagnostics/network-check-target-dx94q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 13:48:28.245707 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:28.245652 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b5e74da-5460-44ed-a599-b01620b64c92-original-pull-secret podName:6b5e74da-5460-44ed-a599-b01620b64c92 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:44.245621574 +0000 UTC m=+34.265142966 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6b5e74da-5460-44ed-a599-b01620b64c92-original-pull-secret") pod "global-pull-secret-syncer-bdmgn" (UID: "6b5e74da-5460-44ed-a599-b01620b64c92") : object "kube-system"/"original-pull-secret" not registered Oct 30 13:48:28.245707 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:28.245588 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 13:48:28.245707 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:28.245673 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf73153f-1421-44d6-abae-b76c3d90235e-kube-api-access-mb8hg podName:bf73153f-1421-44d6-abae-b76c3d90235e nodeName:}" failed. No retries permitted until 2026-10-30 13:48:44.245658608 +0000 UTC m=+34.265179986 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mb8hg" (UniqueName: "kubernetes.io/projected/bf73153f-1421-44d6-abae-b76c3d90235e-kube-api-access-mb8hg") pod "network-check-target-dx94q" (UID: "bf73153f-1421-44d6-abae-b76c3d90235e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 13:48:28.245707 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:28.245705 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da694c74-3bce-4d95-bd30-12a1891ad5e6-metrics-certs podName:da694c74-3bce-4d95-bd30-12a1891ad5e6 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:44.245692433 +0000 UTC m=+34.265213827 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/da694c74-3bce-4d95-bd30-12a1891ad5e6-metrics-certs") pod "network-metrics-daemon-pfg7x" (UID: "da694c74-3bce-4d95-bd30-12a1891ad5e6") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 13:48:28.442157 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:28.442133 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdmgn" Oct 30 13:48:28.442292 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:28.442213 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdmgn" podUID="6b5e74da-5460-44ed-a599-b01620b64c92" Oct 30 13:48:28.671931 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:28.671911 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhxjf_aca47fe8-7068-4743-b6ab-283ee0874f66/ovn-acl-logging/0.log" Oct 30 13:48:28.672395 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:28.672372 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" event={"ID":"aca47fe8-7068-4743-b6ab-283ee0874f66","Type":"ContainerStarted","Data":"c01fb1fc77d5f2811b855447fcb2777a2c6c25be8c6b12c037b8977312bd6cb1"} Oct 30 13:48:29.439522 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:29.439497 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfg7x" Oct 30 13:48:29.439656 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:29.439532 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dx94q" Oct 30 13:48:29.439656 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:29.439602 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pfg7x" podUID="da694c74-3bce-4d95-bd30-12a1891ad5e6" Oct 30 13:48:29.439732 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:29.439716 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dx94q" podUID="bf73153f-1421-44d6-abae-b76c3d90235e" Oct 30 13:48:30.442050 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:30.442024 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdmgn" Oct 30 13:48:30.442484 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:30.442101 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdmgn" podUID="6b5e74da-5460-44ed-a599-b01620b64c92" Oct 30 13:48:31.440205 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:31.440038 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfg7x" Oct 30 13:48:31.440353 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:31.440095 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dx94q" Oct 30 13:48:31.440353 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:31.440273 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pfg7x" podUID="da694c74-3bce-4d95-bd30-12a1891ad5e6" Oct 30 13:48:31.440353 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:31.440342 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dx94q" podUID="bf73153f-1421-44d6-abae-b76c3d90235e" Oct 30 13:48:31.680958 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:31.680932 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhxjf_aca47fe8-7068-4743-b6ab-283ee0874f66/ovn-acl-logging/0.log" Oct 30 13:48:31.681413 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:31.681234 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" event={"ID":"aca47fe8-7068-4743-b6ab-283ee0874f66","Type":"ContainerStarted","Data":"061b698193124c393fd39efd073781a0d3c5471495ddd5d0c5f9c2d471c45d65"} Oct 30 13:48:31.681500 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:31.681480 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:31.681629 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:31.681534 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:31.681714 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:31.681626 2580 scope.go:122] "RemoveContainer" containerID="8ae5eba6d805209a9813e5b8cb249e019de8c8ff2a821150567b953c36724d38" Oct 30 13:48:31.696488 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:31.696468 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:31.696615 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:31.696602 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:32.442226 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:32.442189 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdmgn" Oct 30 13:48:32.442426 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:32.442278 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdmgn" podUID="6b5e74da-5460-44ed-a599-b01620b64c92" Oct 30 13:48:32.686997 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:32.686973 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhxjf_aca47fe8-7068-4743-b6ab-283ee0874f66/ovn-acl-logging/0.log" Oct 30 13:48:32.687376 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:32.687344 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" event={"ID":"aca47fe8-7068-4743-b6ab-283ee0874f66","Type":"ContainerStarted","Data":"730166d89ca41790c645cd3ccbcd8f9624a5f98cf69d34465bfff5e36cdee219"} Oct 30 13:48:32.687440 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:32.687428 2580 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 13:48:32.759387 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:32.759313 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" podStartSLOduration=9.284225339 podStartE2EDuration="22.759300849s" totalImagesPullingTime="13.47507551s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:48:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:32.758536276 +0000 UTC m=+22.778057677" watchObservedRunningTime="2026-10-30 13:48:32.759300849 +0000 UTC m=+22.778822249" Oct 30 13:48:32.809068 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:32.809037 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bdmgn"] Oct 30 13:48:32.809194 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:32.809125 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdmgn" Oct 30 13:48:32.809263 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:32.809201 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdmgn" podUID="6b5e74da-5460-44ed-a599-b01620b64c92" Oct 30 13:48:32.812285 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:32.812257 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pfg7x"] Oct 30 13:48:32.812402 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:32.812348 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfg7x" Oct 30 13:48:32.812464 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:32.812451 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pfg7x" podUID="da694c74-3bce-4d95-bd30-12a1891ad5e6" Oct 30 13:48:32.819066 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:32.819045 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dx94q"] Oct 30 13:48:32.819174 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:32.819162 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dx94q" Oct 30 13:48:32.819247 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:32.819231 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dx94q" podUID="bf73153f-1421-44d6-abae-b76c3d90235e" Oct 30 13:48:33.689811 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:33.689790 2580 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 13:48:34.439608 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:34.439579 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfg7x" Oct 30 13:48:34.439608 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:34.439611 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdmgn" Oct 30 13:48:34.439878 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:34.439579 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dx94q" Oct 30 13:48:34.439878 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:34.439693 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pfg7x" podUID="da694c74-3bce-4d95-bd30-12a1891ad5e6" Oct 30 13:48:34.439878 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:34.439749 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdmgn" podUID="6b5e74da-5460-44ed-a599-b01620b64c92" Oct 30 13:48:34.439878 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:34.439813 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dx94q" podUID="bf73153f-1421-44d6-abae-b76c3d90235e" Oct 30 13:48:36.439680 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:36.439653 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdmgn" Oct 30 13:48:36.440103 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:36.439661 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfg7x" Oct 30 13:48:36.440103 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:36.439745 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dx94q" Oct 30 13:48:36.440103 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:36.439746 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdmgn" podUID="6b5e74da-5460-44ed-a599-b01620b64c92" Oct 30 13:48:36.440103 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:36.439817 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pfg7x" podUID="da694c74-3bce-4d95-bd30-12a1891ad5e6" Oct 30 13:48:36.440103 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:36.439873 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dx94q" podUID="bf73153f-1421-44d6-abae-b76c3d90235e" Oct 30 13:48:38.439833 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.439795 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdmgn" Oct 30 13:48:38.440335 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.439838 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dx94q" Oct 30 13:48:38.440335 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.439795 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfg7x" Oct 30 13:48:38.440335 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:38.439936 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdmgn" podUID="6b5e74da-5460-44ed-a599-b01620b64c92" Oct 30 13:48:38.440335 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:38.439988 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pfg7x" podUID="da694c74-3bce-4d95-bd30-12a1891ad5e6" Oct 30 13:48:38.440335 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:38.440044 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dx94q" podUID="bf73153f-1421-44d6-abae-b76c3d90235e" Oct 30 13:48:38.759245 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.759228 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-105.ec2.internal" event="NodeReady" Oct 30 13:48:38.759363 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.759309 2580 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 13:48:38.808310 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.808285 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-4h2sq"] Oct 30 13:48:38.811353 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.811336 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-4h2sq" Oct 30 13:48:38.812224 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.812198 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-smvmg"] Oct 30 13:48:38.815110 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.815095 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-plvgd"] Oct 30 13:48:38.815210 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.815196 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-smvmg" Oct 30 13:48:38.817820 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.817803 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-kcgbg"] Oct 30 13:48:38.817920 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.817903 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-plvgd" Oct 30 13:48:38.820386 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.820372 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-659879fb99-pjvcq"] Oct 30 13:48:38.820472 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.820460 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-kcgbg" Oct 30 13:48:38.823156 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.823120 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-9r9zr"] Oct 30 13:48:38.823718 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.823699 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-659879fb99-pjvcq" Oct 30 13:48:38.827162 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.827141 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2nsk6"] Oct 30 13:48:38.827364 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.827350 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9r9zr" Oct 30 13:48:38.832448 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.832429 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 13:48:38.833250 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.833231 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-brd96"] Oct 30 13:48:38.833364 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.833348 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2nsk6" Oct 30 13:48:38.835585 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.835564 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 13:48:38.836530 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.836500 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qstzd"] Oct 30 13:48:38.836615 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.836565 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-brd96" Oct 30 13:48:38.838599 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.838574 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 13:48:38.839434 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.839420 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6bc4d9765d-dp7xb"] Oct 30 13:48:38.840137 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.839865 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qstzd" Oct 30 13:48:38.842137 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.842117 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 13:48:38.842273 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.842199 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-4xx27\"" Oct 30 13:48:38.842352 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.842339 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-d6wsl\"" Oct 30 13:48:38.842390 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.842377 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 13:48:38.842691 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.842679 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-tn69c"] Oct 30 13:48:38.842804 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.842787 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bc4d9765d-dp7xb" Oct 30 13:48:38.843687 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.843630 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 13:48:38.845511 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.845492 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-ntlqq"] Oct 30 13:48:38.845607 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.845589 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tn69c" Oct 30 13:48:38.846733 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.846708 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 13:48:38.848422 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.848405 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-2bnbl"] Oct 30 13:48:38.848544 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.848521 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-ntlqq" Oct 30 13:48:38.852161 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.852145 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-bhw8w"] Oct 30 13:48:38.852559 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.852325 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-2bnbl" Oct 30 13:48:38.856570 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.856548 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-87f7c8d8c-mdvsm"] Oct 30 13:48:38.857773 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.857755 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bhw8w" Oct 30 13:48:38.861250 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.860989 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-7lx4t"] Oct 30 13:48:38.861496 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.861480 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-87f7c8d8c-mdvsm" Oct 30 13:48:38.865651 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.865614 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xbkzq"] Oct 30 13:48:38.866286 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.866243 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7lx4t" Oct 30 13:48:38.873840 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.873823 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-4h2sq"] Oct 30 13:48:38.873901 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.873869 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-smvmg"] Oct 30 13:48:38.873901 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.873893 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-plvgd"] Oct 30 13:48:38.873961 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.873920 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-kcgbg"] Oct 30 13:48:38.873961 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.873944 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-brd96"] Oct 30 13:48:38.874018 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.873972 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-9r9zr"] Oct 30 13:48:38.874018 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.873994 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2nsk6"] Oct 30 13:48:38.874072 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.874020 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-2bnbl"] Oct 30 13:48:38.874072 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.874042 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qstzd"] Oct 30 13:48:38.874140 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.874074 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-bhw8w"] Oct 30 13:48:38.874140 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.874103 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xbkzq"] Oct 30 13:48:38.874140 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.874123 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7lx4t"] Oct 30 13:48:38.874254 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.874246 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xbkzq" Oct 30 13:48:38.884292 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.884074 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6bc4d9765d-dp7xb"] Oct 30 13:48:38.888915 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.888527 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-659879fb99-pjvcq"] Oct 30 13:48:38.888915 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.888577 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-tn69c"] Oct 30 13:48:38.894059 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.894040 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-ntlqq"] Oct 30 13:48:38.894832 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.894809 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-87f7c8d8c-mdvsm"] Oct 30 13:48:38.922146 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.922125 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwd54\" (UniqueName: \"kubernetes.io/projected/a4bd1bb2-25ae-423c-ba05-08eb742d340e-kube-api-access-pwd54\") pod \"insights-operator-7c4986d449-kcgbg\" (UID: \"a4bd1bb2-25ae-423c-ba05-08eb742d340e\") " pod="openshift-insights/insights-operator-7c4986d449-kcgbg" Oct 30 13:48:38.922227 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.922149 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d73c6c3-044d-4b88-a105-72bd5e4491cd-service-ca-bundle\") pod \"router-default-659879fb99-pjvcq\" (UID: \"6d73c6c3-044d-4b88-a105-72bd5e4491cd\") " pod="openshift-ingress/router-default-659879fb99-pjvcq" Oct 30 13:48:38.922227 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.922177 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b0b2d906-7995-4769-8222-fa8ba2a30f3d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-9r9zr\" (UID: \"b0b2d906-7995-4769-8222-fa8ba2a30f3d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9r9zr" Oct 30 13:48:38.922227 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.922218 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f707b282-e507-475d-a82a-923c9bd1ff01-registry-certificates\") pod \"image-registry-6bc4d9765d-dp7xb\" (UID: \"f707b282-e507-475d-a82a-923c9bd1ff01\") " pod="openshift-image-registry/image-registry-6bc4d9765d-dp7xb" Oct 30 13:48:38.922347 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.922252 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f707b282-e507-475d-a82a-923c9bd1ff01-registry-tls\") pod \"image-registry-6bc4d9765d-dp7xb\" (UID: \"f707b282-e507-475d-a82a-923c9bd1ff01\") " pod="openshift-image-registry/image-registry-6bc4d9765d-dp7xb" Oct 30 13:48:38.922347 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.922304 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6ac8569a-95ae-421e-aee0-edd189e18d11-cert\") pod \"ingress-canary-xbkzq\" (UID: \"6ac8569a-95ae-421e-aee0-edd189e18d11\") " pod="openshift-ingress-canary/ingress-canary-xbkzq" Oct 30 13:48:38.922347 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.922319 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d73c6c3-044d-4b88-a105-72bd5e4491cd-metrics-certs\") pod \"router-default-659879fb99-pjvcq\" (UID: \"6d73c6c3-044d-4b88-a105-72bd5e4491cd\") " pod="openshift-ingress/router-default-659879fb99-pjvcq" Oct 30 13:48:38.922347 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.922336 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e16ea4af-ce12-442b-91df-092457a7e192-bound-sa-token\") pod \"image-registry-87f7c8d8c-mdvsm\" (UID: \"e16ea4af-ce12-442b-91df-092457a7e192\") " pod="openshift-image-registry/image-registry-87f7c8d8c-mdvsm" Oct 30 13:48:38.922584 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.922350 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f707b282-e507-475d-a82a-923c9bd1ff01-installation-pull-secrets\") pod \"image-registry-6bc4d9765d-dp7xb\" (UID: \"f707b282-e507-475d-a82a-923c9bd1ff01\") " pod="openshift-image-registry/image-registry-6bc4d9765d-dp7xb" Oct 30 13:48:38.922584 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.922366 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ade75be2-e3bb-4b50-bed0-c9c257739707-serving-cert\") pod \"service-ca-operator-544d749c8c-tn69c\" (UID: \"ade75be2-e3bb-4b50-bed0-c9c257739707\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tn69c" Oct 30 13:48:38.922584 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.922420 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e16ea4af-ce12-442b-91df-092457a7e192-registry-certificates\") pod \"image-registry-87f7c8d8c-mdvsm\" (UID: \"e16ea4af-ce12-442b-91df-092457a7e192\") " pod="openshift-image-registry/image-registry-87f7c8d8c-mdvsm" Oct 30 13:48:38.922584 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.922478 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v9rm\" (UniqueName: \"kubernetes.io/projected/e16ea4af-ce12-442b-91df-092457a7e192-kube-api-access-5v9rm\") pod \"image-registry-87f7c8d8c-mdvsm\" (UID: \"e16ea4af-ce12-442b-91df-092457a7e192\") " pod="openshift-image-registry/image-registry-87f7c8d8c-mdvsm" Oct 30 13:48:38.922584 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.922506 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f707b282-e507-475d-a82a-923c9bd1ff01-trusted-ca\") pod \"image-registry-6bc4d9765d-dp7xb\" (UID: \"f707b282-e507-475d-a82a-923c9bd1ff01\") " pod="openshift-image-registry/image-registry-6bc4d9765d-dp7xb" Oct 30 13:48:38.922584 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.922559 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ade75be2-e3bb-4b50-bed0-c9c257739707-config\") pod \"service-ca-operator-544d749c8c-tn69c\" (UID: \"ade75be2-e3bb-4b50-bed0-c9c257739707\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tn69c" Oct 30 13:48:38.922788 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.922584 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwgrr\" (UniqueName: \"kubernetes.io/projected/dad9381c-aebb-448d-80b1-e2b65dd8ac80-kube-api-access-rwgrr\") pod \"kas-connection-checker-685895c7c9-4h2sq\" (UID: \"dad9381c-aebb-448d-80b1-e2b65dd8ac80\") " pod="kube-system/kas-connection-checker-685895c7c9-4h2sq" Oct 30 13:48:38.922788 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.922629 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm9mr\" (UniqueName: \"kubernetes.io/projected/f707b282-e507-475d-a82a-923c9bd1ff01-kube-api-access-vm9mr\") pod \"image-registry-6bc4d9765d-dp7xb\" (UID: \"f707b282-e507-475d-a82a-923c9bd1ff01\") " pod="openshift-image-registry/image-registry-6bc4d9765d-dp7xb" Oct 30 13:48:38.922788 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.922667 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6d73c6c3-044d-4b88-a105-72bd5e4491cd-default-certificate\") pod \"router-default-659879fb99-pjvcq\" (UID: \"6d73c6c3-044d-4b88-a105-72bd5e4491cd\") " pod="openshift-ingress/router-default-659879fb99-pjvcq" Oct 30 13:48:38.922788 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.922692 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p2xh\" (UniqueName: \"kubernetes.io/projected/083c70c2-0d5e-4876-9e88-10c6efbb681c-kube-api-access-6p2xh\") pod \"dns-default-7lx4t\" (UID: \"083c70c2-0d5e-4876-9e88-10c6efbb681c\") " pod="openshift-dns/dns-default-7lx4t" Oct 30 13:48:38.922788 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.922707 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e16ea4af-ce12-442b-91df-092457a7e192-installation-pull-secrets\") pod \"image-registry-87f7c8d8c-mdvsm\" (UID: \"e16ea4af-ce12-442b-91df-092457a7e192\") " pod="openshift-image-registry/image-registry-87f7c8d8c-mdvsm" Oct 30 13:48:38.922788 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.922728 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e16ea4af-ce12-442b-91df-092457a7e192-trusted-ca\") pod \"image-registry-87f7c8d8c-mdvsm\" (UID: \"e16ea4af-ce12-442b-91df-092457a7e192\") " pod="openshift-image-registry/image-registry-87f7c8d8c-mdvsm" Oct 30 13:48:38.922788 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.922766 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkhbw\" (UniqueName: \"kubernetes.io/projected/d7a64da6-7b20-4f20-90d7-bc23a71290e5-kube-api-access-zkhbw\") pod \"kas-connection-checker-685895c7c9-plvgd\" (UID: \"d7a64da6-7b20-4f20-90d7-bc23a71290e5\") " pod="kube-system/kas-connection-checker-685895c7c9-plvgd" Oct 30 13:48:38.923042 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.922801 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/083c70c2-0d5e-4876-9e88-10c6efbb681c-tmp-dir\") pod \"dns-default-7lx4t\" (UID: \"083c70c2-0d5e-4876-9e88-10c6efbb681c\") " pod="openshift-dns/dns-default-7lx4t" Oct 30 13:48:38.923042 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.922847 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dc967875-1546-40bb-8376-fde09416c204-trusted-ca\") pod \"console-operator-7cd56b8dbc-ntlqq\" (UID: \"dc967875-1546-40bb-8376-fde09416c204\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ntlqq" Oct 30 13:48:38.923042 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.922919 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6d73c6c3-044d-4b88-a105-72bd5e4491cd-stats-auth\") pod \"router-default-659879fb99-pjvcq\" (UID: \"6d73c6c3-044d-4b88-a105-72bd5e4491cd\") " pod="openshift-ingress/router-default-659879fb99-pjvcq" Oct 30 13:48:38.923042 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.922962 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/083c70c2-0d5e-4876-9e88-10c6efbb681c-metrics-tls\") pod \"dns-default-7lx4t\" (UID: \"083c70c2-0d5e-4876-9e88-10c6efbb681c\") " pod="openshift-dns/dns-default-7lx4t" Oct 30 13:48:38.923042 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.922992 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4bd1bb2-25ae-423c-ba05-08eb742d340e-tmp\") pod \"insights-operator-7c4986d449-kcgbg\" (UID: \"a4bd1bb2-25ae-423c-ba05-08eb742d340e\") " pod="openshift-insights/insights-operator-7c4986d449-kcgbg" Oct 30 13:48:38.923042 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.923013 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8304f8da-d4a3-45c1-bfe6-f5d08bed17d7-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2nsk6\" (UID: \"8304f8da-d4a3-45c1-bfe6-f5d08bed17d7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2nsk6" Oct 30 13:48:38.923042 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.923031 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/81dcad0b-d124-41b8-8b4e-d41049de217a-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-qstzd\" (UID: \"81dcad0b-d124-41b8-8b4e-d41049de217a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qstzd" Oct 30 13:48:38.923042 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.923047 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx49w\" (UniqueName: \"kubernetes.io/projected/8d9908f6-aefa-4bde-9182-2c199f08a24a-kube-api-access-fx49w\") pod \"volume-data-source-validator-58c9f5f46d-brd96\" (UID: \"8d9908f6-aefa-4bde-9182-2c199f08a24a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-brd96" Oct 30 13:48:38.923301 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.923079 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8309f073-dcb0-4971-9eb1-8a9ee0eb68c6-nginx-conf\") pod \"networking-console-plugin-86c478bb88-2bnbl\" (UID: \"8309f073-dcb0-4971-9eb1-8a9ee0eb68c6\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2bnbl" Oct 30 13:48:38.923301 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.923122 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b0b2d906-7995-4769-8222-fa8ba2a30f3d-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-9r9zr\" (UID: \"b0b2d906-7995-4769-8222-fa8ba2a30f3d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9r9zr" Oct 30 13:48:38.923301 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.923156 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dc967875-1546-40bb-8376-fde09416c204-config\") pod \"console-operator-7cd56b8dbc-ntlqq\" (UID: \"dc967875-1546-40bb-8376-fde09416c204\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ntlqq" Oct 30 13:48:38.923301 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.923174 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dc967875-1546-40bb-8376-fde09416c204-serving-cert\") pod \"console-operator-7cd56b8dbc-ntlqq\" (UID: \"dc967875-1546-40bb-8376-fde09416c204\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ntlqq" Oct 30 13:48:38.923301 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.923188 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n98n9\" (UniqueName: \"kubernetes.io/projected/dc967875-1546-40bb-8376-fde09416c204-kube-api-access-n98n9\") pod \"console-operator-7cd56b8dbc-ntlqq\" (UID: \"dc967875-1546-40bb-8376-fde09416c204\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ntlqq" Oct 30 13:48:38.923301 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.923203 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8309f073-dcb0-4971-9eb1-8a9ee0eb68c6-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2bnbl\" (UID: \"8309f073-dcb0-4971-9eb1-8a9ee0eb68c6\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2bnbl" Oct 30 13:48:38.923301 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.923218 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzwmc\" (UniqueName: \"kubernetes.io/projected/ade75be2-e3bb-4b50-bed0-c9c257739707-kube-api-access-tzwmc\") pod \"service-ca-operator-544d749c8c-tn69c\" (UID: \"ade75be2-e3bb-4b50-bed0-c9c257739707\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tn69c" Oct 30 13:48:38.923301 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.923248 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a4bd1bb2-25ae-423c-ba05-08eb742d340e-serving-cert\") pod \"insights-operator-7c4986d449-kcgbg\" (UID: \"a4bd1bb2-25ae-423c-ba05-08eb742d340e\") " pod="openshift-insights/insights-operator-7c4986d449-kcgbg" Oct 30 13:48:38.923301 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.923262 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfgzb\" (UniqueName: \"kubernetes.io/projected/6d73c6c3-044d-4b88-a105-72bd5e4491cd-kube-api-access-tfgzb\") pod \"router-default-659879fb99-pjvcq\" (UID: \"6d73c6c3-044d-4b88-a105-72bd5e4491cd\") " pod="openshift-ingress/router-default-659879fb99-pjvcq" Oct 30 13:48:38.923301 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.923288 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f707b282-e507-475d-a82a-923c9bd1ff01-image-registry-private-configuration\") pod \"image-registry-6bc4d9765d-dp7xb\" (UID: \"f707b282-e507-475d-a82a-923c9bd1ff01\") " pod="openshift-image-registry/image-registry-6bc4d9765d-dp7xb" Oct 30 13:48:38.923301 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.923303 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f707b282-e507-475d-a82a-923c9bd1ff01-bound-sa-token\") pod \"image-registry-6bc4d9765d-dp7xb\" (UID: \"f707b282-e507-475d-a82a-923c9bd1ff01\") " pod="openshift-image-registry/image-registry-6bc4d9765d-dp7xb" Oct 30 13:48:38.923611 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.923318 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e16ea4af-ce12-442b-91df-092457a7e192-image-registry-private-configuration\") pod \"image-registry-87f7c8d8c-mdvsm\" (UID: \"e16ea4af-ce12-442b-91df-092457a7e192\") " pod="openshift-image-registry/image-registry-87f7c8d8c-mdvsm" Oct 30 13:48:38.923611 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.923334 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f707b282-e507-475d-a82a-923c9bd1ff01-ca-trust-extracted\") pod \"image-registry-6bc4d9765d-dp7xb\" (UID: \"f707b282-e507-475d-a82a-923c9bd1ff01\") " pod="openshift-image-registry/image-registry-6bc4d9765d-dp7xb" Oct 30 13:48:38.923611 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.923347 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/083c70c2-0d5e-4876-9e88-10c6efbb681c-config-volume\") pod \"dns-default-7lx4t\" (UID: \"083c70c2-0d5e-4876-9e88-10c6efbb681c\") " pod="openshift-dns/dns-default-7lx4t" Oct 30 13:48:38.923611 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.923363 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l99vz\" (UniqueName: \"kubernetes.io/projected/cb90396e-d4e9-4eb0-888f-64c59d492fd4-kube-api-access-l99vz\") pod \"kas-connection-checker-685895c7c9-smvmg\" (UID: \"cb90396e-d4e9-4eb0-888f-64c59d492fd4\") " pod="kube-system/kas-connection-checker-685895c7c9-smvmg" Oct 30 13:48:38.923611 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.923378 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4bd1bb2-25ae-423c-ba05-08eb742d340e-service-ca-bundle\") pod \"insights-operator-7c4986d449-kcgbg\" (UID: \"a4bd1bb2-25ae-423c-ba05-08eb742d340e\") " pod="openshift-insights/insights-operator-7c4986d449-kcgbg" Oct 30 13:48:38.923611 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.923392 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69bm9\" (UniqueName: \"kubernetes.io/projected/6ac8569a-95ae-421e-aee0-edd189e18d11-kube-api-access-69bm9\") pod \"ingress-canary-xbkzq\" (UID: \"6ac8569a-95ae-421e-aee0-edd189e18d11\") " pod="openshift-ingress-canary/ingress-canary-xbkzq" Oct 30 13:48:38.923611 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.923413 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e16ea4af-ce12-442b-91df-092457a7e192-registry-tls\") pod \"image-registry-87f7c8d8c-mdvsm\" (UID: \"e16ea4af-ce12-442b-91df-092457a7e192\") " pod="openshift-image-registry/image-registry-87f7c8d8c-mdvsm" Oct 30 13:48:38.923611 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.923428 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8304f8da-d4a3-45c1-bfe6-f5d08bed17d7-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2nsk6\" (UID: \"8304f8da-d4a3-45c1-bfe6-f5d08bed17d7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2nsk6" Oct 30 13:48:38.923611 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.923445 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tzg4\" (UniqueName: \"kubernetes.io/projected/8304f8da-d4a3-45c1-bfe6-f5d08bed17d7-kube-api-access-4tzg4\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2nsk6\" (UID: \"8304f8da-d4a3-45c1-bfe6-f5d08bed17d7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2nsk6" Oct 30 13:48:38.923611 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.923462 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5qnd\" (UniqueName: \"kubernetes.io/projected/b0b2d906-7995-4769-8222-fa8ba2a30f3d-kube-api-access-t5qnd\") pod \"cluster-monitoring-operator-884576c57-9r9zr\" (UID: \"b0b2d906-7995-4769-8222-fa8ba2a30f3d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9r9zr" Oct 30 13:48:38.923611 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.923475 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/81dcad0b-d124-41b8-8b4e-d41049de217a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-qstzd\" (UID: \"81dcad0b-d124-41b8-8b4e-d41049de217a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qstzd" Oct 30 13:48:38.923611 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.923490 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spnhx\" (UniqueName: \"kubernetes.io/projected/81dcad0b-d124-41b8-8b4e-d41049de217a-kube-api-access-spnhx\") pod \"cluster-samples-operator-6ddb6c8b94-qstzd\" (UID: \"81dcad0b-d124-41b8-8b4e-d41049de217a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qstzd" Oct 30 13:48:38.923611 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.923504 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a4bd1bb2-25ae-423c-ba05-08eb742d340e-snapshots\") pod \"insights-operator-7c4986d449-kcgbg\" (UID: \"a4bd1bb2-25ae-423c-ba05-08eb742d340e\") " pod="openshift-insights/insights-operator-7c4986d449-kcgbg" Oct 30 13:48:38.923611 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.923522 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk8lh\" (UniqueName: \"kubernetes.io/projected/806dd609-9a2a-49c4-a656-e50437457fa9-kube-api-access-gk8lh\") pod \"network-check-source-6f78d59f99-bhw8w\" (UID: \"806dd609-9a2a-49c4-a656-e50437457fa9\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bhw8w" Oct 30 13:48:38.924019 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.923536 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4bd1bb2-25ae-423c-ba05-08eb742d340e-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-kcgbg\" (UID: \"a4bd1bb2-25ae-423c-ba05-08eb742d340e\") " pod="openshift-insights/insights-operator-7c4986d449-kcgbg" Oct 30 13:48:38.924019 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.923552 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e16ea4af-ce12-442b-91df-092457a7e192-ca-trust-extracted\") pod \"image-registry-87f7c8d8c-mdvsm\" (UID: \"e16ea4af-ce12-442b-91df-092457a7e192\") " pod="openshift-image-registry/image-registry-87f7c8d8c-mdvsm" Oct 30 13:48:38.937238 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.937223 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 13:48:38.937427 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.937405 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 13:48:38.937506 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.937484 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 13:48:38.937564 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.937534 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 13:48:38.937614 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.937577 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 13:48:38.937688 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.937633 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 13:48:38.937688 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.937667 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 13:48:38.937769 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.937687 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 13:48:38.937870 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.937856 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 13:48:38.937927 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.937875 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 13:48:38.937927 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.937883 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 13:48:38.937927 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.937897 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 13:48:38.938063 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.937932 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 13:48:38.938063 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.937942 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 13:48:38.938063 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.938016 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 13:48:38.938063 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.938017 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 13:48:38.938210 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.938066 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 13:48:38.938210 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.938123 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 13:48:38.938210 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.938129 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 13:48:38.938311 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.938261 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 13:48:38.938311 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.938273 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 13:48:38.938311 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.938298 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 13:48:38.938311 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.938308 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 13:48:38.938489 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.938304 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 13:48:38.938489 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.938326 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 13:48:38.938489 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.938377 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 13:48:38.938489 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.938397 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 13:48:38.938669 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.938542 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 13:48:38.952546 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.952518 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 13:48:38.952682 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.952669 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 13:48:38.952750 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.952699 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 13:48:38.952750 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.952731 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 13:48:38.952845 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.952723 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 13:48:38.952845 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.952773 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 13:48:38.952845 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.952794 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 13:48:38.952845 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.952808 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 13:48:38.952845 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.952837 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-shzzf\"" Oct 30 13:48:38.952845 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.952843 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-qtrmb\"" Oct 30 13:48:38.953038 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.952901 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-snt8q\"" Oct 30 13:48:38.953038 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.952938 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-b7cls\"" Oct 30 13:48:38.953038 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.952960 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 13:48:38.953038 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.952991 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-8strq\"" Oct 30 13:48:38.953169 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.953040 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-wstqj\"" Oct 30 13:48:38.953169 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.953146 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 13:48:38.953238 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.953196 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-5d5mh\"" Oct 30 13:48:38.953314 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.953298 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-9vpkx\"" Oct 30 13:48:38.953314 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.953312 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-t9cdd\"" Oct 30 13:48:38.953477 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.953364 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-j65x4\"" Oct 30 13:48:38.953477 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.953395 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-w8k82\"" Oct 30 13:48:38.964253 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.964239 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 13:48:38.966335 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.966319 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 13:48:38.966422 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.966407 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 13:48:38.967010 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:38.966993 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 13:48:39.023888 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.023840 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b0b2d906-7995-4769-8222-fa8ba2a30f3d-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-9r9zr\" (UID: \"b0b2d906-7995-4769-8222-fa8ba2a30f3d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9r9zr" Oct 30 13:48:39.023888 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.023867 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dc967875-1546-40bb-8376-fde09416c204-config\") pod \"console-operator-7cd56b8dbc-ntlqq\" (UID: \"dc967875-1546-40bb-8376-fde09416c204\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ntlqq" Oct 30 13:48:39.023888 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.023884 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dc967875-1546-40bb-8376-fde09416c204-serving-cert\") pod \"console-operator-7cd56b8dbc-ntlqq\" (UID: \"dc967875-1546-40bb-8376-fde09416c204\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ntlqq" Oct 30 13:48:39.024073 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.023900 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n98n9\" (UniqueName: \"kubernetes.io/projected/dc967875-1546-40bb-8376-fde09416c204-kube-api-access-n98n9\") pod \"console-operator-7cd56b8dbc-ntlqq\" (UID: \"dc967875-1546-40bb-8376-fde09416c204\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ntlqq" Oct 30 13:48:39.024073 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.023919 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8309f073-dcb0-4971-9eb1-8a9ee0eb68c6-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2bnbl\" (UID: \"8309f073-dcb0-4971-9eb1-8a9ee0eb68c6\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2bnbl" Oct 30 13:48:39.024073 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.023942 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzwmc\" (UniqueName: \"kubernetes.io/projected/ade75be2-e3bb-4b50-bed0-c9c257739707-kube-api-access-tzwmc\") pod \"service-ca-operator-544d749c8c-tn69c\" (UID: \"ade75be2-e3bb-4b50-bed0-c9c257739707\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tn69c" Oct 30 13:48:39.024073 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:39.024028 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 13:48:39.024073 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.024056 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a4bd1bb2-25ae-423c-ba05-08eb742d340e-serving-cert\") pod \"insights-operator-7c4986d449-kcgbg\" (UID: \"a4bd1bb2-25ae-423c-ba05-08eb742d340e\") " pod="openshift-insights/insights-operator-7c4986d449-kcgbg" Oct 30 13:48:39.024330 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:39.024087 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8309f073-dcb0-4971-9eb1-8a9ee0eb68c6-networking-console-plugin-cert podName:8309f073-dcb0-4971-9eb1-8a9ee0eb68c6 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:39.524069657 +0000 UTC m=+29.543591053 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8309f073-dcb0-4971-9eb1-8a9ee0eb68c6-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-2bnbl" (UID: "8309f073-dcb0-4971-9eb1-8a9ee0eb68c6") : secret "networking-console-plugin-cert" not found Oct 30 13:48:39.024330 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.024106 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfgzb\" (UniqueName: \"kubernetes.io/projected/6d73c6c3-044d-4b88-a105-72bd5e4491cd-kube-api-access-tfgzb\") pod \"router-default-659879fb99-pjvcq\" (UID: \"6d73c6c3-044d-4b88-a105-72bd5e4491cd\") " pod="openshift-ingress/router-default-659879fb99-pjvcq" Oct 30 13:48:39.024330 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.024147 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f707b282-e507-475d-a82a-923c9bd1ff01-image-registry-private-configuration\") pod \"image-registry-6bc4d9765d-dp7xb\" (UID: \"f707b282-e507-475d-a82a-923c9bd1ff01\") " pod="openshift-image-registry/image-registry-6bc4d9765d-dp7xb" Oct 30 13:48:39.024330 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.024173 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f707b282-e507-475d-a82a-923c9bd1ff01-bound-sa-token\") pod \"image-registry-6bc4d9765d-dp7xb\" (UID: \"f707b282-e507-475d-a82a-923c9bd1ff01\") " pod="openshift-image-registry/image-registry-6bc4d9765d-dp7xb" Oct 30 13:48:39.024330 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.024195 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e16ea4af-ce12-442b-91df-092457a7e192-image-registry-private-configuration\") pod \"image-registry-87f7c8d8c-mdvsm\" (UID: \"e16ea4af-ce12-442b-91df-092457a7e192\") " pod="openshift-image-registry/image-registry-87f7c8d8c-mdvsm" Oct 30 13:48:39.024330 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.024220 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f707b282-e507-475d-a82a-923c9bd1ff01-ca-trust-extracted\") pod \"image-registry-6bc4d9765d-dp7xb\" (UID: \"f707b282-e507-475d-a82a-923c9bd1ff01\") " pod="openshift-image-registry/image-registry-6bc4d9765d-dp7xb" Oct 30 13:48:39.024616 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.024545 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f707b282-e507-475d-a82a-923c9bd1ff01-ca-trust-extracted\") pod \"image-registry-6bc4d9765d-dp7xb\" (UID: \"f707b282-e507-475d-a82a-923c9bd1ff01\") " pod="openshift-image-registry/image-registry-6bc4d9765d-dp7xb" Oct 30 13:48:39.024616 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.024545 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/083c70c2-0d5e-4876-9e88-10c6efbb681c-config-volume\") pod \"dns-default-7lx4t\" (UID: \"083c70c2-0d5e-4876-9e88-10c6efbb681c\") " pod="openshift-dns/dns-default-7lx4t" Oct 30 13:48:39.024616 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.024601 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l99vz\" (UniqueName: \"kubernetes.io/projected/cb90396e-d4e9-4eb0-888f-64c59d492fd4-kube-api-access-l99vz\") pod \"kas-connection-checker-685895c7c9-smvmg\" (UID: \"cb90396e-d4e9-4eb0-888f-64c59d492fd4\") " pod="kube-system/kas-connection-checker-685895c7c9-smvmg" Oct 30 13:48:39.024781 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.024625 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4bd1bb2-25ae-423c-ba05-08eb742d340e-service-ca-bundle\") pod \"insights-operator-7c4986d449-kcgbg\" (UID: \"a4bd1bb2-25ae-423c-ba05-08eb742d340e\") " pod="openshift-insights/insights-operator-7c4986d449-kcgbg" Oct 30 13:48:39.024781 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.024672 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69bm9\" (UniqueName: \"kubernetes.io/projected/6ac8569a-95ae-421e-aee0-edd189e18d11-kube-api-access-69bm9\") pod \"ingress-canary-xbkzq\" (UID: \"6ac8569a-95ae-421e-aee0-edd189e18d11\") " pod="openshift-ingress-canary/ingress-canary-xbkzq" Oct 30 13:48:39.024781 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.024716 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dc967875-1546-40bb-8376-fde09416c204-config\") pod \"console-operator-7cd56b8dbc-ntlqq\" (UID: \"dc967875-1546-40bb-8376-fde09416c204\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ntlqq" Oct 30 13:48:39.024781 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.024732 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e16ea4af-ce12-442b-91df-092457a7e192-registry-tls\") pod \"image-registry-87f7c8d8c-mdvsm\" (UID: \"e16ea4af-ce12-442b-91df-092457a7e192\") " pod="openshift-image-registry/image-registry-87f7c8d8c-mdvsm" Oct 30 13:48:39.024781 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.024764 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b0b2d906-7995-4769-8222-fa8ba2a30f3d-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-9r9zr\" (UID: \"b0b2d906-7995-4769-8222-fa8ba2a30f3d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9r9zr" Oct 30 13:48:39.024781 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.024775 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8304f8da-d4a3-45c1-bfe6-f5d08bed17d7-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2nsk6\" (UID: \"8304f8da-d4a3-45c1-bfe6-f5d08bed17d7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2nsk6" Oct 30 13:48:39.025052 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:39.024809 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 13:48:39.025052 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:39.024822 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-87f7c8d8c-mdvsm: secret "image-registry-tls" not found Oct 30 13:48:39.025052 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.024819 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tzg4\" (UniqueName: \"kubernetes.io/projected/8304f8da-d4a3-45c1-bfe6-f5d08bed17d7-kube-api-access-4tzg4\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2nsk6\" (UID: \"8304f8da-d4a3-45c1-bfe6-f5d08bed17d7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2nsk6" Oct 30 13:48:39.025052 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.024848 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5qnd\" (UniqueName: \"kubernetes.io/projected/b0b2d906-7995-4769-8222-fa8ba2a30f3d-kube-api-access-t5qnd\") pod \"cluster-monitoring-operator-884576c57-9r9zr\" (UID: \"b0b2d906-7995-4769-8222-fa8ba2a30f3d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9r9zr" Oct 30 13:48:39.025052 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:39.024874 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e16ea4af-ce12-442b-91df-092457a7e192-registry-tls podName:e16ea4af-ce12-442b-91df-092457a7e192 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:39.524856226 +0000 UTC m=+29.544377624 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e16ea4af-ce12-442b-91df-092457a7e192-registry-tls") pod "image-registry-87f7c8d8c-mdvsm" (UID: "e16ea4af-ce12-442b-91df-092457a7e192") : secret "image-registry-tls" not found Oct 30 13:48:39.025052 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.024902 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/81dcad0b-d124-41b8-8b4e-d41049de217a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-qstzd\" (UID: \"81dcad0b-d124-41b8-8b4e-d41049de217a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qstzd" Oct 30 13:48:39.025052 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.024947 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spnhx\" (UniqueName: \"kubernetes.io/projected/81dcad0b-d124-41b8-8b4e-d41049de217a-kube-api-access-spnhx\") pod \"cluster-samples-operator-6ddb6c8b94-qstzd\" (UID: \"81dcad0b-d124-41b8-8b4e-d41049de217a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qstzd" Oct 30 13:48:39.025347 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.025123 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/083c70c2-0d5e-4876-9e88-10c6efbb681c-config-volume\") pod \"dns-default-7lx4t\" (UID: \"083c70c2-0d5e-4876-9e88-10c6efbb681c\") " pod="openshift-dns/dns-default-7lx4t" Oct 30 13:48:39.025389 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.025375 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4bd1bb2-25ae-423c-ba05-08eb742d340e-service-ca-bundle\") pod \"insights-operator-7c4986d449-kcgbg\" (UID: \"a4bd1bb2-25ae-423c-ba05-08eb742d340e\") " pod="openshift-insights/insights-operator-7c4986d449-kcgbg" Oct 30 13:48:39.025432 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:39.025422 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 13:48:39.025514 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:39.025499 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81dcad0b-d124-41b8-8b4e-d41049de217a-samples-operator-tls podName:81dcad0b-d124-41b8-8b4e-d41049de217a nodeName:}" failed. No retries permitted until 2026-10-30 13:48:39.525482727 +0000 UTC m=+29.545004119 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/81dcad0b-d124-41b8-8b4e-d41049de217a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-qstzd" (UID: "81dcad0b-d124-41b8-8b4e-d41049de217a") : secret "samples-operator-tls" not found Oct 30 13:48:39.025629 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.025612 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a4bd1bb2-25ae-423c-ba05-08eb742d340e-snapshots\") pod \"insights-operator-7c4986d449-kcgbg\" (UID: \"a4bd1bb2-25ae-423c-ba05-08eb742d340e\") " pod="openshift-insights/insights-operator-7c4986d449-kcgbg" Oct 30 13:48:39.025710 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.025672 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk8lh\" (UniqueName: \"kubernetes.io/projected/806dd609-9a2a-49c4-a656-e50437457fa9-kube-api-access-gk8lh\") pod \"network-check-source-6f78d59f99-bhw8w\" (UID: \"806dd609-9a2a-49c4-a656-e50437457fa9\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bhw8w" Oct 30 13:48:39.025710 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.025700 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4bd1bb2-25ae-423c-ba05-08eb742d340e-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-kcgbg\" (UID: \"a4bd1bb2-25ae-423c-ba05-08eb742d340e\") " pod="openshift-insights/insights-operator-7c4986d449-kcgbg" Oct 30 13:48:39.025821 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.025707 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a4bd1bb2-25ae-423c-ba05-08eb742d340e-snapshots\") pod \"insights-operator-7c4986d449-kcgbg\" (UID: \"a4bd1bb2-25ae-423c-ba05-08eb742d340e\") " pod="openshift-insights/insights-operator-7c4986d449-kcgbg" Oct 30 13:48:39.025821 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.025727 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e16ea4af-ce12-442b-91df-092457a7e192-ca-trust-extracted\") pod \"image-registry-87f7c8d8c-mdvsm\" (UID: \"e16ea4af-ce12-442b-91df-092457a7e192\") " pod="openshift-image-registry/image-registry-87f7c8d8c-mdvsm" Oct 30 13:48:39.025821 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.025754 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwd54\" (UniqueName: \"kubernetes.io/projected/a4bd1bb2-25ae-423c-ba05-08eb742d340e-kube-api-access-pwd54\") pod \"insights-operator-7c4986d449-kcgbg\" (UID: \"a4bd1bb2-25ae-423c-ba05-08eb742d340e\") " pod="openshift-insights/insights-operator-7c4986d449-kcgbg" Oct 30 13:48:39.025821 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.025777 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d73c6c3-044d-4b88-a105-72bd5e4491cd-service-ca-bundle\") pod \"router-default-659879fb99-pjvcq\" (UID: \"6d73c6c3-044d-4b88-a105-72bd5e4491cd\") " pod="openshift-ingress/router-default-659879fb99-pjvcq" Oct 30 13:48:39.025821 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.025814 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b0b2d906-7995-4769-8222-fa8ba2a30f3d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-9r9zr\" (UID: \"b0b2d906-7995-4769-8222-fa8ba2a30f3d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9r9zr" Oct 30 13:48:39.026104 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.025839 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f707b282-e507-475d-a82a-923c9bd1ff01-registry-certificates\") pod \"image-registry-6bc4d9765d-dp7xb\" (UID: \"f707b282-e507-475d-a82a-923c9bd1ff01\") " pod="openshift-image-registry/image-registry-6bc4d9765d-dp7xb" Oct 30 13:48:39.026104 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.025865 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f707b282-e507-475d-a82a-923c9bd1ff01-registry-tls\") pod \"image-registry-6bc4d9765d-dp7xb\" (UID: \"f707b282-e507-475d-a82a-923c9bd1ff01\") " pod="openshift-image-registry/image-registry-6bc4d9765d-dp7xb" Oct 30 13:48:39.026104 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.025891 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6ac8569a-95ae-421e-aee0-edd189e18d11-cert\") pod \"ingress-canary-xbkzq\" (UID: \"6ac8569a-95ae-421e-aee0-edd189e18d11\") " pod="openshift-ingress-canary/ingress-canary-xbkzq" Oct 30 13:48:39.026104 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.025915 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d73c6c3-044d-4b88-a105-72bd5e4491cd-metrics-certs\") pod \"router-default-659879fb99-pjvcq\" (UID: \"6d73c6c3-044d-4b88-a105-72bd5e4491cd\") " pod="openshift-ingress/router-default-659879fb99-pjvcq" Oct 30 13:48:39.026104 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:39.025989 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 13:48:39.026104 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:39.025991 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6d73c6c3-044d-4b88-a105-72bd5e4491cd-service-ca-bundle podName:6d73c6c3-044d-4b88-a105-72bd5e4491cd nodeName:}" failed. No retries permitted until 2026-10-30 13:48:39.525977811 +0000 UTC m=+29.545499209 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6d73c6c3-044d-4b88-a105-72bd5e4491cd-service-ca-bundle") pod "router-default-659879fb99-pjvcq" (UID: "6d73c6c3-044d-4b88-a105-72bd5e4491cd") : configmap references non-existent config key: service-ca.crt Oct 30 13:48:39.026104 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:39.026044 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d73c6c3-044d-4b88-a105-72bd5e4491cd-metrics-certs podName:6d73c6c3-044d-4b88-a105-72bd5e4491cd nodeName:}" failed. No retries permitted until 2026-10-30 13:48:39.526033613 +0000 UTC m=+29.545554996 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d73c6c3-044d-4b88-a105-72bd5e4491cd-metrics-certs") pod "router-default-659879fb99-pjvcq" (UID: "6d73c6c3-044d-4b88-a105-72bd5e4491cd") : secret "router-metrics-certs-default" not found Oct 30 13:48:39.026769 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.026583 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e16ea4af-ce12-442b-91df-092457a7e192-bound-sa-token\") pod \"image-registry-87f7c8d8c-mdvsm\" (UID: \"e16ea4af-ce12-442b-91df-092457a7e192\") " pod="openshift-image-registry/image-registry-87f7c8d8c-mdvsm" Oct 30 13:48:39.026769 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.026614 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f707b282-e507-475d-a82a-923c9bd1ff01-installation-pull-secrets\") pod \"image-registry-6bc4d9765d-dp7xb\" (UID: \"f707b282-e507-475d-a82a-923c9bd1ff01\") " pod="openshift-image-registry/image-registry-6bc4d9765d-dp7xb" Oct 30 13:48:39.026769 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.026659 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ade75be2-e3bb-4b50-bed0-c9c257739707-serving-cert\") pod \"service-ca-operator-544d749c8c-tn69c\" (UID: \"ade75be2-e3bb-4b50-bed0-c9c257739707\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tn69c" Oct 30 13:48:39.026769 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.026691 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e16ea4af-ce12-442b-91df-092457a7e192-registry-certificates\") pod \"image-registry-87f7c8d8c-mdvsm\" (UID: \"e16ea4af-ce12-442b-91df-092457a7e192\") " pod="openshift-image-registry/image-registry-87f7c8d8c-mdvsm" Oct 30 13:48:39.026769 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.026718 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v9rm\" (UniqueName: \"kubernetes.io/projected/e16ea4af-ce12-442b-91df-092457a7e192-kube-api-access-5v9rm\") pod \"image-registry-87f7c8d8c-mdvsm\" (UID: \"e16ea4af-ce12-442b-91df-092457a7e192\") " pod="openshift-image-registry/image-registry-87f7c8d8c-mdvsm" Oct 30 13:48:39.026769 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.026747 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f707b282-e507-475d-a82a-923c9bd1ff01-trusted-ca\") pod \"image-registry-6bc4d9765d-dp7xb\" (UID: \"f707b282-e507-475d-a82a-923c9bd1ff01\") " pod="openshift-image-registry/image-registry-6bc4d9765d-dp7xb" Oct 30 13:48:39.026769 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.026769 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ade75be2-e3bb-4b50-bed0-c9c257739707-config\") pod \"service-ca-operator-544d749c8c-tn69c\" (UID: \"ade75be2-e3bb-4b50-bed0-c9c257739707\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tn69c" Oct 30 13:48:39.027262 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.026794 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwgrr\" (UniqueName: \"kubernetes.io/projected/dad9381c-aebb-448d-80b1-e2b65dd8ac80-kube-api-access-rwgrr\") pod \"kas-connection-checker-685895c7c9-4h2sq\" (UID: \"dad9381c-aebb-448d-80b1-e2b65dd8ac80\") " pod="kube-system/kas-connection-checker-685895c7c9-4h2sq" Oct 30 13:48:39.027262 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.026825 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm9mr\" (UniqueName: \"kubernetes.io/projected/f707b282-e507-475d-a82a-923c9bd1ff01-kube-api-access-vm9mr\") pod \"image-registry-6bc4d9765d-dp7xb\" (UID: \"f707b282-e507-475d-a82a-923c9bd1ff01\") " pod="openshift-image-registry/image-registry-6bc4d9765d-dp7xb" Oct 30 13:48:39.027262 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.026855 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6d73c6c3-044d-4b88-a105-72bd5e4491cd-default-certificate\") pod \"router-default-659879fb99-pjvcq\" (UID: \"6d73c6c3-044d-4b88-a105-72bd5e4491cd\") " pod="openshift-ingress/router-default-659879fb99-pjvcq" Oct 30 13:48:39.027262 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.026888 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p2xh\" (UniqueName: \"kubernetes.io/projected/083c70c2-0d5e-4876-9e88-10c6efbb681c-kube-api-access-6p2xh\") pod \"dns-default-7lx4t\" (UID: \"083c70c2-0d5e-4876-9e88-10c6efbb681c\") " pod="openshift-dns/dns-default-7lx4t" Oct 30 13:48:39.027262 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.026911 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e16ea4af-ce12-442b-91df-092457a7e192-installation-pull-secrets\") pod \"image-registry-87f7c8d8c-mdvsm\" (UID: \"e16ea4af-ce12-442b-91df-092457a7e192\") " pod="openshift-image-registry/image-registry-87f7c8d8c-mdvsm" Oct 30 13:48:39.027262 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.026911 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4bd1bb2-25ae-423c-ba05-08eb742d340e-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-kcgbg\" (UID: \"a4bd1bb2-25ae-423c-ba05-08eb742d340e\") " pod="openshift-insights/insights-operator-7c4986d449-kcgbg" Oct 30 13:48:39.027262 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.026940 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e16ea4af-ce12-442b-91df-092457a7e192-trusted-ca\") pod \"image-registry-87f7c8d8c-mdvsm\" (UID: \"e16ea4af-ce12-442b-91df-092457a7e192\") " pod="openshift-image-registry/image-registry-87f7c8d8c-mdvsm" Oct 30 13:48:39.027262 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.026970 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkhbw\" (UniqueName: \"kubernetes.io/projected/d7a64da6-7b20-4f20-90d7-bc23a71290e5-kube-api-access-zkhbw\") pod \"kas-connection-checker-685895c7c9-plvgd\" (UID: \"d7a64da6-7b20-4f20-90d7-bc23a71290e5\") " pod="kube-system/kas-connection-checker-685895c7c9-plvgd" Oct 30 13:48:39.027262 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.026994 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/083c70c2-0d5e-4876-9e88-10c6efbb681c-tmp-dir\") pod \"dns-default-7lx4t\" (UID: \"083c70c2-0d5e-4876-9e88-10c6efbb681c\") " pod="openshift-dns/dns-default-7lx4t" Oct 30 13:48:39.027262 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.027001 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e16ea4af-ce12-442b-91df-092457a7e192-ca-trust-extracted\") pod \"image-registry-87f7c8d8c-mdvsm\" (UID: \"e16ea4af-ce12-442b-91df-092457a7e192\") " pod="openshift-image-registry/image-registry-87f7c8d8c-mdvsm" Oct 30 13:48:39.027262 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.027022 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dc967875-1546-40bb-8376-fde09416c204-trusted-ca\") pod \"console-operator-7cd56b8dbc-ntlqq\" (UID: \"dc967875-1546-40bb-8376-fde09416c204\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ntlqq" Oct 30 13:48:39.027262 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.027046 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6d73c6c3-044d-4b88-a105-72bd5e4491cd-stats-auth\") pod \"router-default-659879fb99-pjvcq\" (UID: \"6d73c6c3-044d-4b88-a105-72bd5e4491cd\") " pod="openshift-ingress/router-default-659879fb99-pjvcq" Oct 30 13:48:39.027262 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.027076 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/083c70c2-0d5e-4876-9e88-10c6efbb681c-metrics-tls\") pod \"dns-default-7lx4t\" (UID: \"083c70c2-0d5e-4876-9e88-10c6efbb681c\") " pod="openshift-dns/dns-default-7lx4t" Oct 30 13:48:39.027262 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.027099 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4bd1bb2-25ae-423c-ba05-08eb742d340e-tmp\") pod \"insights-operator-7c4986d449-kcgbg\" (UID: \"a4bd1bb2-25ae-423c-ba05-08eb742d340e\") " pod="openshift-insights/insights-operator-7c4986d449-kcgbg" Oct 30 13:48:39.027262 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.027127 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8304f8da-d4a3-45c1-bfe6-f5d08bed17d7-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2nsk6\" (UID: \"8304f8da-d4a3-45c1-bfe6-f5d08bed17d7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2nsk6" Oct 30 13:48:39.027262 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.027151 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/81dcad0b-d124-41b8-8b4e-d41049de217a-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-qstzd\" (UID: \"81dcad0b-d124-41b8-8b4e-d41049de217a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qstzd" Oct 30 13:48:39.028204 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.027180 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx49w\" (UniqueName: \"kubernetes.io/projected/8d9908f6-aefa-4bde-9182-2c199f08a24a-kube-api-access-fx49w\") pod \"volume-data-source-validator-58c9f5f46d-brd96\" (UID: \"8d9908f6-aefa-4bde-9182-2c199f08a24a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-brd96" Oct 30 13:48:39.028204 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.027202 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8309f073-dcb0-4971-9eb1-8a9ee0eb68c6-nginx-conf\") pod \"networking-console-plugin-86c478bb88-2bnbl\" (UID: \"8309f073-dcb0-4971-9eb1-8a9ee0eb68c6\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2bnbl" Oct 30 13:48:39.028204 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.027483 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ade75be2-e3bb-4b50-bed0-c9c257739707-config\") pod \"service-ca-operator-544d749c8c-tn69c\" (UID: \"ade75be2-e3bb-4b50-bed0-c9c257739707\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tn69c" Oct 30 13:48:39.028204 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.027696 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/083c70c2-0d5e-4876-9e88-10c6efbb681c-tmp-dir\") pod \"dns-default-7lx4t\" (UID: \"083c70c2-0d5e-4876-9e88-10c6efbb681c\") " pod="openshift-dns/dns-default-7lx4t" Oct 30 13:48:39.028204 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.027898 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8309f073-dcb0-4971-9eb1-8a9ee0eb68c6-nginx-conf\") pod \"networking-console-plugin-86c478bb88-2bnbl\" (UID: \"8309f073-dcb0-4971-9eb1-8a9ee0eb68c6\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2bnbl" Oct 30 13:48:39.028204 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.027941 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f707b282-e507-475d-a82a-923c9bd1ff01-registry-certificates\") pod \"image-registry-6bc4d9765d-dp7xb\" (UID: \"f707b282-e507-475d-a82a-923c9bd1ff01\") " pod="openshift-image-registry/image-registry-6bc4d9765d-dp7xb" Oct 30 13:48:39.032039 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:39.028529 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 13:48:39.032039 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:39.028575 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0b2d906-7995-4769-8222-fa8ba2a30f3d-cluster-monitoring-operator-tls podName:b0b2d906-7995-4769-8222-fa8ba2a30f3d nodeName:}" failed. No retries permitted until 2026-10-30 13:48:39.528561582 +0000 UTC m=+29.548082977 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b0b2d906-7995-4769-8222-fa8ba2a30f3d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-9r9zr" (UID: "b0b2d906-7995-4769-8222-fa8ba2a30f3d") : secret "cluster-monitoring-operator-tls" not found Oct 30 13:48:39.032039 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.028900 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8304f8da-d4a3-45c1-bfe6-f5d08bed17d7-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2nsk6\" (UID: \"8304f8da-d4a3-45c1-bfe6-f5d08bed17d7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2nsk6" Oct 30 13:48:39.032039 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.028938 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4bd1bb2-25ae-423c-ba05-08eb742d340e-tmp\") pod \"insights-operator-7c4986d449-kcgbg\" (UID: \"a4bd1bb2-25ae-423c-ba05-08eb742d340e\") " pod="openshift-insights/insights-operator-7c4986d449-kcgbg" Oct 30 13:48:39.032039 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.029030 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e16ea4af-ce12-442b-91df-092457a7e192-image-registry-private-configuration\") pod \"image-registry-87f7c8d8c-mdvsm\" (UID: \"e16ea4af-ce12-442b-91df-092457a7e192\") " pod="openshift-image-registry/image-registry-87f7c8d8c-mdvsm" Oct 30 13:48:39.032039 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.029195 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dc967875-1546-40bb-8376-fde09416c204-serving-cert\") pod \"console-operator-7cd56b8dbc-ntlqq\" (UID: \"dc967875-1546-40bb-8376-fde09416c204\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ntlqq" Oct 30 13:48:39.032039 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.029926 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8304f8da-d4a3-45c1-bfe6-f5d08bed17d7-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2nsk6\" (UID: \"8304f8da-d4a3-45c1-bfe6-f5d08bed17d7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2nsk6" Oct 30 13:48:39.032039 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.030527 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/81dcad0b-d124-41b8-8b4e-d41049de217a-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-qstzd\" (UID: \"81dcad0b-d124-41b8-8b4e-d41049de217a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qstzd" Oct 30 13:48:39.032039 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.030571 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a4bd1bb2-25ae-423c-ba05-08eb742d340e-serving-cert\") pod \"insights-operator-7c4986d449-kcgbg\" (UID: \"a4bd1bb2-25ae-423c-ba05-08eb742d340e\") " pod="openshift-insights/insights-operator-7c4986d449-kcgbg" Oct 30 13:48:39.032039 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.030606 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e16ea4af-ce12-442b-91df-092457a7e192-trusted-ca\") pod \"image-registry-87f7c8d8c-mdvsm\" (UID: \"e16ea4af-ce12-442b-91df-092457a7e192\") " pod="openshift-image-registry/image-registry-87f7c8d8c-mdvsm" Oct 30 13:48:39.032039 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.030607 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f707b282-e507-475d-a82a-923c9bd1ff01-installation-pull-secrets\") pod \"image-registry-6bc4d9765d-dp7xb\" (UID: \"f707b282-e507-475d-a82a-923c9bd1ff01\") " pod="openshift-image-registry/image-registry-6bc4d9765d-dp7xb" Oct 30 13:48:39.032039 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:39.030810 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 13:48:39.032039 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:39.030861 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/083c70c2-0d5e-4876-9e88-10c6efbb681c-metrics-tls podName:083c70c2-0d5e-4876-9e88-10c6efbb681c nodeName:}" failed. No retries permitted until 2026-10-30 13:48:39.530848162 +0000 UTC m=+29.550369566 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/083c70c2-0d5e-4876-9e88-10c6efbb681c-metrics-tls") pod "dns-default-7lx4t" (UID: "083c70c2-0d5e-4876-9e88-10c6efbb681c") : secret "dns-default-metrics-tls" not found Oct 30 13:48:39.032039 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.030960 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6ac8569a-95ae-421e-aee0-edd189e18d11-cert\") pod \"ingress-canary-xbkzq\" (UID: \"6ac8569a-95ae-421e-aee0-edd189e18d11\") " pod="openshift-ingress-canary/ingress-canary-xbkzq" Oct 30 13:48:39.032039 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.030969 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f707b282-e507-475d-a82a-923c9bd1ff01-image-registry-private-configuration\") pod \"image-registry-6bc4d9765d-dp7xb\" (UID: \"f707b282-e507-475d-a82a-923c9bd1ff01\") " pod="openshift-image-registry/image-registry-6bc4d9765d-dp7xb" Oct 30 13:48:39.032039 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:39.031064 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 13:48:39.032039 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:39.031076 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bc4d9765d-dp7xb: secret "image-registry-tls" not found Oct 30 13:48:39.032877 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:39.031133 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f707b282-e507-475d-a82a-923c9bd1ff01-registry-tls podName:f707b282-e507-475d-a82a-923c9bd1ff01 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:39.531118421 +0000 UTC m=+29.550639814 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f707b282-e507-475d-a82a-923c9bd1ff01-registry-tls") pod "image-registry-6bc4d9765d-dp7xb" (UID: "f707b282-e507-475d-a82a-923c9bd1ff01") : secret "image-registry-tls" not found Oct 30 13:48:39.032877 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.031356 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dc967875-1546-40bb-8376-fde09416c204-trusted-ca\") pod \"console-operator-7cd56b8dbc-ntlqq\" (UID: \"dc967875-1546-40bb-8376-fde09416c204\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ntlqq" Oct 30 13:48:39.032877 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.031386 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6d73c6c3-044d-4b88-a105-72bd5e4491cd-stats-auth\") pod \"router-default-659879fb99-pjvcq\" (UID: \"6d73c6c3-044d-4b88-a105-72bd5e4491cd\") " pod="openshift-ingress/router-default-659879fb99-pjvcq" Oct 30 13:48:39.032877 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.031450 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e16ea4af-ce12-442b-91df-092457a7e192-registry-certificates\") pod \"image-registry-87f7c8d8c-mdvsm\" (UID: \"e16ea4af-ce12-442b-91df-092457a7e192\") " pod="openshift-image-registry/image-registry-87f7c8d8c-mdvsm" Oct 30 13:48:39.032877 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.031592 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6d73c6c3-044d-4b88-a105-72bd5e4491cd-default-certificate\") pod \"router-default-659879fb99-pjvcq\" (UID: \"6d73c6c3-044d-4b88-a105-72bd5e4491cd\") " pod="openshift-ingress/router-default-659879fb99-pjvcq" Oct 30 13:48:39.032877 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.031748 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f707b282-e507-475d-a82a-923c9bd1ff01-trusted-ca\") pod \"image-registry-6bc4d9765d-dp7xb\" (UID: \"f707b282-e507-475d-a82a-923c9bd1ff01\") " pod="openshift-image-registry/image-registry-6bc4d9765d-dp7xb" Oct 30 13:48:39.032877 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.031751 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e16ea4af-ce12-442b-91df-092457a7e192-installation-pull-secrets\") pod \"image-registry-87f7c8d8c-mdvsm\" (UID: \"e16ea4af-ce12-442b-91df-092457a7e192\") " pod="openshift-image-registry/image-registry-87f7c8d8c-mdvsm" Oct 30 13:48:39.033071 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.033019 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ade75be2-e3bb-4b50-bed0-c9c257739707-serving-cert\") pod \"service-ca-operator-544d749c8c-tn69c\" (UID: \"ade75be2-e3bb-4b50-bed0-c9c257739707\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tn69c" Oct 30 13:48:39.050581 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.050554 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzwmc\" (UniqueName: \"kubernetes.io/projected/ade75be2-e3bb-4b50-bed0-c9c257739707-kube-api-access-tzwmc\") pod \"service-ca-operator-544d749c8c-tn69c\" (UID: \"ade75be2-e3bb-4b50-bed0-c9c257739707\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tn69c" Oct 30 13:48:39.050817 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.050794 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l99vz\" (UniqueName: \"kubernetes.io/projected/cb90396e-d4e9-4eb0-888f-64c59d492fd4-kube-api-access-l99vz\") pod \"kas-connection-checker-685895c7c9-smvmg\" (UID: \"cb90396e-d4e9-4eb0-888f-64c59d492fd4\") " pod="kube-system/kas-connection-checker-685895c7c9-smvmg" Oct 30 13:48:39.051410 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.051385 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5qnd\" (UniqueName: \"kubernetes.io/projected/b0b2d906-7995-4769-8222-fa8ba2a30f3d-kube-api-access-t5qnd\") pod \"cluster-monitoring-operator-884576c57-9r9zr\" (UID: \"b0b2d906-7995-4769-8222-fa8ba2a30f3d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9r9zr" Oct 30 13:48:39.052054 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.051934 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfgzb\" (UniqueName: \"kubernetes.io/projected/6d73c6c3-044d-4b88-a105-72bd5e4491cd-kube-api-access-tfgzb\") pod \"router-default-659879fb99-pjvcq\" (UID: \"6d73c6c3-044d-4b88-a105-72bd5e4491cd\") " pod="openshift-ingress/router-default-659879fb99-pjvcq" Oct 30 13:48:39.052054 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.052004 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f707b282-e507-475d-a82a-923c9bd1ff01-bound-sa-token\") pod \"image-registry-6bc4d9765d-dp7xb\" (UID: \"f707b282-e507-475d-a82a-923c9bd1ff01\") " pod="openshift-image-registry/image-registry-6bc4d9765d-dp7xb" Oct 30 13:48:39.052195 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.052109 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwd54\" (UniqueName: \"kubernetes.io/projected/a4bd1bb2-25ae-423c-ba05-08eb742d340e-kube-api-access-pwd54\") pod \"insights-operator-7c4986d449-kcgbg\" (UID: \"a4bd1bb2-25ae-423c-ba05-08eb742d340e\") " pod="openshift-insights/insights-operator-7c4986d449-kcgbg" Oct 30 13:48:39.052260 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.052217 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tzg4\" (UniqueName: \"kubernetes.io/projected/8304f8da-d4a3-45c1-bfe6-f5d08bed17d7-kube-api-access-4tzg4\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2nsk6\" (UID: \"8304f8da-d4a3-45c1-bfe6-f5d08bed17d7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2nsk6" Oct 30 13:48:39.053558 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.053516 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69bm9\" (UniqueName: \"kubernetes.io/projected/6ac8569a-95ae-421e-aee0-edd189e18d11-kube-api-access-69bm9\") pod \"ingress-canary-xbkzq\" (UID: \"6ac8569a-95ae-421e-aee0-edd189e18d11\") " pod="openshift-ingress-canary/ingress-canary-xbkzq" Oct 30 13:48:39.053742 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.053720 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwgrr\" (UniqueName: \"kubernetes.io/projected/dad9381c-aebb-448d-80b1-e2b65dd8ac80-kube-api-access-rwgrr\") pod \"kas-connection-checker-685895c7c9-4h2sq\" (UID: \"dad9381c-aebb-448d-80b1-e2b65dd8ac80\") " pod="kube-system/kas-connection-checker-685895c7c9-4h2sq" Oct 30 13:48:39.053982 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.053959 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk8lh\" (UniqueName: \"kubernetes.io/projected/806dd609-9a2a-49c4-a656-e50437457fa9-kube-api-access-gk8lh\") pod \"network-check-source-6f78d59f99-bhw8w\" (UID: \"806dd609-9a2a-49c4-a656-e50437457fa9\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bhw8w" Oct 30 13:48:39.054062 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.054004 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm9mr\" (UniqueName: \"kubernetes.io/projected/f707b282-e507-475d-a82a-923c9bd1ff01-kube-api-access-vm9mr\") pod \"image-registry-6bc4d9765d-dp7xb\" (UID: \"f707b282-e507-475d-a82a-923c9bd1ff01\") " pod="openshift-image-registry/image-registry-6bc4d9765d-dp7xb" Oct 30 13:48:39.054481 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.054458 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spnhx\" (UniqueName: \"kubernetes.io/projected/81dcad0b-d124-41b8-8b4e-d41049de217a-kube-api-access-spnhx\") pod \"cluster-samples-operator-6ddb6c8b94-qstzd\" (UID: \"81dcad0b-d124-41b8-8b4e-d41049de217a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qstzd" Oct 30 13:48:39.055033 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.055015 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e16ea4af-ce12-442b-91df-092457a7e192-bound-sa-token\") pod \"image-registry-87f7c8d8c-mdvsm\" (UID: \"e16ea4af-ce12-442b-91df-092457a7e192\") " pod="openshift-image-registry/image-registry-87f7c8d8c-mdvsm" Oct 30 13:48:39.055719 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.055701 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx49w\" (UniqueName: \"kubernetes.io/projected/8d9908f6-aefa-4bde-9182-2c199f08a24a-kube-api-access-fx49w\") pod \"volume-data-source-validator-58c9f5f46d-brd96\" (UID: \"8d9908f6-aefa-4bde-9182-2c199f08a24a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-brd96" Oct 30 13:48:39.057621 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.057597 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p2xh\" (UniqueName: \"kubernetes.io/projected/083c70c2-0d5e-4876-9e88-10c6efbb681c-kube-api-access-6p2xh\") pod \"dns-default-7lx4t\" (UID: \"083c70c2-0d5e-4876-9e88-10c6efbb681c\") " pod="openshift-dns/dns-default-7lx4t" Oct 30 13:48:39.057818 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.057802 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkhbw\" (UniqueName: \"kubernetes.io/projected/d7a64da6-7b20-4f20-90d7-bc23a71290e5-kube-api-access-zkhbw\") pod \"kas-connection-checker-685895c7c9-plvgd\" (UID: \"d7a64da6-7b20-4f20-90d7-bc23a71290e5\") " pod="kube-system/kas-connection-checker-685895c7c9-plvgd" Oct 30 13:48:39.057897 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.057884 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v9rm\" (UniqueName: \"kubernetes.io/projected/e16ea4af-ce12-442b-91df-092457a7e192-kube-api-access-5v9rm\") pod \"image-registry-87f7c8d8c-mdvsm\" (UID: \"e16ea4af-ce12-442b-91df-092457a7e192\") " pod="openshift-image-registry/image-registry-87f7c8d8c-mdvsm" Oct 30 13:48:39.058402 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.058386 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n98n9\" (UniqueName: \"kubernetes.io/projected/dc967875-1546-40bb-8376-fde09416c204-kube-api-access-n98n9\") pod \"console-operator-7cd56b8dbc-ntlqq\" (UID: \"dc967875-1546-40bb-8376-fde09416c204\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ntlqq" Oct 30 13:48:39.125634 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.125616 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-4h2sq" Oct 30 13:48:39.135777 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.135756 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-smvmg" Oct 30 13:48:39.142542 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.142519 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-plvgd" Oct 30 13:48:39.150964 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.150946 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-kcgbg" Oct 30 13:48:39.197528 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.196535 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2nsk6" Oct 30 13:48:39.208678 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.206125 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-brd96" Oct 30 13:48:39.219761 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.219733 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tn69c" Oct 30 13:48:39.225722 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.225477 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-ntlqq" Oct 30 13:48:39.235742 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.235714 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bhw8w" Oct 30 13:48:39.251438 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.251267 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xbkzq" Oct 30 13:48:39.444613 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.444588 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-plvgd"] Oct 30 13:48:39.445495 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.445475 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-4h2sq"] Oct 30 13:48:39.446351 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:39.446329 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddad9381c_aebb_448d_80b1_e2b65dd8ac80.slice/crio-6ea406e47e234617b3a8cbb33024b7dd98c009263a6260d4f172f05267af5052 WatchSource:0}: Error finding container 6ea406e47e234617b3a8cbb33024b7dd98c009263a6260d4f172f05267af5052: Status 404 returned error can't find the container with id 6ea406e47e234617b3a8cbb33024b7dd98c009263a6260d4f172f05267af5052 Oct 30 13:48:39.447032 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:39.447013 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7a64da6_7b20_4f20_90d7_bc23a71290e5.slice/crio-06456fd9d542589957a0a372e8152f9ad26074321460971ecd510d034d5ec1ba WatchSource:0}: Error finding container 06456fd9d542589957a0a372e8152f9ad26074321460971ecd510d034d5ec1ba: Status 404 returned error can't find the container with id 06456fd9d542589957a0a372e8152f9ad26074321460971ecd510d034d5ec1ba Oct 30 13:48:39.465027 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.465006 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-kcgbg"] Oct 30 13:48:39.467662 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:39.467615 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4bd1bb2_25ae_423c_ba05_08eb742d340e.slice/crio-a7756810cb7a94ed1a71e06c72c0510df28c6247a04c0a76cce9756dea1229c5 WatchSource:0}: Error finding container a7756810cb7a94ed1a71e06c72c0510df28c6247a04c0a76cce9756dea1229c5: Status 404 returned error can't find the container with id a7756810cb7a94ed1a71e06c72c0510df28c6247a04c0a76cce9756dea1229c5 Oct 30 13:48:39.488294 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.488247 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-smvmg"] Oct 30 13:48:39.489084 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:39.489065 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb90396e_d4e9_4eb0_888f_64c59d492fd4.slice/crio-c140c8c08bb48e03b8d7fcb1e70922a318eb16dcf7ad807e73a6f51047faab0f WatchSource:0}: Error finding container c140c8c08bb48e03b8d7fcb1e70922a318eb16dcf7ad807e73a6f51047faab0f: Status 404 returned error can't find the container with id c140c8c08bb48e03b8d7fcb1e70922a318eb16dcf7ad807e73a6f51047faab0f Oct 30 13:48:39.533790 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.533755 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8309f073-dcb0-4971-9eb1-8a9ee0eb68c6-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2bnbl\" (UID: \"8309f073-dcb0-4971-9eb1-8a9ee0eb68c6\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2bnbl" Oct 30 13:48:39.533866 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.533837 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e16ea4af-ce12-442b-91df-092457a7e192-registry-tls\") pod \"image-registry-87f7c8d8c-mdvsm\" (UID: \"e16ea4af-ce12-442b-91df-092457a7e192\") " pod="openshift-image-registry/image-registry-87f7c8d8c-mdvsm" Oct 30 13:48:39.533925 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.533872 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/81dcad0b-d124-41b8-8b4e-d41049de217a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-qstzd\" (UID: \"81dcad0b-d124-41b8-8b4e-d41049de217a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qstzd" Oct 30 13:48:39.533925 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.533906 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d73c6c3-044d-4b88-a105-72bd5e4491cd-service-ca-bundle\") pod \"router-default-659879fb99-pjvcq\" (UID: \"6d73c6c3-044d-4b88-a105-72bd5e4491cd\") " pod="openshift-ingress/router-default-659879fb99-pjvcq" Oct 30 13:48:39.536757 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.534333 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b0b2d906-7995-4769-8222-fa8ba2a30f3d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-9r9zr\" (UID: \"b0b2d906-7995-4769-8222-fa8ba2a30f3d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9r9zr" Oct 30 13:48:39.536757 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:39.534374 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 13:48:39.536757 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.534385 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f707b282-e507-475d-a82a-923c9bd1ff01-registry-tls\") pod \"image-registry-6bc4d9765d-dp7xb\" (UID: \"f707b282-e507-475d-a82a-923c9bd1ff01\") " pod="openshift-image-registry/image-registry-6bc4d9765d-dp7xb" Oct 30 13:48:39.536757 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:39.534390 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-87f7c8d8c-mdvsm: secret "image-registry-tls" not found Oct 30 13:48:39.536757 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:39.534467 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e16ea4af-ce12-442b-91df-092457a7e192-registry-tls podName:e16ea4af-ce12-442b-91df-092457a7e192 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:40.53444779 +0000 UTC m=+30.553969181 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e16ea4af-ce12-442b-91df-092457a7e192-registry-tls") pod "image-registry-87f7c8d8c-mdvsm" (UID: "e16ea4af-ce12-442b-91df-092457a7e192") : secret "image-registry-tls" not found Oct 30 13:48:39.536757 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:39.534514 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 13:48:39.536757 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:39.534575 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8309f073-dcb0-4971-9eb1-8a9ee0eb68c6-networking-console-plugin-cert podName:8309f073-dcb0-4971-9eb1-8a9ee0eb68c6 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:40.534550697 +0000 UTC m=+30.554072084 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8309f073-dcb0-4971-9eb1-8a9ee0eb68c6-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-2bnbl" (UID: "8309f073-dcb0-4971-9eb1-8a9ee0eb68c6") : secret "networking-console-plugin-cert" not found Oct 30 13:48:39.536757 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.534628 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d73c6c3-044d-4b88-a105-72bd5e4491cd-metrics-certs\") pod \"router-default-659879fb99-pjvcq\" (UID: \"6d73c6c3-044d-4b88-a105-72bd5e4491cd\") " pod="openshift-ingress/router-default-659879fb99-pjvcq" Oct 30 13:48:39.536757 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.534737 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/083c70c2-0d5e-4876-9e88-10c6efbb681c-metrics-tls\") pod \"dns-default-7lx4t\" (UID: \"083c70c2-0d5e-4876-9e88-10c6efbb681c\") " pod="openshift-dns/dns-default-7lx4t" Oct 30 13:48:39.536757 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:39.534665 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 13:48:39.536757 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:39.534768 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6d73c6c3-044d-4b88-a105-72bd5e4491cd-service-ca-bundle podName:6d73c6c3-044d-4b88-a105-72bd5e4491cd nodeName:}" failed. No retries permitted until 2026-10-30 13:48:40.534735547 +0000 UTC m=+30.554256940 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6d73c6c3-044d-4b88-a105-72bd5e4491cd-service-ca-bundle") pod "router-default-659879fb99-pjvcq" (UID: "6d73c6c3-044d-4b88-a105-72bd5e4491cd") : configmap references non-existent config key: service-ca.crt Oct 30 13:48:39.536757 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:39.534837 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 13:48:39.536757 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:39.534844 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 13:48:39.536757 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:39.535011 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81dcad0b-d124-41b8-8b4e-d41049de217a-samples-operator-tls podName:81dcad0b-d124-41b8-8b4e-d41049de217a nodeName:}" failed. No retries permitted until 2026-10-30 13:48:40.534987 +0000 UTC m=+30.554508390 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/81dcad0b-d124-41b8-8b4e-d41049de217a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-qstzd" (UID: "81dcad0b-d124-41b8-8b4e-d41049de217a") : secret "samples-operator-tls" not found Oct 30 13:48:39.536757 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:39.534901 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 13:48:39.536757 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:39.535032 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/083c70c2-0d5e-4876-9e88-10c6efbb681c-metrics-tls podName:083c70c2-0d5e-4876-9e88-10c6efbb681c nodeName:}" failed. No retries permitted until 2026-10-30 13:48:40.535022025 +0000 UTC m=+30.554543417 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/083c70c2-0d5e-4876-9e88-10c6efbb681c-metrics-tls") pod "dns-default-7lx4t" (UID: "083c70c2-0d5e-4876-9e88-10c6efbb681c") : secret "dns-default-metrics-tls" not found Oct 30 13:48:39.536757 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:39.534906 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 13:48:39.537531 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:39.535054 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0b2d906-7995-4769-8222-fa8ba2a30f3d-cluster-monitoring-operator-tls podName:b0b2d906-7995-4769-8222-fa8ba2a30f3d nodeName:}" failed. No retries permitted until 2026-10-30 13:48:40.535040324 +0000 UTC m=+30.554561708 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b0b2d906-7995-4769-8222-fa8ba2a30f3d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-9r9zr" (UID: "b0b2d906-7995-4769-8222-fa8ba2a30f3d") : secret "cluster-monitoring-operator-tls" not found Oct 30 13:48:39.537531 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:39.535058 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bc4d9765d-dp7xb: secret "image-registry-tls" not found Oct 30 13:48:39.537531 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:39.535093 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d73c6c3-044d-4b88-a105-72bd5e4491cd-metrics-certs podName:6d73c6c3-044d-4b88-a105-72bd5e4491cd nodeName:}" failed. No retries permitted until 2026-10-30 13:48:40.535061972 +0000 UTC m=+30.554583350 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d73c6c3-044d-4b88-a105-72bd5e4491cd-metrics-certs") pod "router-default-659879fb99-pjvcq" (UID: "6d73c6c3-044d-4b88-a105-72bd5e4491cd") : secret "router-metrics-certs-default" not found Oct 30 13:48:39.537531 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:39.535184 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f707b282-e507-475d-a82a-923c9bd1ff01-registry-tls podName:f707b282-e507-475d-a82a-923c9bd1ff01 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:40.535174286 +0000 UTC m=+30.554695667 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f707b282-e507-475d-a82a-923c9bd1ff01-registry-tls") pod "image-registry-6bc4d9765d-dp7xb" (UID: "f707b282-e507-475d-a82a-923c9bd1ff01") : secret "image-registry-tls" not found Oct 30 13:48:39.548212 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.548191 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-brd96"] Oct 30 13:48:39.552151 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.550006 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2nsk6"] Oct 30 13:48:39.585704 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.585683 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xbkzq"] Oct 30 13:48:39.587335 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:39.587306 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ac8569a_95ae_421e_aee0_edd189e18d11.slice/crio-b7fb2959d73fac358b070abb945abc1d6f0c75a2833392da16f4e23a1b623d0f WatchSource:0}: Error finding container b7fb2959d73fac358b070abb945abc1d6f0c75a2833392da16f4e23a1b623d0f: Status 404 returned error can't find the container with id b7fb2959d73fac358b070abb945abc1d6f0c75a2833392da16f4e23a1b623d0f Oct 30 13:48:39.589026 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.588947 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-ntlqq"] Oct 30 13:48:39.594539 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:39.594366 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc967875_1546_40bb_8376_fde09416c204.slice/crio-34d6135192daf776d327c4e638d6497d104f8174e400915f3e099f255171a116 WatchSource:0}: Error finding container 34d6135192daf776d327c4e638d6497d104f8174e400915f3e099f255171a116: Status 404 returned error can't find the container with id 34d6135192daf776d327c4e638d6497d104f8174e400915f3e099f255171a116 Oct 30 13:48:39.595182 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.595117 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-bhw8w"] Oct 30 13:48:39.598817 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.598180 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-tn69c"] Oct 30 13:48:39.701881 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.701857 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xbkzq" event={"ID":"6ac8569a-95ae-421e-aee0-edd189e18d11","Type":"ContainerStarted","Data":"b7fb2959d73fac358b070abb945abc1d6f0c75a2833392da16f4e23a1b623d0f"} Oct 30 13:48:39.703272 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.703253 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-plvgd" event={"ID":"d7a64da6-7b20-4f20-90d7-bc23a71290e5","Type":"ContainerStarted","Data":"d09819d8296a3055cf057bb42d6292dc501e6eca8e80470e66fe8b79c63837e6"} Oct 30 13:48:39.703335 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.703295 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-plvgd" event={"ID":"d7a64da6-7b20-4f20-90d7-bc23a71290e5","Type":"ContainerStarted","Data":"06456fd9d542589957a0a372e8152f9ad26074321460971ecd510d034d5ec1ba"} Oct 30 13:48:39.704542 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.704522 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-4h2sq" event={"ID":"dad9381c-aebb-448d-80b1-e2b65dd8ac80","Type":"ContainerStarted","Data":"4e79edccb36c8253990c4e98780bfeb846b177256486b8c319be0fffb9438d30"} Oct 30 13:48:39.704622 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.704559 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-4h2sq" event={"ID":"dad9381c-aebb-448d-80b1-e2b65dd8ac80","Type":"ContainerStarted","Data":"6ea406e47e234617b3a8cbb33024b7dd98c009263a6260d4f172f05267af5052"} Oct 30 13:48:39.705556 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.705537 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bhw8w" event={"ID":"806dd609-9a2a-49c4-a656-e50437457fa9","Type":"ContainerStarted","Data":"dd0e127ce14f31020f149eb6a7d374f34ef66f6316e05086093e2736a2578e63"} Oct 30 13:48:39.706552 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.706534 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-ntlqq" event={"ID":"dc967875-1546-40bb-8376-fde09416c204","Type":"ContainerStarted","Data":"34d6135192daf776d327c4e638d6497d104f8174e400915f3e099f255171a116"} Oct 30 13:48:39.707518 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.707499 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tn69c" event={"ID":"ade75be2-e3bb-4b50-bed0-c9c257739707","Type":"ContainerStarted","Data":"e6bed7b02e02be3eccc0a1bb65bb2b401caacedbb61032e96df1197725b262f1"} Oct 30 13:48:39.708505 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.708479 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-brd96" event={"ID":"8d9908f6-aefa-4bde-9182-2c199f08a24a","Type":"ContainerStarted","Data":"5e0a45a241a6763de3ae2fcbb368e0abe7bbbf667c4256b2a91200025b1cbb18"} Oct 30 13:48:39.709578 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.709554 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-kcgbg" event={"ID":"a4bd1bb2-25ae-423c-ba05-08eb742d340e","Type":"ContainerStarted","Data":"a7756810cb7a94ed1a71e06c72c0510df28c6247a04c0a76cce9756dea1229c5"} Oct 30 13:48:39.710531 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.710514 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2nsk6" event={"ID":"8304f8da-d4a3-45c1-bfe6-f5d08bed17d7","Type":"ContainerStarted","Data":"4f386b3d868858e3f05c1e4c51e74f8b9046128e43e4a207cd41e454a06e6f1f"} Oct 30 13:48:39.711868 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.711850 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-smvmg" event={"ID":"cb90396e-d4e9-4eb0-888f-64c59d492fd4","Type":"ContainerStarted","Data":"5157e9115cd500faaab63cf37bf633118064a9900459240fb5292502e36fa2b3"} Oct 30 13:48:39.711937 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.711879 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-smvmg" event={"ID":"cb90396e-d4e9-4eb0-888f-64c59d492fd4","Type":"ContainerStarted","Data":"c140c8c08bb48e03b8d7fcb1e70922a318eb16dcf7ad807e73a6f51047faab0f"} Oct 30 13:48:39.749252 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.749206 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-plvgd" podStartSLOduration=286.749194902 podStartE2EDuration="4m46.749194902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:43:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:39.748081329 +0000 UTC m=+29.767602725" watchObservedRunningTime="2026-10-30 13:48:39.749194902 +0000 UTC m=+29.768716299" Oct 30 13:48:39.792136 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:39.792093 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-smvmg" podStartSLOduration=286.792084231 podStartE2EDuration="4m46.792084231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:43:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:39.791789268 +0000 UTC m=+29.811310667" watchObservedRunningTime="2026-10-30 13:48:39.792084231 +0000 UTC m=+29.811605622" Oct 30 13:48:40.444871 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:40.444317 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dx94q" Oct 30 13:48:40.444871 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:40.444513 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfg7x" Oct 30 13:48:40.445430 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:40.444921 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdmgn" Oct 30 13:48:40.464174 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:40.463023 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-smj8j\"" Oct 30 13:48:40.464174 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:40.463083 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 13:48:40.464174 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:40.463718 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 13:48:40.464174 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:40.463843 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-w8shz\"" Oct 30 13:48:40.464174 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:40.463928 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-gw59s\"" Oct 30 13:48:40.545155 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:40.545109 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/81dcad0b-d124-41b8-8b4e-d41049de217a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-qstzd\" (UID: \"81dcad0b-d124-41b8-8b4e-d41049de217a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qstzd" Oct 30 13:48:40.545285 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:40.545169 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d73c6c3-044d-4b88-a105-72bd5e4491cd-service-ca-bundle\") pod \"router-default-659879fb99-pjvcq\" (UID: \"6d73c6c3-044d-4b88-a105-72bd5e4491cd\") " pod="openshift-ingress/router-default-659879fb99-pjvcq" Oct 30 13:48:40.545285 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:40.545207 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b0b2d906-7995-4769-8222-fa8ba2a30f3d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-9r9zr\" (UID: \"b0b2d906-7995-4769-8222-fa8ba2a30f3d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9r9zr" Oct 30 13:48:40.545285 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:40.545234 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f707b282-e507-475d-a82a-923c9bd1ff01-registry-tls\") pod \"image-registry-6bc4d9765d-dp7xb\" (UID: \"f707b282-e507-475d-a82a-923c9bd1ff01\") " pod="openshift-image-registry/image-registry-6bc4d9765d-dp7xb" Oct 30 13:48:40.545285 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:40.545260 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d73c6c3-044d-4b88-a105-72bd5e4491cd-metrics-certs\") pod \"router-default-659879fb99-pjvcq\" (UID: \"6d73c6c3-044d-4b88-a105-72bd5e4491cd\") " pod="openshift-ingress/router-default-659879fb99-pjvcq" Oct 30 13:48:40.545479 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:40.545320 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/083c70c2-0d5e-4876-9e88-10c6efbb681c-metrics-tls\") pod \"dns-default-7lx4t\" (UID: \"083c70c2-0d5e-4876-9e88-10c6efbb681c\") " pod="openshift-dns/dns-default-7lx4t" Oct 30 13:48:40.545479 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:40.545368 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8309f073-dcb0-4971-9eb1-8a9ee0eb68c6-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2bnbl\" (UID: \"8309f073-dcb0-4971-9eb1-8a9ee0eb68c6\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2bnbl" Oct 30 13:48:40.545479 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:40.545429 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e16ea4af-ce12-442b-91df-092457a7e192-registry-tls\") pod \"image-registry-87f7c8d8c-mdvsm\" (UID: \"e16ea4af-ce12-442b-91df-092457a7e192\") " pod="openshift-image-registry/image-registry-87f7c8d8c-mdvsm" Oct 30 13:48:40.545604 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:40.545549 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 13:48:40.545604 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:40.545563 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-87f7c8d8c-mdvsm: secret "image-registry-tls" not found Oct 30 13:48:40.545719 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:40.545614 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e16ea4af-ce12-442b-91df-092457a7e192-registry-tls podName:e16ea4af-ce12-442b-91df-092457a7e192 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:42.545596948 +0000 UTC m=+32.565118339 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e16ea4af-ce12-442b-91df-092457a7e192-registry-tls") pod "image-registry-87f7c8d8c-mdvsm" (UID: "e16ea4af-ce12-442b-91df-092457a7e192") : secret "image-registry-tls" not found Oct 30 13:48:40.546025 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:40.546001 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 13:48:40.546135 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:40.546051 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81dcad0b-d124-41b8-8b4e-d41049de217a-samples-operator-tls podName:81dcad0b-d124-41b8-8b4e-d41049de217a nodeName:}" failed. No retries permitted until 2026-10-30 13:48:42.546037954 +0000 UTC m=+32.565559332 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/81dcad0b-d124-41b8-8b4e-d41049de217a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-qstzd" (UID: "81dcad0b-d124-41b8-8b4e-d41049de217a") : secret "samples-operator-tls" not found Oct 30 13:48:40.546135 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:40.546114 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6d73c6c3-044d-4b88-a105-72bd5e4491cd-service-ca-bundle podName:6d73c6c3-044d-4b88-a105-72bd5e4491cd nodeName:}" failed. No retries permitted until 2026-10-30 13:48:42.546104612 +0000 UTC m=+32.565625993 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6d73c6c3-044d-4b88-a105-72bd5e4491cd-service-ca-bundle") pod "router-default-659879fb99-pjvcq" (UID: "6d73c6c3-044d-4b88-a105-72bd5e4491cd") : configmap references non-existent config key: service-ca.crt Oct 30 13:48:40.546250 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:40.546168 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 13:48:40.546250 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:40.546200 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0b2d906-7995-4769-8222-fa8ba2a30f3d-cluster-monitoring-operator-tls podName:b0b2d906-7995-4769-8222-fa8ba2a30f3d nodeName:}" failed. No retries permitted until 2026-10-30 13:48:42.54619089 +0000 UTC m=+32.565712282 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b0b2d906-7995-4769-8222-fa8ba2a30f3d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-9r9zr" (UID: "b0b2d906-7995-4769-8222-fa8ba2a30f3d") : secret "cluster-monitoring-operator-tls" not found Oct 30 13:48:40.546349 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:40.546254 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 13:48:40.546349 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:40.546264 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bc4d9765d-dp7xb: secret "image-registry-tls" not found Oct 30 13:48:40.546349 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:40.546293 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f707b282-e507-475d-a82a-923c9bd1ff01-registry-tls podName:f707b282-e507-475d-a82a-923c9bd1ff01 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:42.546283451 +0000 UTC m=+32.565804844 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f707b282-e507-475d-a82a-923c9bd1ff01-registry-tls") pod "image-registry-6bc4d9765d-dp7xb" (UID: "f707b282-e507-475d-a82a-923c9bd1ff01") : secret "image-registry-tls" not found Oct 30 13:48:40.546349 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:40.546342 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 13:48:40.546535 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:40.546374 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d73c6c3-044d-4b88-a105-72bd5e4491cd-metrics-certs podName:6d73c6c3-044d-4b88-a105-72bd5e4491cd nodeName:}" failed. No retries permitted until 2026-10-30 13:48:42.546361999 +0000 UTC m=+32.565883378 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d73c6c3-044d-4b88-a105-72bd5e4491cd-metrics-certs") pod "router-default-659879fb99-pjvcq" (UID: "6d73c6c3-044d-4b88-a105-72bd5e4491cd") : secret "router-metrics-certs-default" not found Oct 30 13:48:40.546535 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:40.546427 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 13:48:40.546535 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:40.546454 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/083c70c2-0d5e-4876-9e88-10c6efbb681c-metrics-tls podName:083c70c2-0d5e-4876-9e88-10c6efbb681c nodeName:}" failed. No retries permitted until 2026-10-30 13:48:42.54644554 +0000 UTC m=+32.565966933 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/083c70c2-0d5e-4876-9e88-10c6efbb681c-metrics-tls") pod "dns-default-7lx4t" (UID: "083c70c2-0d5e-4876-9e88-10c6efbb681c") : secret "dns-default-metrics-tls" not found Oct 30 13:48:40.546535 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:40.546501 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 13:48:40.546535 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:40.546530 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8309f073-dcb0-4971-9eb1-8a9ee0eb68c6-networking-console-plugin-cert podName:8309f073-dcb0-4971-9eb1-8a9ee0eb68c6 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:42.546521002 +0000 UTC m=+32.566042394 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8309f073-dcb0-4971-9eb1-8a9ee0eb68c6-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-2bnbl" (UID: "8309f073-dcb0-4971-9eb1-8a9ee0eb68c6") : secret "networking-console-plugin-cert" not found Oct 30 13:48:40.802198 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:40.802042 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:41.003257 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:41.003218 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:41.362686 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:41.361919 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:41.363825 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:41.363121 2580 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 13:48:41.410318 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:41.409957 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:41.505294 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:41.504730 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-4h2sq" podStartSLOduration=288.50471196 podStartE2EDuration="4m48.50471196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:43:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:39.834725273 +0000 UTC m=+29.854246672" watchObservedRunningTime="2026-10-30 13:48:41.50471196 +0000 UTC m=+31.524233342" Oct 30 13:48:42.568284 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:42.566448 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/083c70c2-0d5e-4876-9e88-10c6efbb681c-metrics-tls\") pod \"dns-default-7lx4t\" (UID: \"083c70c2-0d5e-4876-9e88-10c6efbb681c\") " pod="openshift-dns/dns-default-7lx4t" Oct 30 13:48:42.568284 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:42.566514 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8309f073-dcb0-4971-9eb1-8a9ee0eb68c6-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2bnbl\" (UID: \"8309f073-dcb0-4971-9eb1-8a9ee0eb68c6\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2bnbl" Oct 30 13:48:42.568284 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:42.566577 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e16ea4af-ce12-442b-91df-092457a7e192-registry-tls\") pod \"image-registry-87f7c8d8c-mdvsm\" (UID: \"e16ea4af-ce12-442b-91df-092457a7e192\") " pod="openshift-image-registry/image-registry-87f7c8d8c-mdvsm" Oct 30 13:48:42.568284 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:42.566609 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/81dcad0b-d124-41b8-8b4e-d41049de217a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-qstzd\" (UID: \"81dcad0b-d124-41b8-8b4e-d41049de217a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qstzd" Oct 30 13:48:42.568284 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:42.566659 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d73c6c3-044d-4b88-a105-72bd5e4491cd-service-ca-bundle\") pod \"router-default-659879fb99-pjvcq\" (UID: \"6d73c6c3-044d-4b88-a105-72bd5e4491cd\") " pod="openshift-ingress/router-default-659879fb99-pjvcq" Oct 30 13:48:42.568284 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:42.566697 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b0b2d906-7995-4769-8222-fa8ba2a30f3d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-9r9zr\" (UID: \"b0b2d906-7995-4769-8222-fa8ba2a30f3d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9r9zr" Oct 30 13:48:42.568284 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:42.566721 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f707b282-e507-475d-a82a-923c9bd1ff01-registry-tls\") pod \"image-registry-6bc4d9765d-dp7xb\" (UID: \"f707b282-e507-475d-a82a-923c9bd1ff01\") " pod="openshift-image-registry/image-registry-6bc4d9765d-dp7xb" Oct 30 13:48:42.568284 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:42.566743 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d73c6c3-044d-4b88-a105-72bd5e4491cd-metrics-certs\") pod \"router-default-659879fb99-pjvcq\" (UID: \"6d73c6c3-044d-4b88-a105-72bd5e4491cd\") " pod="openshift-ingress/router-default-659879fb99-pjvcq" Oct 30 13:48:42.568284 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:42.566864 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 13:48:42.568284 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:42.566920 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d73c6c3-044d-4b88-a105-72bd5e4491cd-metrics-certs podName:6d73c6c3-044d-4b88-a105-72bd5e4491cd nodeName:}" failed. No retries permitted until 2026-10-30 13:48:46.566905104 +0000 UTC m=+36.586426500 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d73c6c3-044d-4b88-a105-72bd5e4491cd-metrics-certs") pod "router-default-659879fb99-pjvcq" (UID: "6d73c6c3-044d-4b88-a105-72bd5e4491cd") : secret "router-metrics-certs-default" not found Oct 30 13:48:42.568284 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:42.567044 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 13:48:42.568284 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:42.567076 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/083c70c2-0d5e-4876-9e88-10c6efbb681c-metrics-tls podName:083c70c2-0d5e-4876-9e88-10c6efbb681c nodeName:}" failed. No retries permitted until 2026-10-30 13:48:46.567064578 +0000 UTC m=+36.586585969 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/083c70c2-0d5e-4876-9e88-10c6efbb681c-metrics-tls") pod "dns-default-7lx4t" (UID: "083c70c2-0d5e-4876-9e88-10c6efbb681c") : secret "dns-default-metrics-tls" not found Oct 30 13:48:42.568284 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:42.567125 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6d73c6c3-044d-4b88-a105-72bd5e4491cd-service-ca-bundle podName:6d73c6c3-044d-4b88-a105-72bd5e4491cd nodeName:}" failed. No retries permitted until 2026-10-30 13:48:46.567118814 +0000 UTC m=+36.586640206 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6d73c6c3-044d-4b88-a105-72bd5e4491cd-service-ca-bundle") pod "router-default-659879fb99-pjvcq" (UID: "6d73c6c3-044d-4b88-a105-72bd5e4491cd") : configmap references non-existent config key: service-ca.crt Oct 30 13:48:42.568284 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:42.567164 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 13:48:42.568284 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:42.567194 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8309f073-dcb0-4971-9eb1-8a9ee0eb68c6-networking-console-plugin-cert podName:8309f073-dcb0-4971-9eb1-8a9ee0eb68c6 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:46.567186709 +0000 UTC m=+36.586708100 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8309f073-dcb0-4971-9eb1-8a9ee0eb68c6-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-2bnbl" (UID: "8309f073-dcb0-4971-9eb1-8a9ee0eb68c6") : secret "networking-console-plugin-cert" not found Oct 30 13:48:42.569430 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:42.567235 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 13:48:42.569430 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:42.567243 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-87f7c8d8c-mdvsm: secret "image-registry-tls" not found Oct 30 13:48:42.569430 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:42.567262 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e16ea4af-ce12-442b-91df-092457a7e192-registry-tls podName:e16ea4af-ce12-442b-91df-092457a7e192 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:46.567255285 +0000 UTC m=+36.586776676 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e16ea4af-ce12-442b-91df-092457a7e192-registry-tls") pod "image-registry-87f7c8d8c-mdvsm" (UID: "e16ea4af-ce12-442b-91df-092457a7e192") : secret "image-registry-tls" not found Oct 30 13:48:42.569430 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:42.567306 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 13:48:42.569430 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:42.567329 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81dcad0b-d124-41b8-8b4e-d41049de217a-samples-operator-tls podName:81dcad0b-d124-41b8-8b4e-d41049de217a nodeName:}" failed. No retries permitted until 2026-10-30 13:48:46.567323017 +0000 UTC m=+36.586844410 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/81dcad0b-d124-41b8-8b4e-d41049de217a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-qstzd" (UID: "81dcad0b-d124-41b8-8b4e-d41049de217a") : secret "samples-operator-tls" not found Oct 30 13:48:42.569430 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:42.567415 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 13:48:42.569430 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:42.567435 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bc4d9765d-dp7xb: secret "image-registry-tls" not found Oct 30 13:48:42.569430 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:42.567484 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f707b282-e507-475d-a82a-923c9bd1ff01-registry-tls podName:f707b282-e507-475d-a82a-923c9bd1ff01 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:46.567469523 +0000 UTC m=+36.586990926 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f707b282-e507-475d-a82a-923c9bd1ff01-registry-tls") pod "image-registry-6bc4d9765d-dp7xb" (UID: "f707b282-e507-475d-a82a-923c9bd1ff01") : secret "image-registry-tls" not found Oct 30 13:48:42.569430 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:42.567496 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 13:48:42.569430 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:42.567689 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0b2d906-7995-4769-8222-fa8ba2a30f3d-cluster-monitoring-operator-tls podName:b0b2d906-7995-4769-8222-fa8ba2a30f3d nodeName:}" failed. No retries permitted until 2026-10-30 13:48:46.567670875 +0000 UTC m=+36.587192265 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b0b2d906-7995-4769-8222-fa8ba2a30f3d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-9r9zr" (UID: "b0b2d906-7995-4769-8222-fa8ba2a30f3d") : secret "cluster-monitoring-operator-tls" not found Oct 30 13:48:42.630421 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:42.630348 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:42.751341 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:42.751303 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-brd96" event={"ID":"8d9908f6-aefa-4bde-9182-2c199f08a24a","Type":"ContainerStarted","Data":"163ab782144185e5357d12223673e8e81ae7127e2a647db97cb20f1011782cd9"} Oct 30 13:48:42.774777 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:42.774708 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:42.796583 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:42.796142 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-brd96" podStartSLOduration=255.589244701 podStartE2EDuration="4m17.796125679s" totalImagesPullingTime="2.206880978s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:44:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:42.794217754 +0000 UTC m=+32.813739155" watchObservedRunningTime="2026-10-30 13:48:42.796125679 +0000 UTC m=+32.815647077" Oct 30 13:48:42.944719 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:42.944683 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:43.425702 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:43.425583 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:43.592731 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:43.592653 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:43.760070 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:43.760049 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ntlqq_dc967875-1546-40bb-8376-fde09416c204/console-operator/0.log" Oct 30 13:48:43.760188 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:43.760084 2580 generic.go:356] "Generic (PLEG): container finished" podID="dc967875-1546-40bb-8376-fde09416c204" containerID="2d36dbd075fd576e37a0be17ddf28abce3e7081abf2564964f056b77228ed483" exitCode=255 Oct 30 13:48:43.760700 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:43.760683 2580 scope.go:122] "RemoveContainer" containerID="2d36dbd075fd576e37a0be17ddf28abce3e7081abf2564964f056b77228ed483" Oct 30 13:48:43.760796 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:43.760701 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-ntlqq" event={"ID":"dc967875-1546-40bb-8376-fde09416c204","Type":"ContainerDied","Data":"2d36dbd075fd576e37a0be17ddf28abce3e7081abf2564964f056b77228ed483"} Oct 30 13:48:44.285297 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:44.285270 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb8hg\" (UniqueName: \"kubernetes.io/projected/bf73153f-1421-44d6-abae-b76c3d90235e-kube-api-access-mb8hg\") pod \"network-check-target-dx94q\" (UID: \"bf73153f-1421-44d6-abae-b76c3d90235e\") " pod="openshift-network-diagnostics/network-check-target-dx94q" Oct 30 13:48:44.285398 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:44.285320 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da694c74-3bce-4d95-bd30-12a1891ad5e6-metrics-certs\") pod \"network-metrics-daemon-pfg7x\" (UID: \"da694c74-3bce-4d95-bd30-12a1891ad5e6\") " pod="openshift-multus/network-metrics-daemon-pfg7x" Oct 30 13:48:44.285460 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:44.285395 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b5e74da-5460-44ed-a599-b01620b64c92-original-pull-secret\") pod \"global-pull-secret-syncer-bdmgn\" (UID: \"6b5e74da-5460-44ed-a599-b01620b64c92\") " pod="kube-system/global-pull-secret-syncer-bdmgn" Oct 30 13:48:44.285590 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:44.285501 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 13:48:44.285590 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:44.285560 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da694c74-3bce-4d95-bd30-12a1891ad5e6-metrics-certs podName:da694c74-3bce-4d95-bd30-12a1891ad5e6 nodeName:}" failed. No retries permitted until 2026-10-30 13:49:16.285546098 +0000 UTC m=+66.305067489 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/da694c74-3bce-4d95-bd30-12a1891ad5e6-metrics-certs") pod "network-metrics-daemon-pfg7x" (UID: "da694c74-3bce-4d95-bd30-12a1891ad5e6") : secret "metrics-daemon-secret" not found Oct 30 13:48:44.289527 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:44.289240 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb8hg\" (UniqueName: \"kubernetes.io/projected/bf73153f-1421-44d6-abae-b76c3d90235e-kube-api-access-mb8hg\") pod \"network-check-target-dx94q\" (UID: \"bf73153f-1421-44d6-abae-b76c3d90235e\") " pod="openshift-network-diagnostics/network-check-target-dx94q" Oct 30 13:48:44.289527 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:44.289498 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b5e74da-5460-44ed-a599-b01620b64c92-original-pull-secret\") pod \"global-pull-secret-syncer-bdmgn\" (UID: \"6b5e74da-5460-44ed-a599-b01620b64c92\") " pod="kube-system/global-pull-secret-syncer-bdmgn" Oct 30 13:48:44.394767 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:44.394737 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dx94q" Oct 30 13:48:44.417184 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:44.417160 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdmgn" Oct 30 13:48:44.750350 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:44.750323 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:44.759174 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:44.759143 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-whnh9"] Oct 30 13:48:44.766271 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:44.766251 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ntlqq_dc967875-1546-40bb-8376-fde09416c204/console-operator/1.log" Oct 30 13:48:44.766784 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:44.766690 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ntlqq_dc967875-1546-40bb-8376-fde09416c204/console-operator/0.log" Oct 30 13:48:44.766784 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:44.766722 2580 generic.go:356] "Generic (PLEG): container finished" podID="dc967875-1546-40bb-8376-fde09416c204" containerID="08dafe630716a5a619c9abe02a35bcce9e3b7cb581311f535ccb7aff3f8803cb" exitCode=255 Oct 30 13:48:44.767101 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:44.767082 2580 scope.go:122] "RemoveContainer" containerID="08dafe630716a5a619c9abe02a35bcce9e3b7cb581311f535ccb7aff3f8803cb" Oct 30 13:48:44.767321 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:44.767300 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-ntlqq_openshift-console-operator(dc967875-1546-40bb-8376-fde09416c204)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-ntlqq" podUID="dc967875-1546-40bb-8376-fde09416c204" Oct 30 13:48:44.783330 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:44.783260 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-v2x2w"] Oct 30 13:48:44.783448 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:44.783360 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-whnh9" Oct 30 13:48:44.803193 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:44.803061 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-7kqhp\"" Oct 30 13:48:44.803193 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:44.803133 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 13:48:44.803193 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:44.803166 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 13:48:44.803601 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:44.803582 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-whnh9"] Oct 30 13:48:44.803677 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:44.803616 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-v2x2w"] Oct 30 13:48:44.803677 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:44.803632 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xbkzq" event={"ID":"6ac8569a-95ae-421e-aee0-edd189e18d11","Type":"ContainerStarted","Data":"dcb7aecd81cc603e90d1dcdac0430082fa02ce1a8b2cf701772e299471e898d3"} Oct 30 13:48:44.803760 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:44.803677 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-ntlqq" event={"ID":"dc967875-1546-40bb-8376-fde09416c204","Type":"ContainerDied","Data":"08dafe630716a5a619c9abe02a35bcce9e3b7cb581311f535ccb7aff3f8803cb"} Oct 30 13:48:44.803760 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:44.803696 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tn69c" event={"ID":"ade75be2-e3bb-4b50-bed0-c9c257739707","Type":"ContainerStarted","Data":"a6b7c298ab2f474b66c97b7a60bd466dc4ef705ba5497913a3a6f7200a9e66a4"} Oct 30 13:48:44.803760 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:44.803710 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-kcgbg" event={"ID":"a4bd1bb2-25ae-423c-ba05-08eb742d340e","Type":"ContainerStarted","Data":"1c78b570941379eccb1af6dee987cc38964001897338ac0d0ac628f5ed9e5190"} Oct 30 13:48:44.803760 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:44.803722 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2nsk6" event={"ID":"8304f8da-d4a3-45c1-bfe6-f5d08bed17d7","Type":"ContainerStarted","Data":"cf977f7a9365f43c6bbab4365b6941118e3b94afb26b89a5a377f8b8f72c7f78"} Oct 30 13:48:44.803952 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:44.803788 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-v2x2w" Oct 30 13:48:44.804043 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:44.804015 2580 scope.go:122] "RemoveContainer" containerID="2d36dbd075fd576e37a0be17ddf28abce3e7081abf2564964f056b77228ed483" Oct 30 13:48:44.893994 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:44.893921 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhdhm\" (UniqueName: \"kubernetes.io/projected/ccafe9ae-44d7-4c31-a6a5-ba8506bc0cea-kube-api-access-xhdhm\") pod \"migrator-66df5d496b-v2x2w\" (UID: \"ccafe9ae-44d7-4c31-a6a5-ba8506bc0cea\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-v2x2w" Oct 30 13:48:44.894363 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:44.894336 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srtk8\" (UniqueName: \"kubernetes.io/projected/7cd2b593-c362-43c2-b0bb-7e5021791214-kube-api-access-srtk8\") pod \"migrator-66df5d496b-whnh9\" (UID: \"7cd2b593-c362-43c2-b0bb-7e5021791214\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-whnh9" Oct 30 13:48:44.905497 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:44.905465 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:44.914261 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:44.914238 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dx94q"] Oct 30 13:48:44.916987 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:44.916939 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2nsk6" podStartSLOduration=255.730101459 podStartE2EDuration="4m19.916922349s" totalImagesPullingTime="4.18682089s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:44:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:44.91599978 +0000 UTC m=+34.935521180" watchObservedRunningTime="2026-10-30 13:48:44.916922349 +0000 UTC m=+34.936443750" Oct 30 13:48:44.929264 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:44.929245 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bdmgn"] Oct 30 13:48:44.967184 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:44.967130 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tn69c" podStartSLOduration=255.329295161 podStartE2EDuration="4m19.967113944s" totalImagesPullingTime="4.637818783s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:44:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:44.96635325 +0000 UTC m=+34.985874652" watchObservedRunningTime="2026-10-30 13:48:44.967113944 +0000 UTC m=+34.986635345" Oct 30 13:48:44.995759 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:44.995724 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srtk8\" (UniqueName: \"kubernetes.io/projected/7cd2b593-c362-43c2-b0bb-7e5021791214-kube-api-access-srtk8\") pod \"migrator-66df5d496b-whnh9\" (UID: \"7cd2b593-c362-43c2-b0bb-7e5021791214\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-whnh9" Oct 30 13:48:44.995883 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:44.995788 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhdhm\" (UniqueName: \"kubernetes.io/projected/ccafe9ae-44d7-4c31-a6a5-ba8506bc0cea-kube-api-access-xhdhm\") pod \"migrator-66df5d496b-v2x2w\" (UID: \"ccafe9ae-44d7-4c31-a6a5-ba8506bc0cea\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-v2x2w" Oct 30 13:48:45.021617 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:45.021585 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srtk8\" (UniqueName: \"kubernetes.io/projected/7cd2b593-c362-43c2-b0bb-7e5021791214-kube-api-access-srtk8\") pod \"migrator-66df5d496b-whnh9\" (UID: \"7cd2b593-c362-43c2-b0bb-7e5021791214\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-whnh9" Oct 30 13:48:45.021795 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:45.021717 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhdhm\" (UniqueName: \"kubernetes.io/projected/ccafe9ae-44d7-4c31-a6a5-ba8506bc0cea-kube-api-access-xhdhm\") pod \"migrator-66df5d496b-v2x2w\" (UID: \"ccafe9ae-44d7-4c31-a6a5-ba8506bc0cea\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-v2x2w" Oct 30 13:48:45.055676 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:45.055607 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-kcgbg" podStartSLOduration=255.258760361 podStartE2EDuration="4m20.055593889s" totalImagesPullingTime="4.796833528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:44:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:45.054134835 +0000 UTC m=+35.073656234" watchObservedRunningTime="2026-10-30 13:48:45.055593889 +0000 UTC m=+35.075115285" Oct 30 13:48:45.063173 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:45.063147 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:45.093165 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:45.093135 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf73153f_1421_44d6_abae_b76c3d90235e.slice/crio-7ac0fb5309e0357a2007b2d08f06eef95701e311031c68d8099504d151dc465d WatchSource:0}: Error finding container 7ac0fb5309e0357a2007b2d08f06eef95701e311031c68d8099504d151dc465d: Status 404 returned error can't find the container with id 7ac0fb5309e0357a2007b2d08f06eef95701e311031c68d8099504d151dc465d Oct 30 13:48:45.094130 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:45.094106 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b5e74da_5460_44ed_a599_b01620b64c92.slice/crio-c02569bc0c161f51caedc865f18e9bfffc42568ce87441eb0ea60b1d4c09f4c6 WatchSource:0}: Error finding container c02569bc0c161f51caedc865f18e9bfffc42568ce87441eb0ea60b1d4c09f4c6: Status 404 returned error can't find the container with id c02569bc0c161f51caedc865f18e9bfffc42568ce87441eb0ea60b1d4c09f4c6 Oct 30 13:48:45.110744 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:45.110719 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-whnh9" Oct 30 13:48:45.132596 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:45.132512 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-v2x2w" Oct 30 13:48:45.294286 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:45.294255 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-87kd9"] Oct 30 13:48:45.308924 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:45.308903 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-87kd9" Oct 30 13:48:45.320204 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:45.320181 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-87kd9"] Oct 30 13:48:45.343831 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:45.343789 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 13:48:45.398288 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:45.398229 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:45.399253 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:45.399235 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6b536198-f804-4868-beb9-123229c3ae51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-87kd9\" (UID: \"6b536198-f804-4868-beb9-123229c3ae51\") " pod="openshift-insights/insights-runtime-extractor-87kd9" Oct 30 13:48:45.399319 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:45.399262 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r9cl\" (UniqueName: \"kubernetes.io/projected/6b536198-f804-4868-beb9-123229c3ae51-kube-api-access-5r9cl\") pod \"insights-runtime-extractor-87kd9\" (UID: \"6b536198-f804-4868-beb9-123229c3ae51\") " pod="openshift-insights/insights-runtime-extractor-87kd9" Oct 30 13:48:45.399319 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:45.399289 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6b536198-f804-4868-beb9-123229c3ae51-crio-socket\") pod \"insights-runtime-extractor-87kd9\" (UID: \"6b536198-f804-4868-beb9-123229c3ae51\") " pod="openshift-insights/insights-runtime-extractor-87kd9" Oct 30 13:48:45.399478 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:45.399441 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6b536198-f804-4868-beb9-123229c3ae51-data-volume\") pod \"insights-runtime-extractor-87kd9\" (UID: \"6b536198-f804-4868-beb9-123229c3ae51\") " pod="openshift-insights/insights-runtime-extractor-87kd9" Oct 30 13:48:45.399511 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:45.399500 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6b536198-f804-4868-beb9-123229c3ae51-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-87kd9\" (UID: \"6b536198-f804-4868-beb9-123229c3ae51\") " pod="openshift-insights/insights-runtime-extractor-87kd9" Oct 30 13:48:45.399881 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:45.399853 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 13:48:45.400115 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:45.400091 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-w6ztc\"" Oct 30 13:48:45.417134 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:45.417111 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-whnh9"] Oct 30 13:48:45.417699 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:45.417677 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cd2b593_c362_43c2_b0bb_7e5021791214.slice/crio-1420d7fc54824d5882f58897ef950c1efcdbd4980a308175b1e490f5688683e2 WatchSource:0}: Error finding container 1420d7fc54824d5882f58897ef950c1efcdbd4980a308175b1e490f5688683e2: Status 404 returned error can't find the container with id 1420d7fc54824d5882f58897ef950c1efcdbd4980a308175b1e490f5688683e2 Oct 30 13:48:45.466449 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:45.466429 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-v2x2w"] Oct 30 13:48:45.466920 ip-10-0-140-105 kubenswrapper[2580]: W1030 13:48:45.466895 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccafe9ae_44d7_4c31_a6a5_ba8506bc0cea.slice/crio-f1116d41e0e81d8d5834fa70a3fbaebbc48e81b5ec63a46345ed6b7875bdd5dd WatchSource:0}: Error finding container f1116d41e0e81d8d5834fa70a3fbaebbc48e81b5ec63a46345ed6b7875bdd5dd: Status 404 returned error can't find the container with id f1116d41e0e81d8d5834fa70a3fbaebbc48e81b5ec63a46345ed6b7875bdd5dd Oct 30 13:48:45.501028 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:45.500997 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6b536198-f804-4868-beb9-123229c3ae51-data-volume\") pod \"insights-runtime-extractor-87kd9\" (UID: \"6b536198-f804-4868-beb9-123229c3ae51\") " pod="openshift-insights/insights-runtime-extractor-87kd9" Oct 30 13:48:45.501119 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:45.501056 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6b536198-f804-4868-beb9-123229c3ae51-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-87kd9\" (UID: \"6b536198-f804-4868-beb9-123229c3ae51\") " pod="openshift-insights/insights-runtime-extractor-87kd9" Oct 30 13:48:45.501186 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:45.501138 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6b536198-f804-4868-beb9-123229c3ae51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-87kd9\" (UID: \"6b536198-f804-4868-beb9-123229c3ae51\") " pod="openshift-insights/insights-runtime-extractor-87kd9" Oct 30 13:48:45.501186 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:45.501165 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r9cl\" (UniqueName: \"kubernetes.io/projected/6b536198-f804-4868-beb9-123229c3ae51-kube-api-access-5r9cl\") pod \"insights-runtime-extractor-87kd9\" (UID: \"6b536198-f804-4868-beb9-123229c3ae51\") " pod="openshift-insights/insights-runtime-extractor-87kd9" Oct 30 13:48:45.501285 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:45.501206 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6b536198-f804-4868-beb9-123229c3ae51-crio-socket\") pod \"insights-runtime-extractor-87kd9\" (UID: \"6b536198-f804-4868-beb9-123229c3ae51\") " pod="openshift-insights/insights-runtime-extractor-87kd9" Oct 30 13:48:45.501338 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:45.501304 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6b536198-f804-4868-beb9-123229c3ae51-crio-socket\") pod \"insights-runtime-extractor-87kd9\" (UID: \"6b536198-f804-4868-beb9-123229c3ae51\") " pod="openshift-insights/insights-runtime-extractor-87kd9" Oct 30 13:48:45.501338 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:45.501327 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6b536198-f804-4868-beb9-123229c3ae51-data-volume\") pod \"insights-runtime-extractor-87kd9\" (UID: \"6b536198-f804-4868-beb9-123229c3ae51\") " pod="openshift-insights/insights-runtime-extractor-87kd9" Oct 30 13:48:45.501420 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:45.501368 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 13:48:45.501420 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:45.501416 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6b536198-f804-4868-beb9-123229c3ae51-kube-rbac-proxy-cm podName:6b536198-f804-4868-beb9-123229c3ae51 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:46.001398578 +0000 UTC m=+36.020919964 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6b536198-f804-4868-beb9-123229c3ae51-kube-rbac-proxy-cm") pod "insights-runtime-extractor-87kd9" (UID: "6b536198-f804-4868-beb9-123229c3ae51") : configmap "kube-rbac-proxy" not found Oct 30 13:48:45.501519 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:45.501432 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 13:48:45.501519 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:45.501482 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b536198-f804-4868-beb9-123229c3ae51-insights-runtime-extractor-tls podName:6b536198-f804-4868-beb9-123229c3ae51 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:46.00146407 +0000 UTC m=+36.020985462 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6b536198-f804-4868-beb9-123229c3ae51-insights-runtime-extractor-tls") pod "insights-runtime-extractor-87kd9" (UID: "6b536198-f804-4868-beb9-123229c3ae51") : secret "insights-runtime-extractor-tls" not found Oct 30 13:48:45.534516 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:45.534492 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r9cl\" (UniqueName: \"kubernetes.io/projected/6b536198-f804-4868-beb9-123229c3ae51-kube-api-access-5r9cl\") pod \"insights-runtime-extractor-87kd9\" (UID: \"6b536198-f804-4868-beb9-123229c3ae51\") " pod="openshift-insights/insights-runtime-extractor-87kd9" Oct 30 13:48:45.569122 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:45.569094 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:45.721270 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:45.721238 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:45.780864 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:45.780162 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bhw8w" event={"ID":"806dd609-9a2a-49c4-a656-e50437457fa9","Type":"ContainerStarted","Data":"d613c86038a18423e3ba26a743de3cf63a34de86106a8b952cedc3e6cd401543"} Oct 30 13:48:45.782537 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:45.782517 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ntlqq_dc967875-1546-40bb-8376-fde09416c204/console-operator/1.log" Oct 30 13:48:45.782879 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:45.782862 2580 scope.go:122] "RemoveContainer" containerID="08dafe630716a5a619c9abe02a35bcce9e3b7cb581311f535ccb7aff3f8803cb" Oct 30 13:48:45.783064 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:45.783045 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-ntlqq_openshift-console-operator(dc967875-1546-40bb-8376-fde09416c204)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-ntlqq" podUID="dc967875-1546-40bb-8376-fde09416c204" Oct 30 13:48:45.784774 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:45.784746 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-whnh9" event={"ID":"7cd2b593-c362-43c2-b0bb-7e5021791214","Type":"ContainerStarted","Data":"1420d7fc54824d5882f58897ef950c1efcdbd4980a308175b1e490f5688683e2"} Oct 30 13:48:45.786376 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:45.786348 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bdmgn" event={"ID":"6b5e74da-5460-44ed-a599-b01620b64c92","Type":"ContainerStarted","Data":"c02569bc0c161f51caedc865f18e9bfffc42568ce87441eb0ea60b1d4c09f4c6"} Oct 30 13:48:45.787807 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:45.787781 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-v2x2w" event={"ID":"ccafe9ae-44d7-4c31-a6a5-ba8506bc0cea","Type":"ContainerStarted","Data":"f1116d41e0e81d8d5834fa70a3fbaebbc48e81b5ec63a46345ed6b7875bdd5dd"} Oct 30 13:48:45.789106 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:45.789084 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dx94q" event={"ID":"bf73153f-1421-44d6-abae-b76c3d90235e","Type":"ContainerStarted","Data":"7ac0fb5309e0357a2007b2d08f06eef95701e311031c68d8099504d151dc465d"} Oct 30 13:48:45.824727 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:45.824686 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bhw8w" podStartSLOduration=253.306901168 podStartE2EDuration="4m18.824673235s" totalImagesPullingTime="5.517772067s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:44:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:45.823502327 +0000 UTC m=+35.843023726" watchObservedRunningTime="2026-10-30 13:48:45.824673235 +0000 UTC m=+35.844194633" Oct 30 13:48:46.008770 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:46.008071 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6b536198-f804-4868-beb9-123229c3ae51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-87kd9\" (UID: \"6b536198-f804-4868-beb9-123229c3ae51\") " pod="openshift-insights/insights-runtime-extractor-87kd9" Oct 30 13:48:46.008770 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:46.008234 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 13:48:46.008770 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:46.008253 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6b536198-f804-4868-beb9-123229c3ae51-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-87kd9\" (UID: \"6b536198-f804-4868-beb9-123229c3ae51\") " pod="openshift-insights/insights-runtime-extractor-87kd9" Oct 30 13:48:46.008770 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:46.008319 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b536198-f804-4868-beb9-123229c3ae51-insights-runtime-extractor-tls podName:6b536198-f804-4868-beb9-123229c3ae51 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:47.008297097 +0000 UTC m=+37.027818493 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6b536198-f804-4868-beb9-123229c3ae51-insights-runtime-extractor-tls") pod "insights-runtime-extractor-87kd9" (UID: "6b536198-f804-4868-beb9-123229c3ae51") : secret "insights-runtime-extractor-tls" not found Oct 30 13:48:46.008770 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:46.008345 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 13:48:46.008770 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:46.008396 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6b536198-f804-4868-beb9-123229c3ae51-kube-rbac-proxy-cm podName:6b536198-f804-4868-beb9-123229c3ae51 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:47.008382196 +0000 UTC m=+37.027903591 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6b536198-f804-4868-beb9-123229c3ae51-kube-rbac-proxy-cm") pod "insights-runtime-extractor-87kd9" (UID: "6b536198-f804-4868-beb9-123229c3ae51") : configmap "kube-rbac-proxy" not found Oct 30 13:48:46.438322 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:46.438287 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:46.584834 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:46.584754 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:46.614996 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:46.614961 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d73c6c3-044d-4b88-a105-72bd5e4491cd-service-ca-bundle\") pod \"router-default-659879fb99-pjvcq\" (UID: \"6d73c6c3-044d-4b88-a105-72bd5e4491cd\") " pod="openshift-ingress/router-default-659879fb99-pjvcq" Oct 30 13:48:46.615136 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:46.615015 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b0b2d906-7995-4769-8222-fa8ba2a30f3d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-9r9zr\" (UID: \"b0b2d906-7995-4769-8222-fa8ba2a30f3d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9r9zr" Oct 30 13:48:46.615136 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:46.615047 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f707b282-e507-475d-a82a-923c9bd1ff01-registry-tls\") pod \"image-registry-6bc4d9765d-dp7xb\" (UID: \"f707b282-e507-475d-a82a-923c9bd1ff01\") " pod="openshift-image-registry/image-registry-6bc4d9765d-dp7xb" Oct 30 13:48:46.615136 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:46.615075 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d73c6c3-044d-4b88-a105-72bd5e4491cd-metrics-certs\") pod \"router-default-659879fb99-pjvcq\" (UID: \"6d73c6c3-044d-4b88-a105-72bd5e4491cd\") " pod="openshift-ingress/router-default-659879fb99-pjvcq" Oct 30 13:48:46.615288 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:46.615158 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/083c70c2-0d5e-4876-9e88-10c6efbb681c-metrics-tls\") pod \"dns-default-7lx4t\" (UID: \"083c70c2-0d5e-4876-9e88-10c6efbb681c\") " pod="openshift-dns/dns-default-7lx4t" Oct 30 13:48:46.615288 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:46.615191 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 13:48:46.615288 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:46.615217 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8309f073-dcb0-4971-9eb1-8a9ee0eb68c6-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2bnbl\" (UID: \"8309f073-dcb0-4971-9eb1-8a9ee0eb68c6\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2bnbl" Oct 30 13:48:46.615288 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:46.615194 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 13:48:46.615288 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:46.615264 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0b2d906-7995-4769-8222-fa8ba2a30f3d-cluster-monitoring-operator-tls podName:b0b2d906-7995-4769-8222-fa8ba2a30f3d nodeName:}" failed. No retries permitted until 2026-10-30 13:48:54.615245428 +0000 UTC m=+44.634766818 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b0b2d906-7995-4769-8222-fa8ba2a30f3d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-9r9zr" (UID: "b0b2d906-7995-4769-8222-fa8ba2a30f3d") : secret "cluster-monitoring-operator-tls" not found Oct 30 13:48:46.615288 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:46.615199 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 13:48:46.615521 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:46.615292 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 13:48:46.615521 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:46.615302 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d73c6c3-044d-4b88-a105-72bd5e4491cd-metrics-certs podName:6d73c6c3-044d-4b88-a105-72bd5e4491cd nodeName:}" failed. No retries permitted until 2026-10-30 13:48:54.615283 +0000 UTC m=+44.634804392 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d73c6c3-044d-4b88-a105-72bd5e4491cd-metrics-certs") pod "router-default-659879fb99-pjvcq" (UID: "6d73c6c3-044d-4b88-a105-72bd5e4491cd") : secret "router-metrics-certs-default" not found Oct 30 13:48:46.615521 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:46.615328 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 13:48:46.615521 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:46.615293 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bc4d9765d-dp7xb: secret "image-registry-tls" not found Oct 30 13:48:46.615521 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:46.615340 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6d73c6c3-044d-4b88-a105-72bd5e4491cd-service-ca-bundle podName:6d73c6c3-044d-4b88-a105-72bd5e4491cd nodeName:}" failed. No retries permitted until 2026-10-30 13:48:54.615330857 +0000 UTC m=+44.634852248 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6d73c6c3-044d-4b88-a105-72bd5e4491cd-service-ca-bundle") pod "router-default-659879fb99-pjvcq" (UID: "6d73c6c3-044d-4b88-a105-72bd5e4491cd") : configmap references non-existent config key: service-ca.crt Oct 30 13:48:46.615521 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:46.615355 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/083c70c2-0d5e-4876-9e88-10c6efbb681c-metrics-tls podName:083c70c2-0d5e-4876-9e88-10c6efbb681c nodeName:}" failed. No retries permitted until 2026-10-30 13:48:54.615347809 +0000 UTC m=+44.634869188 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/083c70c2-0d5e-4876-9e88-10c6efbb681c-metrics-tls") pod "dns-default-7lx4t" (UID: "083c70c2-0d5e-4876-9e88-10c6efbb681c") : secret "dns-default-metrics-tls" not found Oct 30 13:48:46.615521 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:46.615386 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e16ea4af-ce12-442b-91df-092457a7e192-registry-tls\") pod \"image-registry-87f7c8d8c-mdvsm\" (UID: \"e16ea4af-ce12-442b-91df-092457a7e192\") " pod="openshift-image-registry/image-registry-87f7c8d8c-mdvsm" Oct 30 13:48:46.615521 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:46.615428 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/81dcad0b-d124-41b8-8b4e-d41049de217a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-qstzd\" (UID: \"81dcad0b-d124-41b8-8b4e-d41049de217a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qstzd" Oct 30 13:48:46.615521 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:46.615445 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8309f073-dcb0-4971-9eb1-8a9ee0eb68c6-networking-console-plugin-cert podName:8309f073-dcb0-4971-9eb1-8a9ee0eb68c6 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:54.615427896 +0000 UTC m=+44.634949279 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8309f073-dcb0-4971-9eb1-8a9ee0eb68c6-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-2bnbl" (UID: "8309f073-dcb0-4971-9eb1-8a9ee0eb68c6") : secret "networking-console-plugin-cert" not found Oct 30 13:48:46.615521 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:46.615469 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f707b282-e507-475d-a82a-923c9bd1ff01-registry-tls podName:f707b282-e507-475d-a82a-923c9bd1ff01 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:54.615459358 +0000 UTC m=+44.634980751 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f707b282-e507-475d-a82a-923c9bd1ff01-registry-tls") pod "image-registry-6bc4d9765d-dp7xb" (UID: "f707b282-e507-475d-a82a-923c9bd1ff01") : secret "image-registry-tls" not found Oct 30 13:48:46.615521 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:46.615486 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 13:48:46.615979 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:46.615536 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81dcad0b-d124-41b8-8b4e-d41049de217a-samples-operator-tls podName:81dcad0b-d124-41b8-8b4e-d41049de217a nodeName:}" failed. No retries permitted until 2026-10-30 13:48:54.61552533 +0000 UTC m=+44.635046726 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/81dcad0b-d124-41b8-8b4e-d41049de217a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-qstzd" (UID: "81dcad0b-d124-41b8-8b4e-d41049de217a") : secret "samples-operator-tls" not found Oct 30 13:48:46.615979 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:46.615576 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 13:48:46.615979 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:46.615588 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-87f7c8d8c-mdvsm: secret "image-registry-tls" not found Oct 30 13:48:46.615979 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:46.615628 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e16ea4af-ce12-442b-91df-092457a7e192-registry-tls podName:e16ea4af-ce12-442b-91df-092457a7e192 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:54.615617554 +0000 UTC m=+44.635138935 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e16ea4af-ce12-442b-91df-092457a7e192-registry-tls") pod "image-registry-87f7c8d8c-mdvsm" (UID: "e16ea4af-ce12-442b-91df-092457a7e192") : secret "image-registry-tls" not found Oct 30 13:48:46.801760 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:46.801683 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dx94q" event={"ID":"bf73153f-1421-44d6-abae-b76c3d90235e","Type":"ContainerStarted","Data":"1005d9f54f5d8e42f258e57e3121de7730f306e1f018eb0735ded235a9d26708"} Oct 30 13:48:46.802190 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:46.801994 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-dx94q" Oct 30 13:48:46.846561 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:46.846504 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xbkzq" podStartSLOduration=3.774768569 podStartE2EDuration="8.846487801s" totalImagesPullingTime="5.071719232s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:48:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:45.926136286 +0000 UTC m=+35.945657686" watchObservedRunningTime="2026-10-30 13:48:46.846487801 +0000 UTC m=+36.866009202" Oct 30 13:48:46.847106 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:46.847066 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-dx94q" podStartSLOduration=36.248993283 podStartE2EDuration="36.847059438s" totalImagesPullingTime="598.066155ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:48:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:46.845834091 +0000 UTC m=+36.865355492" watchObservedRunningTime="2026-10-30 13:48:46.847059438 +0000 UTC m=+36.866580838" Oct 30 13:48:47.020064 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:47.020026 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6b536198-f804-4868-beb9-123229c3ae51-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-87kd9\" (UID: \"6b536198-f804-4868-beb9-123229c3ae51\") " pod="openshift-insights/insights-runtime-extractor-87kd9" Oct 30 13:48:47.020225 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:47.020121 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6b536198-f804-4868-beb9-123229c3ae51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-87kd9\" (UID: \"6b536198-f804-4868-beb9-123229c3ae51\") " pod="openshift-insights/insights-runtime-extractor-87kd9" Oct 30 13:48:47.020225 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:47.020147 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 13:48:47.020225 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:47.020214 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6b536198-f804-4868-beb9-123229c3ae51-kube-rbac-proxy-cm podName:6b536198-f804-4868-beb9-123229c3ae51 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:49.020196338 +0000 UTC m=+39.039717740 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6b536198-f804-4868-beb9-123229c3ae51-kube-rbac-proxy-cm") pod "insights-runtime-extractor-87kd9" (UID: "6b536198-f804-4868-beb9-123229c3ae51") : configmap "kube-rbac-proxy" not found Oct 30 13:48:47.020370 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:47.020288 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 13:48:47.020370 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:47.020356 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b536198-f804-4868-beb9-123229c3ae51-insights-runtime-extractor-tls podName:6b536198-f804-4868-beb9-123229c3ae51 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:49.020338872 +0000 UTC m=+39.039860268 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6b536198-f804-4868-beb9-123229c3ae51-insights-runtime-extractor-tls") pod "insights-runtime-extractor-87kd9" (UID: "6b536198-f804-4868-beb9-123229c3ae51") : secret "insights-runtime-extractor-tls" not found Oct 30 13:48:47.158279 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:47.158249 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:47.807835 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:47.807797 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-whnh9" event={"ID":"7cd2b593-c362-43c2-b0bb-7e5021791214","Type":"ContainerStarted","Data":"445ba2280239ca09c992e9881338aa2d69a37fd5685264006ce078bb01ffb8c9"} Oct 30 13:48:47.809825 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:47.809798 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-v2x2w" event={"ID":"ccafe9ae-44d7-4c31-a6a5-ba8506bc0cea","Type":"ContainerStarted","Data":"9a003ae9f39b88400767b9591eb8746f136cf24b034e0937a66f015d694890d4"} Oct 30 13:48:47.852709 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:47.852654 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-whnh9" podStartSLOduration=2.164717963 podStartE2EDuration="3.85262367s" totalImagesPullingTime="1.687905707s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:48:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:47.851792361 +0000 UTC m=+37.871313760" watchObservedRunningTime="2026-10-30 13:48:47.85262367 +0000 UTC m=+37.872145072" Oct 30 13:48:47.895129 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:47.895077 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-v2x2w" podStartSLOduration=2.254134907 podStartE2EDuration="3.895063997s" totalImagesPullingTime="1.64092909s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:48:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:47.89478529 +0000 UTC m=+37.914306690" watchObservedRunningTime="2026-10-30 13:48:47.895063997 +0000 UTC m=+37.914585395" Oct 30 13:48:49.040411 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:49.040372 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6b536198-f804-4868-beb9-123229c3ae51-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-87kd9\" (UID: \"6b536198-f804-4868-beb9-123229c3ae51\") " pod="openshift-insights/insights-runtime-extractor-87kd9" Oct 30 13:48:49.040891 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:49.040474 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6b536198-f804-4868-beb9-123229c3ae51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-87kd9\" (UID: \"6b536198-f804-4868-beb9-123229c3ae51\") " pod="openshift-insights/insights-runtime-extractor-87kd9" Oct 30 13:48:49.040891 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:49.040492 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 13:48:49.040891 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:49.040559 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6b536198-f804-4868-beb9-123229c3ae51-kube-rbac-proxy-cm podName:6b536198-f804-4868-beb9-123229c3ae51 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:53.040544383 +0000 UTC m=+43.060065779 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6b536198-f804-4868-beb9-123229c3ae51-kube-rbac-proxy-cm") pod "insights-runtime-extractor-87kd9" (UID: "6b536198-f804-4868-beb9-123229c3ae51") : configmap "kube-rbac-proxy" not found Oct 30 13:48:49.040891 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:49.040603 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 13:48:49.040891 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:49.040679 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b536198-f804-4868-beb9-123229c3ae51-insights-runtime-extractor-tls podName:6b536198-f804-4868-beb9-123229c3ae51 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:53.040660879 +0000 UTC m=+43.060182273 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6b536198-f804-4868-beb9-123229c3ae51-insights-runtime-extractor-tls") pod "insights-runtime-extractor-87kd9" (UID: "6b536198-f804-4868-beb9-123229c3ae51") : secret "insights-runtime-extractor-tls" not found Oct 30 13:48:49.226697 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:49.226665 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-ntlqq" Oct 30 13:48:49.226931 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:49.226724 2580 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-ntlqq" Oct 30 13:48:49.227124 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:49.227106 2580 scope.go:122] "RemoveContainer" containerID="08dafe630716a5a619c9abe02a35bcce9e3b7cb581311f535ccb7aff3f8803cb" Oct 30 13:48:49.227357 ip-10-0-140-105 kubenswrapper[2580]: E1030 13:48:49.227327 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-ntlqq_openshift-console-operator(dc967875-1546-40bb-8376-fde09416c204)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-ntlqq" podUID="dc967875-1546-40bb-8376-fde09416c204" Oct 30 13:48:49.320946 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:49.320912 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:49.476652 ip-10-0-140-105 kubenswrapper[2580]: I1030 13:48:49.476619 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:50.169601 ip-10-0-140-105 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 13:48:50.192998 ip-10-0-140-105 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 13:48:50.193161 ip-10-0-140-105 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 13:48:50.193327 ip-10-0-140-105 systemd[1]: kubelet.service: Consumed 2.840s CPU time. Oct 30 13:48:50.213090 ip-10-0-140-105 systemd[1]: Starting Kubernetes Kubelet... Oct 30 13:48:50.314539 ip-10-0-140-105 kubenswrapper[4982]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 13:48:50.314539 ip-10-0-140-105 kubenswrapper[4982]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 13:48:50.314539 ip-10-0-140-105 kubenswrapper[4982]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 13:48:50.314539 ip-10-0-140-105 kubenswrapper[4982]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 13:48:50.316730 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.316659 4982 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 13:48:50.316730 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.316727 4982 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 13:48:50.316730 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.316732 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 13:48:50.316944 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.316738 4982 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 13:48:50.316944 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.316742 4982 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 13:48:50.316944 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.316749 4982 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 13:48:50.316944 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.316755 4982 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 13:48:50.316944 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.316760 4982 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 13:48:50.316944 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.316771 4982 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 13:48:50.316944 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.316776 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 13:48:50.316944 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.316780 4982 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 13:48:50.316944 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.316784 4982 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 13:48:50.316944 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.316788 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 13:48:50.316944 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.316793 4982 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 13:48:50.316944 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.316796 4982 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 13:48:50.316944 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.316800 4982 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 13:48:50.316944 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.316804 4982 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 13:48:50.316944 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.316808 4982 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 13:48:50.316944 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.316812 4982 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 13:48:50.316944 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.316816 4982 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 13:48:50.316944 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.316820 4982 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 13:48:50.316944 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.316824 4982 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 13:48:50.317789 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.316827 4982 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 13:48:50.317789 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.316831 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 13:48:50.317789 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.316835 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 13:48:50.317789 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.316839 4982 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 13:48:50.317789 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.316843 4982 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 13:48:50.317789 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.316847 4982 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 13:48:50.317789 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.316852 4982 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 13:48:50.317789 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.316856 4982 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 13:48:50.317789 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.316861 4982 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 13:48:50.317789 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.316865 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 13:48:50.317789 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.316870 4982 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 13:48:50.317789 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.316874 4982 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 13:48:50.317789 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.316878 4982 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 13:48:50.317789 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.316882 4982 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 13:48:50.317789 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.316886 4982 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 13:48:50.317789 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.316890 4982 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 13:48:50.317789 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.316894 4982 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 13:48:50.317789 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.316908 4982 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 13:48:50.317789 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.316915 4982 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 13:48:50.318618 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.316921 4982 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 13:48:50.318618 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.316926 4982 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 13:48:50.318618 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.316930 4982 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 13:48:50.318618 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.316936 4982 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 13:48:50.318618 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.316940 4982 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 13:48:50.318618 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.316945 4982 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 13:48:50.318618 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.316949 4982 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 13:48:50.318618 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.316953 4982 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 13:48:50.318618 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.316957 4982 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 13:48:50.318618 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.316963 4982 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 13:48:50.318618 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.316967 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 13:48:50.318618 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.316971 4982 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 13:48:50.318618 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.316975 4982 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 13:48:50.318618 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.316979 4982 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 13:48:50.318618 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.316984 4982 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 13:48:50.318618 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.316988 4982 feature_gate.go:386] unrecognized feature gate: Example Oct 30 13:48:50.318618 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.316992 4982 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 13:48:50.318618 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.316996 4982 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 13:48:50.318618 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.317001 4982 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 13:48:50.318618 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.317005 4982 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 13:48:50.319504 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.317009 4982 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 13:48:50.319504 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.317013 4982 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 13:48:50.319504 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.317018 4982 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 13:48:50.319504 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.317022 4982 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 13:48:50.319504 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.317026 4982 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 13:48:50.319504 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.317030 4982 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 13:48:50.319504 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.317036 4982 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 13:48:50.319504 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.317040 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 13:48:50.319504 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.317044 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 13:48:50.319504 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.317049 4982 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 13:48:50.319504 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.317053 4982 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 13:48:50.319504 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.317066 4982 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 13:48:50.319504 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.317071 4982 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 13:48:50.319504 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.317075 4982 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 13:48:50.319504 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.317080 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 13:48:50.319504 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.317084 4982 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 13:48:50.319504 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.317088 4982 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 13:48:50.319504 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.317093 4982 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 13:48:50.319504 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.317097 4982 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 13:48:50.319504 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.317101 4982 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 13:48:50.320419 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.317105 4982 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 13:48:50.320419 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.317109 4982 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 13:48:50.320419 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.317114 4982 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 13:48:50.320419 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.317119 4982 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 13:48:50.320419 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.317125 4982 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 13:48:50.320419 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.317129 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 13:48:50.320419 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.317133 4982 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 13:48:50.320419 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.317137 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 13:48:50.320419 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.317142 4982 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 13:48:50.320419 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.317146 4982 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 13:48:50.320419 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.317151 4982 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 13:48:50.320419 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.317155 4982 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 13:48:50.320419 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.317159 4982 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 13:48:50.320419 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.317163 4982 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 13:48:50.320419 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.317167 4982 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 13:48:50.320419 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.317171 4982 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 13:48:50.320419 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.317176 4982 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 13:48:50.320419 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.317180 4982 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 13:48:50.320419 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.317184 4982 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 13:48:50.321285 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.317188 4982 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 13:48:50.321285 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.317192 4982 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 13:48:50.321285 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.317197 4982 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 13:48:50.321285 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.317201 4982 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 13:48:50.321285 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.317205 4982 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 13:48:50.321285 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.317219 4982 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 13:48:50.321285 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318416 4982 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 13:48:50.321285 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318426 4982 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 13:48:50.321285 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318430 4982 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 13:48:50.321285 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318435 4982 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 13:48:50.321285 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318439 4982 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 13:48:50.321285 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318444 4982 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 13:48:50.321285 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318449 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 13:48:50.321285 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318453 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 13:48:50.321285 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318458 4982 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 13:48:50.321285 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318462 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 13:48:50.321285 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318466 4982 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 13:48:50.321285 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318471 4982 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 13:48:50.321285 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318475 4982 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 13:48:50.322169 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318479 4982 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 13:48:50.322169 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318483 4982 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 13:48:50.322169 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318487 4982 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 13:48:50.322169 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318492 4982 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 13:48:50.322169 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318497 4982 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 13:48:50.322169 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318501 4982 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 13:48:50.322169 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318505 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 13:48:50.322169 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318509 4982 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 13:48:50.322169 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318514 4982 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 13:48:50.322169 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318518 4982 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 13:48:50.322169 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318523 4982 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 13:48:50.322169 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318527 4982 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 13:48:50.322169 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318531 4982 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 13:48:50.322169 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318535 4982 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 13:48:50.322169 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318540 4982 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 13:48:50.322169 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318544 4982 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 13:48:50.322169 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318548 4982 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 13:48:50.322169 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318552 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 13:48:50.322169 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318556 4982 feature_gate.go:386] unrecognized feature gate: Example Oct 30 13:48:50.322169 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318562 4982 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 13:48:50.322967 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318566 4982 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 13:48:50.322967 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318570 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 13:48:50.322967 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318575 4982 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 13:48:50.322967 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318579 4982 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 13:48:50.322967 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318584 4982 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 13:48:50.322967 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318588 4982 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 13:48:50.322967 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318592 4982 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 13:48:50.322967 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318600 4982 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 13:48:50.322967 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318607 4982 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 13:48:50.322967 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318612 4982 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 13:48:50.322967 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318616 4982 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 13:48:50.322967 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318620 4982 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 13:48:50.322967 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318624 4982 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 13:48:50.322967 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318628 4982 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 13:48:50.322967 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318633 4982 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 13:48:50.322967 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318655 4982 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 13:48:50.322967 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318660 4982 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 13:48:50.322967 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318664 4982 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 13:48:50.322967 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318669 4982 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 13:48:50.322967 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318673 4982 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 13:48:50.323495 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318677 4982 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 13:48:50.323495 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318684 4982 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 13:48:50.323495 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318688 4982 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 13:48:50.323495 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318692 4982 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 13:48:50.323495 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318696 4982 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 13:48:50.323495 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318700 4982 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 13:48:50.323495 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318705 4982 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 13:48:50.323495 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318708 4982 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 13:48:50.323495 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318713 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 13:48:50.323495 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318717 4982 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 13:48:50.323495 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318722 4982 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 13:48:50.323495 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318726 4982 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 13:48:50.323495 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318731 4982 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 13:48:50.323495 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318735 4982 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 13:48:50.323495 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318739 4982 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 13:48:50.323495 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318744 4982 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 13:48:50.323495 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318748 4982 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 13:48:50.323495 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318752 4982 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 13:48:50.323495 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318759 4982 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 13:48:50.323976 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318765 4982 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 13:48:50.323976 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318770 4982 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 13:48:50.323976 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318775 4982 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 13:48:50.323976 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318779 4982 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 13:48:50.323976 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318783 4982 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 13:48:50.323976 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318788 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 13:48:50.323976 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318792 4982 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 13:48:50.323976 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318796 4982 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 13:48:50.323976 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318800 4982 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 13:48:50.323976 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318804 4982 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 13:48:50.323976 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318809 4982 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 13:48:50.323976 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318813 4982 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 13:48:50.323976 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318817 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 13:48:50.323976 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318822 4982 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 13:48:50.323976 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318827 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 13:48:50.323976 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318832 4982 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 13:48:50.323976 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318836 4982 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 13:48:50.323976 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318840 4982 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 13:48:50.323976 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318844 4982 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 13:48:50.323976 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318848 4982 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 13:48:50.324536 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318852 4982 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 13:48:50.324536 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318856 4982 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 13:48:50.324536 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318861 4982 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 13:48:50.324536 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318865 4982 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 13:48:50.324536 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318870 4982 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 13:48:50.324536 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318874 4982 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 13:48:50.324536 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318879 4982 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 13:48:50.324536 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318883 4982 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 13:48:50.324536 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318887 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 13:48:50.324536 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318892 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 13:48:50.324536 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318896 4982 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 13:48:50.324536 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318901 4982 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 13:48:50.324536 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318905 4982 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 13:48:50.324536 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.318911 4982 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 13:48:50.324536 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319686 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 13:48:50.324536 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319695 4982 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 13:48:50.324536 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319700 4982 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 13:48:50.324536 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319704 4982 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 13:48:50.324536 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319708 4982 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 13:48:50.325047 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319712 4982 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 13:48:50.325047 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319717 4982 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 13:48:50.325047 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319721 4982 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 13:48:50.325047 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319725 4982 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 13:48:50.325047 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319730 4982 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 13:48:50.325047 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319734 4982 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 13:48:50.325047 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319738 4982 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 13:48:50.325047 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319743 4982 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 13:48:50.325047 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319747 4982 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 13:48:50.325047 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319751 4982 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 13:48:50.325047 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319755 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 13:48:50.325047 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319760 4982 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 13:48:50.325047 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319764 4982 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 13:48:50.325047 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319768 4982 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 13:48:50.325047 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319774 4982 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 13:48:50.325047 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319780 4982 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 13:48:50.325047 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319785 4982 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 13:48:50.325047 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319789 4982 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 13:48:50.325047 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319793 4982 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 13:48:50.325500 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319798 4982 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 13:48:50.325500 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319802 4982 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 13:48:50.325500 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319806 4982 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 13:48:50.325500 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319810 4982 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 13:48:50.325500 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319814 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 13:48:50.325500 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319821 4982 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 13:48:50.325500 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319826 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 13:48:50.325500 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319832 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 13:48:50.325500 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319836 4982 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 13:48:50.325500 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319840 4982 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 13:48:50.325500 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319845 4982 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 13:48:50.325500 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319849 4982 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 13:48:50.325500 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319853 4982 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 13:48:50.325500 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319857 4982 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 13:48:50.325500 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319862 4982 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 13:48:50.325500 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319867 4982 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 13:48:50.325500 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319872 4982 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 13:48:50.325500 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319876 4982 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 13:48:50.325500 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319880 4982 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 13:48:50.325989 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319884 4982 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 13:48:50.325989 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319889 4982 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 13:48:50.325989 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319893 4982 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 13:48:50.325989 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319897 4982 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 13:48:50.325989 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319901 4982 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 13:48:50.325989 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319905 4982 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 13:48:50.325989 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319910 4982 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 13:48:50.325989 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319914 4982 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 13:48:50.325989 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319918 4982 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 13:48:50.325989 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319922 4982 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 13:48:50.325989 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319927 4982 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 13:48:50.325989 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319931 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 13:48:50.325989 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319935 4982 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 13:48:50.325989 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319939 4982 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 13:48:50.325989 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319944 4982 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 13:48:50.325989 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319948 4982 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 13:48:50.325989 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319952 4982 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 13:48:50.325989 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319956 4982 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 13:48:50.325989 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319961 4982 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 13:48:50.325989 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319965 4982 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 13:48:50.326512 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319969 4982 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 13:48:50.326512 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319973 4982 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 13:48:50.326512 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319977 4982 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 13:48:50.326512 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319981 4982 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 13:48:50.326512 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319985 4982 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 13:48:50.326512 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319989 4982 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 13:48:50.326512 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.319993 4982 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 13:48:50.326512 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.320003 4982 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 13:48:50.326512 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.320007 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 13:48:50.326512 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.320011 4982 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 13:48:50.326512 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.320016 4982 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 13:48:50.326512 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.320020 4982 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 13:48:50.326512 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.320024 4982 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 13:48:50.326512 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.320028 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 13:48:50.326512 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.320034 4982 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 13:48:50.326512 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.320038 4982 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 13:48:50.326512 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.320042 4982 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 13:48:50.326512 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.320047 4982 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 13:48:50.326512 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.320051 4982 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 13:48:50.326512 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.320056 4982 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 13:48:50.327015 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.320060 4982 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 13:48:50.327015 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.320064 4982 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 13:48:50.327015 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.320068 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 13:48:50.327015 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.320073 4982 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 13:48:50.327015 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.320077 4982 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 13:48:50.327015 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.320082 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 13:48:50.327015 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.320086 4982 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 13:48:50.327015 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.320091 4982 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 13:48:50.327015 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.320096 4982 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 13:48:50.327015 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.320102 4982 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 13:48:50.327015 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.320106 4982 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 13:48:50.327015 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.320110 4982 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 13:48:50.327015 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.320114 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 13:48:50.327015 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.320119 4982 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 13:48:50.327015 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.320123 4982 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 13:48:50.327015 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.320127 4982 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 13:48:50.327015 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.320132 4982 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 13:48:50.327015 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.320136 4982 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 13:48:50.327015 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.320140 4982 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 13:48:50.327784 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.320146 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 13:48:50.327784 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.320150 4982 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 13:48:50.327784 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.320154 4982 feature_gate.go:386] unrecognized feature gate: Example Oct 30 13:48:50.327784 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.320158 4982 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 13:48:50.327784 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320303 4982 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 13:48:50.327784 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320315 4982 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 13:48:50.327784 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320323 4982 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 13:48:50.327784 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320329 4982 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 13:48:50.327784 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320336 4982 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 13:48:50.327784 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320342 4982 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 13:48:50.327784 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320348 4982 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 13:48:50.327784 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320354 4982 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 13:48:50.327784 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320359 4982 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 13:48:50.327784 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320364 4982 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 13:48:50.327784 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320369 4982 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 13:48:50.327784 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320374 4982 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 13:48:50.327784 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320379 4982 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 13:48:50.327784 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320384 4982 flags.go:64] FLAG: --cgroup-root="" Oct 30 13:48:50.327784 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320389 4982 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 13:48:50.327784 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320393 4982 flags.go:64] FLAG: --client-ca-file="" Oct 30 13:48:50.327784 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320398 4982 flags.go:64] FLAG: --cloud-provider="external" Oct 30 13:48:50.327784 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320404 4982 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 13:48:50.327784 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320412 4982 flags.go:64] FLAG: --cluster-domain="" Oct 30 13:48:50.327784 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320416 4982 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 13:48:50.328711 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320421 4982 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 13:48:50.328711 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320426 4982 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 13:48:50.328711 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320431 4982 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 13:48:50.328711 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320437 4982 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 13:48:50.328711 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320441 4982 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 13:48:50.328711 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320446 4982 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 13:48:50.328711 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320451 4982 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 13:48:50.328711 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320456 4982 flags.go:64] FLAG: --contention-profiling="false" Oct 30 13:48:50.328711 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320463 4982 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 13:48:50.328711 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320468 4982 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 13:48:50.328711 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320472 4982 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 13:48:50.328711 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320477 4982 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 13:48:50.328711 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320483 4982 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 13:48:50.328711 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320488 4982 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 13:48:50.328711 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320493 4982 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 13:48:50.328711 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320497 4982 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 13:48:50.328711 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320502 4982 flags.go:64] FLAG: --enable-server="true" Oct 30 13:48:50.328711 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320507 4982 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 13:48:50.328711 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320513 4982 flags.go:64] FLAG: --event-burst="100" Oct 30 13:48:50.328711 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320518 4982 flags.go:64] FLAG: --event-qps="50" Oct 30 13:48:50.328711 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320523 4982 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 13:48:50.328711 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320527 4982 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 13:48:50.328711 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320532 4982 flags.go:64] FLAG: --eviction-hard="" Oct 30 13:48:50.328711 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320539 4982 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 13:48:50.329764 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320543 4982 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 13:48:50.329764 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320548 4982 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 13:48:50.329764 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320553 4982 flags.go:64] FLAG: --eviction-soft="" Oct 30 13:48:50.329764 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320558 4982 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 13:48:50.329764 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320562 4982 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 13:48:50.329764 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320567 4982 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 13:48:50.329764 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320574 4982 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 13:48:50.329764 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320579 4982 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 13:48:50.329764 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320583 4982 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 13:48:50.329764 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320588 4982 flags.go:64] FLAG: --feature-gates="" Oct 30 13:48:50.329764 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320594 4982 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 13:48:50.329764 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320599 4982 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 13:48:50.329764 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320604 4982 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 13:48:50.329764 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320609 4982 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 13:48:50.329764 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320614 4982 flags.go:64] FLAG: --healthz-port="10248" Oct 30 13:48:50.329764 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320619 4982 flags.go:64] FLAG: --help="false" Oct 30 13:48:50.329764 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320626 4982 flags.go:64] FLAG: --hostname-override="ip-10-0-140-105.ec2.internal" Oct 30 13:48:50.329764 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320631 4982 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 13:48:50.329764 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320652 4982 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 13:48:50.329764 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320657 4982 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 13:48:50.329764 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320663 4982 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 13:48:50.329764 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320668 4982 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 13:48:50.329764 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320673 4982 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 13:48:50.329764 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320677 4982 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 13:48:50.330779 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320682 4982 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 13:48:50.330779 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320686 4982 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 13:48:50.330779 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320691 4982 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 13:48:50.330779 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320695 4982 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 13:48:50.330779 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320700 4982 flags.go:64] FLAG: --kube-reserved="" Oct 30 13:48:50.330779 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320705 4982 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 13:48:50.330779 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320709 4982 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 13:48:50.330779 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320714 4982 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 13:48:50.330779 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320719 4982 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 13:48:50.330779 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320724 4982 flags.go:64] FLAG: --lock-file="" Oct 30 13:48:50.330779 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320728 4982 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 13:48:50.330779 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320733 4982 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 13:48:50.330779 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320737 4982 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 13:48:50.330779 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320746 4982 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 13:48:50.330779 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320752 4982 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 13:48:50.330779 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320758 4982 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 13:48:50.330779 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320763 4982 flags.go:64] FLAG: --logging-format="text" Oct 30 13:48:50.330779 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320768 4982 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 13:48:50.330779 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320773 4982 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 13:48:50.330779 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320778 4982 flags.go:64] FLAG: --manifest-url="" Oct 30 13:48:50.330779 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320782 4982 flags.go:64] FLAG: --manifest-url-header="" Oct 30 13:48:50.330779 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320789 4982 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 13:48:50.330779 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320794 4982 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 13:48:50.330779 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320800 4982 flags.go:64] FLAG: --max-pods="110" Oct 30 13:48:50.330779 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320806 4982 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 13:48:50.331866 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320811 4982 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 13:48:50.331866 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320815 4982 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 13:48:50.331866 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320820 4982 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 13:48:50.331866 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320825 4982 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 13:48:50.331866 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320829 4982 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 13:48:50.331866 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320834 4982 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 13:48:50.331866 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320844 4982 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 13:48:50.331866 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320850 4982 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 13:48:50.331866 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320854 4982 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 13:48:50.331866 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320859 4982 flags.go:64] FLAG: --pod-cidr="" Oct 30 13:48:50.331866 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320864 4982 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 13:48:50.331866 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320868 4982 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 13:48:50.331866 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320873 4982 flags.go:64] FLAG: --pods-per-core="0" Oct 30 13:48:50.331866 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320877 4982 flags.go:64] FLAG: --port="10250" Oct 30 13:48:50.331866 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320882 4982 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 13:48:50.331866 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320887 4982 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c265bd34ba759816" Oct 30 13:48:50.331866 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320892 4982 flags.go:64] FLAG: --qos-reserved="" Oct 30 13:48:50.331866 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320897 4982 flags.go:64] FLAG: --read-only-port="10255" Oct 30 13:48:50.331866 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320902 4982 flags.go:64] FLAG: --register-node="true" Oct 30 13:48:50.331866 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320906 4982 flags.go:64] FLAG: --register-with-taints="" Oct 30 13:48:50.331866 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320916 4982 flags.go:64] FLAG: --registry-burst="10" Oct 30 13:48:50.331866 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320923 4982 flags.go:64] FLAG: --registry-qps="5" Oct 30 13:48:50.331866 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320928 4982 flags.go:64] FLAG: --reserved-cpus="" Oct 30 13:48:50.331866 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320933 4982 flags.go:64] FLAG: --reserved-memory="" Oct 30 13:48:50.331866 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320939 4982 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 13:48:50.331866 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320944 4982 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 13:48:50.332979 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320949 4982 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 13:48:50.332979 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320954 4982 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 13:48:50.332979 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320959 4982 flags.go:64] FLAG: --runonce="false" Oct 30 13:48:50.332979 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320964 4982 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 13:48:50.332979 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320968 4982 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 13:48:50.332979 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320975 4982 flags.go:64] FLAG: --seccomp-default="false" Oct 30 13:48:50.332979 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320979 4982 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 13:48:50.332979 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320984 4982 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 13:48:50.332979 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320989 4982 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 13:48:50.332979 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320994 4982 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 13:48:50.332979 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.320998 4982 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 13:48:50.332979 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.321003 4982 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 13:48:50.332979 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.321008 4982 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 13:48:50.332979 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.321012 4982 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 13:48:50.332979 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.321017 4982 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 13:48:50.332979 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.321021 4982 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 13:48:50.332979 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.321026 4982 flags.go:64] FLAG: --system-cgroups="" Oct 30 13:48:50.332979 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.321030 4982 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 13:48:50.332979 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.321039 4982 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 13:48:50.332979 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.321044 4982 flags.go:64] FLAG: --tls-cert-file="" Oct 30 13:48:50.332979 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.321049 4982 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 13:48:50.332979 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.321055 4982 flags.go:64] FLAG: --tls-min-version="" Oct 30 13:48:50.332979 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.321060 4982 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 13:48:50.332979 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.321064 4982 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 13:48:50.332979 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.321069 4982 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 13:48:50.333840 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.321073 4982 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 13:48:50.333840 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.321078 4982 flags.go:64] FLAG: --v="2" Oct 30 13:48:50.333840 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.321086 4982 flags.go:64] FLAG: --version="false" Oct 30 13:48:50.333840 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.321093 4982 flags.go:64] FLAG: --vmodule="" Oct 30 13:48:50.333840 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.321099 4982 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 13:48:50.333840 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.321105 4982 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 13:48:50.333840 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321299 4982 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 13:48:50.333840 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321306 4982 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 13:48:50.333840 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321312 4982 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 13:48:50.333840 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321317 4982 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 13:48:50.333840 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321322 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 13:48:50.333840 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321328 4982 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 13:48:50.333840 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321337 4982 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 13:48:50.333840 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321342 4982 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 13:48:50.333840 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321347 4982 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 13:48:50.333840 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321352 4982 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 13:48:50.333840 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321357 4982 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 13:48:50.333840 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321361 4982 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 13:48:50.333840 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321366 4982 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 13:48:50.333840 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321370 4982 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 13:48:50.334840 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321375 4982 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 13:48:50.334840 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321379 4982 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 13:48:50.334840 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321383 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 13:48:50.334840 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321387 4982 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 13:48:50.334840 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321392 4982 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 13:48:50.334840 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321396 4982 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 13:48:50.334840 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321400 4982 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 13:48:50.334840 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321405 4982 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 13:48:50.334840 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321409 4982 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 13:48:50.334840 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321414 4982 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 13:48:50.334840 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321418 4982 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 13:48:50.334840 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321422 4982 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 13:48:50.334840 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321427 4982 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 13:48:50.334840 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321431 4982 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 13:48:50.334840 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321437 4982 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 13:48:50.334840 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321442 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 13:48:50.334840 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321446 4982 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 13:48:50.334840 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321450 4982 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 13:48:50.334840 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321455 4982 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 13:48:50.334840 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321459 4982 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 13:48:50.335523 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321463 4982 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 13:48:50.335523 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321468 4982 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 13:48:50.335523 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321473 4982 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 13:48:50.335523 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321477 4982 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 13:48:50.335523 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321483 4982 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 13:48:50.335523 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321487 4982 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 13:48:50.335523 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321491 4982 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 13:48:50.335523 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321495 4982 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 13:48:50.335523 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321499 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 13:48:50.335523 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321506 4982 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 13:48:50.335523 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321513 4982 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 13:48:50.335523 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321518 4982 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 13:48:50.335523 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321522 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 13:48:50.335523 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321528 4982 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 13:48:50.335523 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321532 4982 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 13:48:50.335523 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321536 4982 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 13:48:50.335523 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321540 4982 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 13:48:50.335523 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321545 4982 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 13:48:50.335523 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321549 4982 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 13:48:50.335523 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321554 4982 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 13:48:50.336109 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321558 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 13:48:50.336109 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321562 4982 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 13:48:50.336109 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321566 4982 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 13:48:50.336109 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321570 4982 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 13:48:50.336109 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321574 4982 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 13:48:50.336109 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321579 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 13:48:50.336109 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321586 4982 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 13:48:50.336109 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321590 4982 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 13:48:50.336109 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321594 4982 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 13:48:50.336109 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321598 4982 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 13:48:50.336109 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321602 4982 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 13:48:50.336109 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321608 4982 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 13:48:50.336109 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321612 4982 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 13:48:50.336109 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321616 4982 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 13:48:50.336109 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321620 4982 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 13:48:50.336109 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321624 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 13:48:50.336109 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321631 4982 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 13:48:50.336109 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321652 4982 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 13:48:50.336109 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321657 4982 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 13:48:50.336109 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321661 4982 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 13:48:50.336665 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321665 4982 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 13:48:50.336665 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321669 4982 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 13:48:50.336665 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321673 4982 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 13:48:50.336665 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321678 4982 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 13:48:50.336665 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321682 4982 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 13:48:50.336665 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321686 4982 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 13:48:50.336665 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321690 4982 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 13:48:50.336665 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321694 4982 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 13:48:50.336665 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321698 4982 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 13:48:50.336665 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321703 4982 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 13:48:50.336665 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321707 4982 feature_gate.go:386] unrecognized feature gate: Example Oct 30 13:48:50.336665 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321711 4982 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 13:48:50.336665 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321716 4982 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 13:48:50.336665 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321720 4982 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 13:48:50.336665 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321724 4982 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 13:48:50.336665 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321728 4982 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 13:48:50.336665 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321733 4982 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 13:48:50.336665 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321738 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 13:48:50.336665 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321744 4982 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 13:48:50.336665 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321749 4982 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 13:48:50.337154 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321753 4982 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 13:48:50.337154 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321757 4982 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 13:48:50.337154 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321762 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 13:48:50.337154 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321766 4982 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 13:48:50.337154 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321771 4982 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 13:48:50.337154 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321775 4982 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 13:48:50.337154 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321780 4982 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 13:48:50.337154 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321784 4982 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 13:48:50.337154 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321790 4982 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 13:48:50.337154 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321794 4982 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 13:48:50.337154 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321799 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 13:48:50.337154 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321803 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 13:48:50.337154 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.321846 4982 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 13:48:50.337154 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321900 4982 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 13:48:50.337154 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321906 4982 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 13:48:50.337154 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321911 4982 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 13:48:50.337535 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321915 4982 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 13:48:50.337535 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321920 4982 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 13:48:50.337535 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321924 4982 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 13:48:50.337535 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321928 4982 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 13:48:50.337535 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321932 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 13:48:50.337535 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321937 4982 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 13:48:50.337535 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321941 4982 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 13:48:50.337535 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321945 4982 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 13:48:50.337535 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321949 4982 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 13:48:50.337535 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321953 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 13:48:50.337535 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321957 4982 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 13:48:50.337535 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321962 4982 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 13:48:50.337535 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321966 4982 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 13:48:50.337535 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321971 4982 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 13:48:50.337535 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321977 4982 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 13:48:50.337535 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321981 4982 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 13:48:50.337535 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321985 4982 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 13:48:50.337535 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321990 4982 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 13:48:50.337535 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321994 4982 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 13:48:50.337535 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.321998 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 13:48:50.338133 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322002 4982 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 13:48:50.338133 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322007 4982 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 13:48:50.338133 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322012 4982 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 13:48:50.338133 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322016 4982 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 13:48:50.338133 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322020 4982 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 13:48:50.338133 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322026 4982 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 13:48:50.338133 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322030 4982 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 13:48:50.338133 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322034 4982 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 13:48:50.338133 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322039 4982 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 13:48:50.338133 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322043 4982 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 13:48:50.338133 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322047 4982 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 13:48:50.338133 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322052 4982 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 13:48:50.338133 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322056 4982 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 13:48:50.338133 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322060 4982 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 13:48:50.338133 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322065 4982 feature_gate.go:386] unrecognized feature gate: Example Oct 30 13:48:50.338133 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322069 4982 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 13:48:50.338133 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322073 4982 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 13:48:50.338133 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322078 4982 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 13:48:50.338133 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322083 4982 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 13:48:50.338133 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322087 4982 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 13:48:50.338630 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322091 4982 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 13:48:50.338630 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322096 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 13:48:50.338630 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322100 4982 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 13:48:50.338630 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322104 4982 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 13:48:50.338630 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322109 4982 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 13:48:50.338630 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322113 4982 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 13:48:50.338630 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322119 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 13:48:50.338630 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322123 4982 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 13:48:50.338630 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322127 4982 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 13:48:50.338630 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322132 4982 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 13:48:50.338630 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322136 4982 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 13:48:50.338630 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322140 4982 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 13:48:50.338630 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322144 4982 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 13:48:50.338630 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322148 4982 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 13:48:50.338630 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322155 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 13:48:50.338630 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322159 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 13:48:50.338630 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322163 4982 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 13:48:50.338630 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322169 4982 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 13:48:50.338630 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322173 4982 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 13:48:50.339103 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322178 4982 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 13:48:50.339103 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322182 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 13:48:50.339103 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322187 4982 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 13:48:50.339103 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322193 4982 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 13:48:50.339103 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322198 4982 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 13:48:50.339103 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322203 4982 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 13:48:50.339103 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322207 4982 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 13:48:50.339103 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322211 4982 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 13:48:50.339103 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322215 4982 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 13:48:50.339103 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322220 4982 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 13:48:50.339103 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322224 4982 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 13:48:50.339103 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322228 4982 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 13:48:50.339103 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322232 4982 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 13:48:50.339103 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322236 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 13:48:50.339103 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322240 4982 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 13:48:50.339103 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322244 4982 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 13:48:50.339103 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322249 4982 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 13:48:50.339103 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322253 4982 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 13:48:50.339103 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322257 4982 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 13:48:50.339559 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322263 4982 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 13:48:50.339559 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322267 4982 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 13:48:50.339559 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322272 4982 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 13:48:50.339559 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322276 4982 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 13:48:50.339559 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322280 4982 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 13:48:50.339559 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322284 4982 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 13:48:50.339559 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322289 4982 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 13:48:50.339559 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322293 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 13:48:50.339559 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322297 4982 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 13:48:50.339559 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322302 4982 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 13:48:50.339559 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322306 4982 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 13:48:50.339559 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322312 4982 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 13:48:50.339559 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322316 4982 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 13:48:50.339559 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322320 4982 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 13:48:50.339559 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322324 4982 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 13:48:50.339559 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322328 4982 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 13:48:50.339559 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322332 4982 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 13:48:50.339559 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322337 4982 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 13:48:50.339559 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322341 4982 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 13:48:50.339559 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322345 4982 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 13:48:50.340056 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322349 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 13:48:50.340056 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322356 4982 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 13:48:50.340056 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322363 4982 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 13:48:50.340056 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322367 4982 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 13:48:50.340056 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.322372 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 13:48:50.340056 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.325814 4982 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 13:48:50.340056 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.325829 4982 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 13:48:50.340056 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.325846 4982 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 13:48:50.340056 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.325852 4982 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 13:48:50.340056 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.325902 4982 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 13:48:50.340056 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.325907 4982 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 13:48:50.340056 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.325910 4982 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 13:48:50.340056 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.325914 4982 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 13:48:50.340056 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.325916 4982 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 13:48:50.340056 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.325919 4982 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 13:48:50.340056 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.325922 4982 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 13:48:50.340056 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.325925 4982 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 13:48:50.340056 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.325928 4982 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 13:48:50.340056 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.325931 4982 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 13:48:50.340562 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.325935 4982 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 13:48:50.340562 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.325938 4982 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 13:48:50.340562 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.325941 4982 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 13:48:50.340562 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.325943 4982 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 13:48:50.340562 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.325947 4982 feature_gate.go:386] unrecognized feature gate: Example Oct 30 13:48:50.340562 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.325949 4982 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 13:48:50.340562 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.325952 4982 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 13:48:50.340562 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.325954 4982 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 13:48:50.340562 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.325958 4982 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 13:48:50.340562 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.325962 4982 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 13:48:50.340562 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.325966 4982 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 13:48:50.340562 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.325969 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 13:48:50.340562 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.325971 4982 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 13:48:50.340562 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.325974 4982 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 13:48:50.340562 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.325977 4982 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 13:48:50.340562 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.325980 4982 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 13:48:50.340562 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.325983 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 13:48:50.340562 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.325986 4982 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 13:48:50.340562 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.325988 4982 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 13:48:50.341036 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.325992 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 13:48:50.341036 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.325995 4982 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 13:48:50.341036 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.325998 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 13:48:50.341036 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326001 4982 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 13:48:50.341036 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326003 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 13:48:50.341036 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326006 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 13:48:50.341036 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326009 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 13:48:50.341036 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326011 4982 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 13:48:50.341036 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326014 4982 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 13:48:50.341036 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326016 4982 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 13:48:50.341036 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326019 4982 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 13:48:50.341036 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326021 4982 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 13:48:50.341036 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326024 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 13:48:50.341036 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326027 4982 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 13:48:50.341036 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326029 4982 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 13:48:50.341036 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326032 4982 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 13:48:50.341036 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326034 4982 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 13:48:50.341036 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326037 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 13:48:50.341036 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326040 4982 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 13:48:50.341473 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326042 4982 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 13:48:50.341473 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326045 4982 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 13:48:50.341473 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326047 4982 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 13:48:50.341473 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326050 4982 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 13:48:50.341473 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326052 4982 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 13:48:50.341473 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326055 4982 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 13:48:50.341473 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326057 4982 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 13:48:50.341473 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326060 4982 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 13:48:50.341473 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326062 4982 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 13:48:50.341473 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326065 4982 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 13:48:50.341473 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326068 4982 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 13:48:50.341473 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326071 4982 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 13:48:50.341473 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326073 4982 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 13:48:50.341473 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326076 4982 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 13:48:50.341473 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326079 4982 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 13:48:50.341473 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326081 4982 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 13:48:50.341473 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326084 4982 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 13:48:50.341473 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326086 4982 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 13:48:50.341473 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326089 4982 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 13:48:50.341939 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326091 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 13:48:50.341939 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326094 4982 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 13:48:50.341939 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326097 4982 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 13:48:50.341939 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326099 4982 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 13:48:50.341939 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326102 4982 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 13:48:50.341939 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326104 4982 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 13:48:50.341939 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326107 4982 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 13:48:50.341939 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326110 4982 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 13:48:50.341939 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326112 4982 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 13:48:50.341939 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326115 4982 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 13:48:50.341939 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326117 4982 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 13:48:50.341939 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326120 4982 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 13:48:50.341939 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326122 4982 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 13:48:50.341939 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326125 4982 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 13:48:50.341939 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326127 4982 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 13:48:50.341939 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326130 4982 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 13:48:50.341939 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326133 4982 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 13:48:50.341939 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326135 4982 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 13:48:50.341939 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326138 4982 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 13:48:50.341939 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326141 4982 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 13:48:50.342425 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326143 4982 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 13:48:50.342425 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326146 4982 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 13:48:50.342425 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326148 4982 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 13:48:50.342425 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326151 4982 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 13:48:50.342425 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326154 4982 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 13:48:50.342425 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326157 4982 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 13:48:50.342425 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326159 4982 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 13:48:50.342425 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326162 4982 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 13:48:50.342425 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326164 4982 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 13:48:50.342425 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326167 4982 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 13:48:50.342425 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326169 4982 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 13:48:50.342425 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326172 4982 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 13:48:50.342425 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326174 4982 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 13:48:50.342425 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326177 4982 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 13:48:50.342425 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326179 4982 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 13:48:50.342425 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326182 4982 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 13:48:50.342425 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326184 4982 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 13:48:50.342425 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326187 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 13:48:50.342425 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326189 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 13:48:50.342882 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.326215 4982 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 13:48:50.342882 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326347 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 13:48:50.342882 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326354 4982 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 13:48:50.342882 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326358 4982 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 13:48:50.342882 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326362 4982 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 13:48:50.342882 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326366 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 13:48:50.342882 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326371 4982 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 13:48:50.342882 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326375 4982 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 13:48:50.342882 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326379 4982 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 13:48:50.342882 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326381 4982 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 13:48:50.342882 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326384 4982 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 13:48:50.342882 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326387 4982 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 13:48:50.342882 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326390 4982 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 13:48:50.342882 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326392 4982 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 13:48:50.342882 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326395 4982 feature_gate.go:386] unrecognized feature gate: Example Oct 30 13:48:50.342882 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326397 4982 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 13:48:50.343250 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326400 4982 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 13:48:50.343250 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326403 4982 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 13:48:50.343250 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326406 4982 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 13:48:50.343250 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326408 4982 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 13:48:50.343250 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326411 4982 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 13:48:50.343250 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326413 4982 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 13:48:50.343250 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326417 4982 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 13:48:50.343250 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326420 4982 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 13:48:50.343250 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326422 4982 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 13:48:50.343250 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326424 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 13:48:50.343250 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326427 4982 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 13:48:50.343250 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326429 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 13:48:50.343250 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326432 4982 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 13:48:50.343250 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326435 4982 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 13:48:50.343250 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326437 4982 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 13:48:50.343250 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326440 4982 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 13:48:50.343250 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326442 4982 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 13:48:50.343250 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326445 4982 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 13:48:50.343250 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326447 4982 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 13:48:50.343250 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326450 4982 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 13:48:50.343728 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326454 4982 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 13:48:50.343728 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326457 4982 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 13:48:50.343728 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326460 4982 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 13:48:50.343728 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326463 4982 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 13:48:50.343728 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326466 4982 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 13:48:50.343728 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326468 4982 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 13:48:50.343728 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326471 4982 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 13:48:50.343728 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326473 4982 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 13:48:50.343728 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326476 4982 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 13:48:50.343728 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326478 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 13:48:50.343728 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326481 4982 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 13:48:50.343728 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326483 4982 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 13:48:50.343728 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326486 4982 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 13:48:50.343728 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326488 4982 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 13:48:50.343728 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326491 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 13:48:50.343728 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326493 4982 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 13:48:50.343728 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326496 4982 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 13:48:50.343728 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326498 4982 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 13:48:50.343728 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326501 4982 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 13:48:50.344247 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326504 4982 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 13:48:50.344247 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326506 4982 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 13:48:50.344247 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326509 4982 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 13:48:50.344247 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326511 4982 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 13:48:50.344247 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326514 4982 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 13:48:50.344247 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326517 4982 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 13:48:50.344247 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326520 4982 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 13:48:50.344247 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326522 4982 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 13:48:50.344247 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326525 4982 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 13:48:50.344247 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326528 4982 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 13:48:50.344247 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326530 4982 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 13:48:50.344247 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326533 4982 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 13:48:50.344247 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326535 4982 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 13:48:50.344247 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326538 4982 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 13:48:50.344247 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326540 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 13:48:50.344247 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326543 4982 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 13:48:50.344247 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326546 4982 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 13:48:50.344247 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326548 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 13:48:50.344247 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326551 4982 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 13:48:50.344742 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326553 4982 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 13:48:50.344742 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326556 4982 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 13:48:50.344742 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326558 4982 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 13:48:50.344742 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326561 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 13:48:50.344742 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326563 4982 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 13:48:50.344742 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326566 4982 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 13:48:50.344742 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326568 4982 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 13:48:50.344742 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326571 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 13:48:50.344742 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326573 4982 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 13:48:50.344742 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326576 4982 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 13:48:50.344742 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326578 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 13:48:50.344742 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326581 4982 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 13:48:50.344742 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326583 4982 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 13:48:50.344742 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326585 4982 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 13:48:50.344742 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326588 4982 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 13:48:50.344742 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326592 4982 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 13:48:50.344742 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326596 4982 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 13:48:50.344742 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326599 4982 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 13:48:50.344742 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326601 4982 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 13:48:50.345225 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326604 4982 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 13:48:50.345225 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326607 4982 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 13:48:50.345225 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326610 4982 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 13:48:50.345225 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326612 4982 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 13:48:50.345225 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326615 4982 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 13:48:50.345225 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326617 4982 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 13:48:50.345225 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326620 4982 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 13:48:50.345225 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326622 4982 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 13:48:50.345225 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326625 4982 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 13:48:50.345225 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326627 4982 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 13:48:50.345225 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326630 4982 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 13:48:50.345225 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326632 4982 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 13:48:50.345225 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326635 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 13:48:50.345225 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326653 4982 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 13:48:50.345225 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.326678 4982 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 13:48:50.345225 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326708 4982 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 13:48:50.345696 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326712 4982 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 13:48:50.345696 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326715 4982 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 13:48:50.345696 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326718 4982 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 13:48:50.345696 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326722 4982 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 13:48:50.345696 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326726 4982 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 13:48:50.345696 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326730 4982 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 13:48:50.345696 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326732 4982 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 13:48:50.345696 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326735 4982 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 13:48:50.345696 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326738 4982 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 13:48:50.345696 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326741 4982 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 13:48:50.345696 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326743 4982 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 13:48:50.345696 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326746 4982 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 13:48:50.345696 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326749 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 13:48:50.345696 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326752 4982 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 13:48:50.345696 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326754 4982 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 13:48:50.345696 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326756 4982 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 13:48:50.345696 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326759 4982 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 13:48:50.345696 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326761 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 13:48:50.345696 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326764 4982 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 13:48:50.346452 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326766 4982 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 13:48:50.346452 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326769 4982 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 13:48:50.346452 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326771 4982 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 13:48:50.346452 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326774 4982 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 13:48:50.346452 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326776 4982 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 13:48:50.346452 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326779 4982 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 13:48:50.346452 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326781 4982 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 13:48:50.346452 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326783 4982 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 13:48:50.346452 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326786 4982 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 13:48:50.346452 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326789 4982 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 13:48:50.346452 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326791 4982 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 13:48:50.346452 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326794 4982 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 13:48:50.346452 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326796 4982 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 13:48:50.346452 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326798 4982 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 13:48:50.346452 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326801 4982 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 13:48:50.346452 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326803 4982 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 13:48:50.346452 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326810 4982 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 13:48:50.346452 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326812 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 13:48:50.346452 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326815 4982 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 13:48:50.346452 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326817 4982 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 13:48:50.347111 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326820 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 13:48:50.347111 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326822 4982 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 13:48:50.347111 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326826 4982 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 13:48:50.347111 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326829 4982 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 13:48:50.347111 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326832 4982 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 13:48:50.347111 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326834 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 13:48:50.347111 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326837 4982 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 13:48:50.347111 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326840 4982 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 13:48:50.347111 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326842 4982 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 13:48:50.347111 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326845 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 13:48:50.347111 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326848 4982 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 13:48:50.347111 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326850 4982 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 13:48:50.347111 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326852 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 13:48:50.347111 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326855 4982 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 13:48:50.347111 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326857 4982 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 13:48:50.347111 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326860 4982 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 13:48:50.347111 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326862 4982 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 13:48:50.347111 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326867 4982 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 13:48:50.347111 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326870 4982 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 13:48:50.347698 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326873 4982 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 13:48:50.347698 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326876 4982 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 13:48:50.347698 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326879 4982 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 13:48:50.347698 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326882 4982 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 13:48:50.347698 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326885 4982 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 13:48:50.347698 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326888 4982 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 13:48:50.347698 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326890 4982 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 13:48:50.347698 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326893 4982 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 13:48:50.347698 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326896 4982 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 13:48:50.347698 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326899 4982 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 13:48:50.347698 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326902 4982 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 13:48:50.347698 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326905 4982 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 13:48:50.347698 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326907 4982 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 13:48:50.347698 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326910 4982 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 13:48:50.347698 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326912 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 13:48:50.347698 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326916 4982 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 13:48:50.347698 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326919 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 13:48:50.347698 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326921 4982 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 13:48:50.347698 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326924 4982 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 13:48:50.347698 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326927 4982 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 13:48:50.348236 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326930 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 13:48:50.348236 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326933 4982 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 13:48:50.348236 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326935 4982 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 13:48:50.348236 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326938 4982 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 13:48:50.348236 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326941 4982 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 13:48:50.348236 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326944 4982 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 13:48:50.348236 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326946 4982 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 13:48:50.348236 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326949 4982 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 13:48:50.348236 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326951 4982 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 13:48:50.348236 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326954 4982 feature_gate.go:386] unrecognized feature gate: Example Oct 30 13:48:50.348236 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326956 4982 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 13:48:50.348236 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326959 4982 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 13:48:50.348236 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326962 4982 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 13:48:50.348236 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326964 4982 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 13:48:50.348236 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326967 4982 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 13:48:50.348236 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326969 4982 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 13:48:50.348236 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326972 4982 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 13:48:50.348236 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326974 4982 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 13:48:50.348236 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326978 4982 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 13:48:50.348236 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326980 4982 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 13:48:50.348823 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326983 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 13:48:50.348823 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326986 4982 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 13:48:50.348823 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326989 4982 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 13:48:50.348823 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326991 4982 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 13:48:50.348823 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326994 4982 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 13:48:50.348823 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326996 4982 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 13:48:50.348823 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:50.326999 4982 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 13:48:50.348823 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.327188 4982 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 13:48:50.348823 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.329518 4982 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 13:48:50.348823 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.329587 4982 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 13:48:50.348823 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.329855 4982 server.go:1014] "Starting client certificate rotation" Oct 30 13:48:50.348823 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.329979 4982 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 13:48:50.348823 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.330037 4982 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 13:43:10 +0000 UTC" deadline="2028-05-26 19:57:11.300136424 +0000 UTC" Oct 30 13:48:50.348823 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.330064 4982 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13782h8m20.970073954s" Oct 30 13:48:50.348823 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.330534 4982 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 13:48:50.349181 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.332087 4982 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 13:48:50.349181 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.333176 4982 log.go:25] "Validated CRI v1 runtime API" Oct 30 13:48:50.349181 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.335169 4982 log.go:25] "Validated CRI v1 image API" Oct 30 13:48:50.349181 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.335894 4982 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 13:48:50.349181 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.337599 4982 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8a52321c-5064-4169-842a-85c03a43f145:/dev/nvme0n1p3 99ed940f-ac2c-4e29-a416-ab44cbc6aa68:/dev/nvme0n1p4] Oct 30 13:48:50.349342 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.337617 4982 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/05342334dd6693d001bf65221626386848cbec0f12347b577c8df568ccf4ef04/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/05342334dd6693d001bf65221626386848cbec0f12347b577c8df568ccf4ef04/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/06456fd9d542589957a0a372e8152f9ad26074321460971ecd510d034d5ec1ba/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/06456fd9d542589957a0a372e8152f9ad26074321460971ecd510d034d5ec1ba/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/08647e20db8ce95dd4fb26e2abb25c8b6667307bd738215b3b50be2d36b4e463/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/08647e20db8ce95dd4fb26e2abb25c8b6667307bd738215b3b50be2d36b4e463/userdata/shm major:0 minor:100 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/119282a48a7da81396d040950eab5be9eed0d50580fde28f1b5714cf9ac5d7f4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/119282a48a7da81396d040950eab5be9eed0d50580fde28f1b5714cf9ac5d7f4/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1420d7fc54824d5882f58897ef950c1efcdbd4980a308175b1e490f5688683e2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1420d7fc54824d5882f58897ef950c1efcdbd4980a308175b1e490f5688683e2/userdata/shm major:0 minor:328 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1c3374fc75c25c90d2209a22f831ad738912d8e0e51227dafebb8818b4a6ea8d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1c3374fc75c25c90d2209a22f831ad738912d8e0e51227dafebb8818b4a6ea8d/userdata/shm major:0 minor:77 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/20a900a2eab78697079ac3639d0fcc4d077d0e5dcc9fd358b16c4b1215400dc0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/20a900a2eab78697079ac3639d0fcc4d077d0e5dcc9fd358b16c4b1215400dc0/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/346de427586bad84a8f4fb0455636dda4d7688ea0d7137117cfecc01d588bf21/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/346de427586bad84a8f4fb0455636dda4d7688ea0d7137117cfecc01d588bf21/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/347f2dc700ce6b660126fb5349d4f0be1ff9e5b6c7f0371edad033f971a5ee6e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/347f2dc700ce6b660126fb5349d4f0be1ff9e5b6c7f0371edad033f971a5ee6e/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/34d6135192daf776d327c4e638d6497d104f8174e400915f3e099f255171a116/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/34d6135192daf776d327c4e638d6497d104f8174e400915f3e099f255171a116/userdata/shm major:0 minor:241 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4f386b3d868858e3f05c1e4c51e74f8b9046128e43e4a207cd41e454a06e6f1f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4f386b3d868858e3f05c1e4c51e74f8b9046128e43e4a207cd41e454a06e6f1f/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5e0a45a241a6763de3ae2fcbb368e0abe7bbbf667c4256b2a91200025b1cbb18/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5e0a45a241a6763de3ae2fcbb368e0abe7bbbf667c4256b2a91200025b1cbb18/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5e13fec44995927b6b7363560b232626b27499eec9ba59e9958238bd861f3dd1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5e13fec44995927b6b7363560b232626b27499eec9ba59e9958238bd861f3dd1/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6ea406e47e234617b3a8cbb33024b7dd98c009263a6260d4f172f05267af5052/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6ea406e47e234617b3a8cbb33024b7dd98c009263a6260d4f172f05267af5052/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7ac0fb5309e0357a2007b2d08f06eef95701e311031c68d8099504d151dc465d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7ac0fb5309e0357a2007b2d08f06eef95701e311031c68d8099504d151dc465d/userdata/shm major:0 minor:304 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a7756810cb7a94ed1a71e06c72c0510df28c6247a04c0a76cce9756dea1229c5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a7756810cb7a94ed1a71e06c72c0510df28c6247a04c0a76cce9756dea1229c5/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/acfa03e8884898362b46b545016cc54a3e8c6767802f494b08ccd9e8d9a7b45e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/acfa03e8884898362b46b545016cc54a3e8c6767802f494b08ccd9e8d9a7b45e/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b7fb2959d73fac358b070abb945abc1d6f0c75a2833392da16f4e23a1b623d0f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b7fb2959d73fac358b070abb945abc1d6f0c75a2833392da16f4e23a1b623d0f/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c02569bc0c161f51caedc865f18e9bfffc42568ce87441eb0ea60b1d4c09f4c6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c02569bc0c161f51caedc865f18e9bfffc42568ce87441eb0ea60b1d4c09f4c6/userdata/shm major:0 minor:305 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c140c8c08bb48e03b8d7fcb1e70922a318eb16dcf7ad807e73a6f51047faab0f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c140c8c08bb48e03b8d7fcb1e70922a318eb16dcf7ad807e73a6f51047faab0f/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d59463f528816d4b59dd570cc059bb929537bcfab0c33399f92624e5bb8edc20/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d59463f528816d4b59dd570cc059bb929537bcfab0c33399f92624e5bb8edc20/userdata/shm major:0 minor:69 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dd0e127ce14f31020f149eb6a7d374f34ef66f6316e05086093e2736a2578e63/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dd0e127ce14f31020f149eb6a7d374f34ef66f6316e05086093e2736a2578e63/userdata/shm major:0 minor:243 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e6bed7b02e02be3eccc0a1bb65bb2b401caacedbb61032e96df1197725b262f1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e6bed7b02e02be3eccc0a1bb65bb2b401caacedbb61032e96df1197725b262f1/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f1116d41e0e81d8d5834fa70a3fbaebbc48e81b5ec63a46345ed6b7875bdd5dd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f1116d41e0e81d8d5834fa70a3fbaebbc48e81b5ec63a46345ed6b7875bdd5dd/userdata/shm major:0 minor:333 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/06f40dde-8b71-41e0-9acb-cede4f1d4347/volumes/kubernetes.io~projected/kube-api-access-7kdz4:{mountpoint:/var/lib/kubelet/pods/06f40dde-8b71-41e0-9acb-cede4f1d4347/volumes/kubernetes.io~projected/kube-api-access-7kdz4 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/083c70c2-0d5e-4876-9e88-10c6efbb681c/volumes/kubernetes.io~projected/kube-api-access-6p2xh:{mountpoint:/var/lib/kubelet/pods/083c70c2-0d5e-4876-9e88-10c6efbb681c/volumes/kubernetes.io~projected/kube-api-access-6p2xh major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203/volumes/kubernetes.io~projected/kube-api-access-xfktw:{mountpoint:/var/lib/kubelet/pods/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203/volumes/kubernetes.io~projected/kube-api-access-xfktw major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0dda4c8a-827d-428f-a6a2-5e810210662b/volumes/kubernetes.io~projected/kube-api-access-gt65k:{mountpoint:/var/lib/kubelet/pods/0dda4c8a-827d-428f-a6a2-5e810210662b/volumes/kubernetes.io~projected/kube-api-access-gt65k major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6ac8569a-95ae-421e-aee0-edd189e18d11/volumes/kubernetes.io~projected/kube-api-access-69bm9:{mountpoint:/var/lib/kubelet/pods/6ac8569a-95ae-421e-aee0-edd189e18d11/volumes/kubernetes.io~projected/kube-api-access-69bm9 major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6ac8569a-95ae-421e-aee0-edd189e18d11/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/6ac8569a-95ae-421e-aee0-edd189e18d11/volumes/kubernetes.io~secret/cert major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6b536198-f804-4868-beb9-123229c3ae51/volumes/kubernetes.io~projected/kube-api-access-5r9cl:{mountpoint:/var/lib/kubelet/pods/6b536198-f804-4868-beb9-123229c3ae51/volumes/kubernetes.io~projected/kube-api-access-5r9cl major:0 minor:342 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6b5e74da-5460-44ed-a599-b01620b64c92/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/6b5e74da-5460-44ed-a599-b01620b64c92/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:303 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6d73c6c3-044d-4b88-a105-72bd5e4491cd/volumes/kubernetes.io~projected/kube-api-access-tfgzb:{mountpoint:/var/lib/kubelet/pods/6d73c6c3-044d-4b88-a105-72bd5e4491cd/volumes/kubernetes.io~projected/kube-api-access-tfgzb major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6d73c6c3-044d-4b88-a105-72bd5e4491cd/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/6d73c6c3-044d-4b88-a105-72bd5e4491cd/volumes/kubernetes.io~secret/default-certificate major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6d73c6c3-044d-4b88-a105-72bd5e4491cd/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/6d73c6c3-044d-4b88-a105-72bd5e4491cd/volumes/kubernetes.io~secret/stats-auth major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7cd2b593-c362-43c2-b0bb-7e5021791214/volumes/kubernetes.io~projected/kube-api-access-srtk8:{mountpoint:/var/lib/kubelet/pods/7cd2b593-c362-43c2-b0bb-7e5021791214/volumes/kubernetes.io~projected/kube-api-access-srtk8 major:0 minor:318 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/806dd609-9a2a-49c4-a656-e50437457fa9/volumes/kubernetes.io~projected/kube-api-access-gk8lh:{mountpoint:/var/lib/kubelet/pods/806dd609-9a2a-49c4-a656-e50437457fa9/volumes/kubernetes.io~projected/kube-api-access-gk8lh major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/81dcad0b-d124-41b8-8b4e-d41049de217a/volumes/kubernetes.io~projected/kube-api-access-spnhx:{mountpoint:/var/lib/kubelet/pods/81dcad0b-d124-41b8-8b4e-d41049de217a/volumes/kubernetes.io~projected/kube-api-access-spnhx major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8304f8da-d4a3-45c1-bfe6-f5d08bed17d7/volumes/kubernetes.io~projected/kube-api-access-4tzg4:{mountpoint:/var/lib/kubelet/pods/8304f8da-d4a3-45c1-bfe6-f5d08bed17d7/volumes/kubernetes.io~projected/kube-api-access-4tzg4 major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8304f8da-d4a3-45c1-bfe6-f5d08bed17d7/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/8304f8da-d4a3-45c1-bfe6-f5d08bed17d7/volumes/kubernetes.io~secret/serving-cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d9908f6-aefa-4bde-9182-2c199f08a24a/volumes/kubernetes.io~projected/kube-api-access-fx49w:{mountpoint:/var/lib/kubelet/pods/8d9908f6-aefa-4bde-9182-2c199f08a24a/volumes/kubernetes.io~projected/kube-api-access-fx49w major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4bd1bb2-25ae-423c-ba05-08eb742d340e/volumes/kubernetes.io~projected/kube-api-access-pwd54:{mountpoint:/var/lib/kubelet/pods/a4bd1bb2-25ae-423c-ba05-08eb742d340e/volumes/kubernetes.io~projected/kube-api-access-pwd54 major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4bd1bb2-25ae-423c-ba05-08eb742d340e/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/a4bd1bb2-25ae-423c-ba05-08eb742d340e/volumes/kubernetes.io~secret/serving-cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a85af541-54ed-4d24-b4c5-d69244396157/volumes/kubernetes.io~projected/kube-api-access-gp4w5:{mountpoint:/var/lib/kubelet/pods/a85af541-54ed-4d24-b4c5-d69244396157/volumes/kubernetes.io~projected/kube-api-access-gp4w5 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aca47fe8-7068-4743-b6ab-283ee0874f66/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/aca47fe8-7068-4743-b6ab-283ee0874f66/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aca47fe8-7068-4743-b6ab-283ee0874f66/volumes/kubernetes.io~projected/kube-api-access-b6xq2:{mountpoint:/var/lib/kubelet/pods/aca47fe8-7068-4743-b6ab-283ee0874f66/volumes/kubernetes.io~projected/kube-api-access-b6xq2 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aca47fe8-7068-4743-b6ab-283ee0874f66/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/aca47fe8-7068-4743-b6ab-283ee0874f66/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ade75be2-e3bb-4b50-bed0-c9c257739707/volumes/kubernetes.io~projected/kube-api-access-tzwmc:{mountpoint:/var/lib/kubelet/pods/ade75be2-e3bb-4b50-bed0-c9c257739707/volumes/kubernetes.io~projected/kube-api-access-tzwmc major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ade75be2-e3bb-4b50-bed0-c9c257739707/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/ade75be2-e3bb-4b50-bed0-c9c257739707/volumes/kubernetes.io~secret/serving-cert major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b0b2d906-7995-4769-8222-fa8ba2a30f3d/volumes/kubernetes.io~projected/kube-api-access-t5qnd:{mountpoint:/var/lib/kubelet/pods/b0b2d906-7995-4769-8222-fa8ba2a30f3d/volumes/kubernetes.io~projected/kube-api-access-t5qnd major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf73153f-1421-44d6-abae-b76c3d90235e/volumes/kubernetes.io~projected/kube-api-access-mb8hg:{mountpoint:/var/lib/kubelet/pods/bf73153f-1421-44d6-abae-b76c3d90235e/volumes/kubernetes.io~projected/kube-api-access-mb8hg major:0 minor:302 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c06b4b47-6141-4263-94b3-803d9f28bb94/volumes/kubernetes.io~projected/kube-api-access-brjls:{mountpoint:/var/lib/kubelet/pods/c06b4b47-6141-4263-94b3-803d9f28bb94/volumes/kubernetes.io~projected/kube-api-access-brjls major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cb90396e-d4e9-4eb0-888f-64c59d492fd4/volumes/kubernetes.io~projected/kube-api-access-l99vz:{mountpoint:/var/lib/kubelet/pods/cb90396e-d4e9-4eb0-888f-64c59d492fd4/volumes/kubernetes.io~projected/kube-api-access-l99vz major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ccafe9ae-44d7-4c31-a6a5-ba8506bc0cea/volumes/kubernetes.io~projected/kube-api-access-xhdhm:{mountpoint:/var/lib/kubelet/pods/ccafe9ae-44d7-4c31-a6a5-ba8506bc0cea/volumes/kubernetes.io~projected/kube-api-access-xhdhm major:0 minor:323 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d38c246c-86f8-4b57-89bd-db4cd0d3f6d1/volumes/kubernetes.io~projected/kube-api-access-zdt2n:{mountpoint:/var/lib/kubelet/pods/d38c246c-86f8-4b57-89bd-db4cd0d3f6d1/volumes/kubernetes.io~projected/kube-api-access-zdt2n major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d7a64da6-7b20-4f20-90d7-bc23a71290e5/volumes/kubernetes.io~projected/kube-api-access-zkhbw:{mountpoint:/var/lib/kubelet/pods/d7a64da6-7b20-4f20-90d7-bc23a71290e5/volumes/kubernetes.io~projected/kube-api-access-zkhbw major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da694c74-3bce-4d95-bd30-12a1891ad5e6/volumes/kubernetes.io~projected/kube-api-access-bn2f4:{mountpoint:/var/lib/kubelet/pods/da694c74-3bce-4d95-bd30-12a1891ad5e6/volumes/kubernetes.io~projected/kube-api-access-bn2f4 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dad9381c-aebb-448d-80b1-e2b65dd8ac80/volumes/kubernetes.io~projected/kube-api-access-rwgrr:{mountpoint:/var/lib/kubelet/pods/dad9381c-aebb-448d-80b1-e2b65dd8ac80/volumes/kubernetes.io~projected/kube-api-access-rwgrr major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc967875-1546-40bb-8376-fde09416c204/volumes/kubernetes.io~projected/kube-api-access-n98n9:{mountpoint:/var/lib/kubelet/pods/dc967875-1546-40bb-8376-fde09416c204/volumes/kubernetes.io~projected/kube-api-access-n98n9 major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc967875-1546-40bb-8376-fde09416c204/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/dc967875-1546-40bb-8376-fde09416c204/volumes/kubernetes.io~secret/serving-cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e16ea4af-ce12-442b-91df-092457a7e192/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/e16ea4af-ce12-442b-91df-092457a7e192/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e16ea4af-ce12-442b-91df-092457a7e192/volumes/kubernetes.io~projected/kube-api-access-5v9rm:{mountpoint:/var/lib/kubelet/pods/e16ea4af-ce12-442b-91df-092457a7e192/volumes/kubernetes.io~projected/kube-api-access-5v9rm major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e16ea4af-ce12-442b-91df-092457a7e192/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/e16ea4af-ce12-442b-91df-092457a7e192/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e16ea4af-ce12-442b-91df-092457a7e192/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/e16ea4af-ce12-442b-91df-092457a7e192/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e9a035cc-e78b-4bdb-b0c6-ef13bc07249a/volumes/kubernetes.io~projected/kube-api-access-ltkwj:{mountpoint:/var/lib/kubelet/pods/e9a035cc-e78b-4bdb-b0c6-ef13bc07249a/volumes/kubernetes.io~projected/kube-api-access-ltkwj major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f707b282-e507-475d-a82a-923c9bd1ff01/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/f707b282-e507-475d-a82a-923c9bd1ff01/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f707b282-e507-475d-a82a-923c9bd1ff01/volumes/kubernetes.io~projected/kube-api-access-vm9mr:{mountpoint:/var/lib/kubelet/pods/f707b282-e507-475d-a82a-923c9bd1ff01/volumes/kubernetes.io~projected/kube-api-access-vm9mr major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f707b282-e507-475d-a82a-923c9bd1ff01/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/f707b282-e507-475d-a82a-923c9bd1ff01/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f707b282-e507-475d-a82a-923c9bd1ff01/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/f707b282-e507-475d-a82a-923c9bd1ff01/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:205 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/483bd2f2431fa458da22a08eeb4dad9d6efbef10b588f612f856b6b559dda94d/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-105:{mountpoint:/var/lib/containers/storage/overlay/d7f263cdd72c8aa4e287412545db405898fcea54825e679acb742c058a9dd4dc/merged major:0 minor:105 fsType:overlay blockSize:0} overlay_0-108:{mountpoint:/var/lib/containers/storage/overlay/3db64b62e62ca73a4c3fbdce8e5f8a57ce6b46e565284ef5a647313a0c10b74e/merged major:0 minor:108 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/26b0fc84e45a7033754bc015a5b882d540e70953e63bab303a0dd47125dd4def/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/6485977beb7a714b7446253dd0bb2b073dccee4eb699df7244be8df0d3e0efb3/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/86ae29f42c0d8695361a74d731ac530c87ce558d3413c15f5eaad828306ebc13/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/f5fe7e5daf4b0bd0ff870ac9b1df5d14d4c609abba734c0656bc480d3d959024/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/cb96b13f5aafc31f739a5b76e52517ee8147803b871d68e989304521109c238f/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/0c0e9359508334278159283689a68b80c856becaed89dcd01c520075681b9b29/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-138:{mountpoint:/var/lib/containers/storage/overlay/4476498e51a53f828046bf9f750692efb11ce2d438027f176d19d6ce67506d7d/merged major:0 minor:138 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/7ac00695de0708872f0a729a5e9f2eab29a2ad4c6e1dc17485304442553ddb28/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/917590ef232cd80a074a280733a5abbe582e0bd93706a7b260b9ac28e9df0014/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/3bf0676245ec2ef46552a6625abedccb9e841c3ac741940ea5073c134b25159a/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/6e7ad6bfc7617d857b204fa503faa5aa74b08043b043221bf030b915cecc5134/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/81c76d15b85811a0c5a45f8c6595d1c9a4c351e484e8905e08190f239f27bd0b/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/f1403240ee17f0ed0751e50b0d118782e0cfaad1fd49e33e3378a9fd931df6fb/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/d3d23fa7c2dcd8368cb100be6171b39223255dc8f7e352489fdb87fa1ade3448/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/741699c859681ab189eabf8d5cb8c54ade8fb4ffc1740b04cfaf3e801eb0468b/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/14ca7c0096e06a8f930329414fcabd1758c7c3226570853d96ba695e1cbed01e/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/8bc1f8d91889f06d6ab1af30f061a6a694d377260757d25bf234ea3e70204a5e/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/dd4357fefb26461ab1ad70b4027e6cb05480664ddc53661ab7f78f48fbdb90a1/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/bf15c173480d4e649c57325a55536c173a9ed6271a82c758575db5a78a24aa21/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/94eb2923b29ebd3cc621e7a1ca16cb7324515e4601ee7b620cc53fbdeed126cf/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/6dd7233915bc30772f4d4906aef2b67b35d393956b51c56fcf10d7cd214ebe74/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/23ba75a63157426e7d680351351bf98b2a9f4719297f7ec38f668917930e3a02/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/b880cc8cddb2e53a80ad08ccf6a5d44b2df0972c10d8bfcbf7d773e781d3709a/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/07da492ddfc3f5e9cdace6532b60a7a37dbf841e591494b86bc2c052cdd2a80e/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/9a254583bd90b05ee7d824590082a1b4160a8ebb0825b52a79d91145a1bd0831/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/e0e0f6abcec36c4f0cbd08646be4997dedce71a2a785e3cb0f3b43748b034e21/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/bddc7496883b3fe8b0e72156e2146fe949a406b7d7779c0feb7ab51dc289762d/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/239272065d1c68c4cfa238efa3df060a7bc94a52ed61b7bb18e179061e1c3883/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/b956ebce5797f582e3d2a3df60bd3330c1ce4c7511e0e04d9044f3f042c99619/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-279:{mountpoint:/var/lib/containers/storage/overlay/38cde27f80811e2ed400ee5ac7fed660d3564161dedd02b6dc9b2818c13b6520/merged major:0 minor:279 fsType:overlay blockSize:0} overlay_0-285:{mountpoint:/var/lib/containers/storage/overlay/c8ce94c70e74a7578d786e02f1010a9ba38e80244084c79c4357b337bd272b95/merged major:0 minor:285 fsType:overlay blockSize:0} overlay_0-288:{mountpoint:/var/lib/containers/storage/overlay/07f3ea845d2cd9d4e9d681362682e4bb3badfb87cd067887b78afdb52d838042/merged major:0 minor:288 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/c276d2fbb0ddaf93d0d9ccf79dd1a159affd69f177b0f870e3f94598904a51d6/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/20737a6f46faf17944f526a98e8b31e75faa6075439cb5ff96160493fab54c66/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-316:{mountpoint:/var/lib/containers/storage/overlay/c15d83f45fe0eb8cd23a01c11863ba4ffadbc77a72b92d96746c53d418d149d8/merged major:0 minor:316 fsType:overlay blockSize:0} overlay_0-324:{mountpoint:/var/lib/containers/storage/overlay/a51dd9502a44abf65629819747844a4420eb82b0342b664c08fb108bd2a09299/merged major:0 minor:324 fsType:overlay blockSize:0} overlay_0-326:{mountpoint:/var/lib/containers/storage/overlay/89405908d79cd076d4f1ac94ee1b2332f73db2ec7beccd8f699d5622a27cba99/merged major:0 minor:326 fsType:overlay blockSize:0} overlay_0-330:{mountpoint:/var/lib/containers/storage/overlay/3ef5d971a85bb0c536070770ac39fabf3123d4ff2da128816366fe2e16197e98/merged major:0 minor:330 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/8dbf7450c38443f289eaf0555325529aaea93c44bcb7116d8173b9930567e7d4/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/7d1748065fb036fd28b545c90d6e9d46abea2e17f795f28b8bff1a8bafe532ae/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-343:{mountpoint:/var/lib/containers/storage/overlay/6980db038bf0b881bf7644ed57ad6550032214da497215e6bb8293e65ee80e2f/merged major:0 minor:343 fsType:overlay blockSize:0} overlay_0-345:{mountpoint:/var/lib/containers/storage/overlay/8df63216816403515d4c64cfd336160855ef4a23b6bfb452a0d0aa9599a39998/merged major:0 minor:345 fsType:overlay blockSize:0} overlay_0-351:{mountpoint:/var/lib/containers/storage/overlay/e270d7fa873c6af697fc55cb18b043c48078b6d51047bf710c77897cb945dbe6/merged major:0 minor:351 fsType:overlay blockSize:0} overlay_0-353:{mountpoint:/var/lib/containers/storage/overlay/83b37249dca26b1da7efd6c35446d3b30b8c8874fc93c80bb2028e833b713b39/merged major:0 minor:353 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/f2b2a2a2634094117c96be426059c5d418ebe17dba3455358f139096485d31e9/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/8ad387d1c62a536a5ea1c776f1ecf513bb96838918c23956bac6578cac58eb0d/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-71:{mountpoint:/var/lib/containers/storage/overlay/0822f2981275e800546a4d4f8117324bd4f999f06a9451a951fbb71f0aaaf6a1/merged major:0 minor:71 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/f09fcc633bd4253b80c0b4ab666e19ec73d4ea6b8e39783f1981d86283d833c7/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-81:{mountpoint:/var/lib/containers/storage/overlay/b0e2cd22fe5455eb9de6233252da7b71e2fc735b656936ff028f6471b5580841/merged major:0 minor:81 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/53d1cb72edad54d894571493f4bffd4c7688497cf570502551ea2440a9ed8af6/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/a9d865350eea8f84b5d12ce2cfb6edbe4756335341a7d0abf5ef1f3254742dda/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/8b7bb353b8156b56e03fecae659c62dd35710c813b42af2439090cc2b9f3fc47/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/08526ed7289cb9df981f7c5c00e163cc56dca7edc66d4ca69e21cc1542d8be6d/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/5407c6328c8a13e96e6b7deed9ab18afb4d8be1b4e16803e8e387748e5c3ef3f/merged major:0 minor:96 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/b84e5dcec49c094066f97b1525dcf4ef04924bae89b0bb1628720f02e1b7a02f/merged major:0 minor:98 fsType:overlay blockSize:0}] Oct 30 13:48:50.349550 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.345259 4982 manager.go:217] Machine: {Timestamp:2026-10-30 13:48:50.344832919 +0000 UTC m=+0.088966952 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3197193 MemoryCapacity:33164324864 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22720807688383c6a0278a1338e7f7 SystemUUID:ec227208-0768-8383-c6a0-278a1338e7f7 BootID:bdf54b49-b7fe-4723-969d-fc69e66e639f Filesystems:[{Device:overlay_0-81 DeviceMajor:0 DeviceMinor:81 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f707b282-e507-475d-a82a-923c9bd1ff01/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:205 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e6bed7b02e02be3eccc0a1bb65bb2b401caacedbb61032e96df1197725b262f1/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-324 DeviceMajor:0 DeviceMinor:324 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/05342334dd6693d001bf65221626386848cbec0f12347b577c8df568ccf4ef04/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cb90396e-d4e9-4eb0-888f-64c59d492fd4/volumes/kubernetes.io~projected/kube-api-access-l99vz DeviceMajor:0 DeviceMinor:211 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/a4bd1bb2-25ae-423c-ba05-08eb742d340e/volumes/kubernetes.io~projected/kube-api-access-pwd54 DeviceMajor:0 DeviceMinor:215 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ccafe9ae-44d7-4c31-a6a5-ba8506bc0cea/volumes/kubernetes.io~projected/kube-api-access-xhdhm DeviceMajor:0 DeviceMinor:323 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-351 DeviceMajor:0 DeviceMinor:351 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/06f40dde-8b71-41e0-9acb-cede4f1d4347/volumes/kubernetes.io~projected/kube-api-access-7kdz4 DeviceMajor:0 DeviceMinor:59 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/aca47fe8-7068-4743-b6ab-283ee0874f66/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/dad9381c-aebb-448d-80b1-e2b65dd8ac80/volumes/kubernetes.io~projected/kube-api-access-rwgrr DeviceMajor:0 DeviceMinor:218 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5e0a45a241a6763de3ae2fcbb368e0abe7bbbf667c4256b2a91200025b1cbb18/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-316 DeviceMajor:0 DeviceMinor:316 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/aca47fe8-7068-4743-b6ab-283ee0874f66/volumes/kubernetes.io~projected/kube-api-access-b6xq2 DeviceMajor:0 DeviceMinor:62 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/6d73c6c3-044d-4b88-a105-72bd5e4491cd/volumes/kubernetes.io~projected/kube-api-access-tfgzb DeviceMajor:0 DeviceMinor:212 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/81dcad0b-d124-41b8-8b4e-d41049de217a/volumes/kubernetes.io~projected/kube-api-access-spnhx DeviceMajor:0 DeviceMinor:221 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203/volumes/kubernetes.io~projected/kube-api-access-xfktw DeviceMajor:0 DeviceMinor:63 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6d73c6c3-044d-4b88-a105-72bd5e4491cd/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:207 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-285 DeviceMajor:0 DeviceMinor:285 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-288 DeviceMajor:0 DeviceMinor:288 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6b536198-f804-4868-beb9-123229c3ae51/volumes/kubernetes.io~projected/kube-api-access-5r9cl DeviceMajor:0 DeviceMinor:342 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-105 DeviceMajor:0 DeviceMinor:105 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e16ea4af-ce12-442b-91df-092457a7e192/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:209 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-345 DeviceMajor:0 DeviceMinor:345 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8d9908f6-aefa-4bde-9182-2c199f08a24a/volumes/kubernetes.io~projected/kube-api-access-fx49w DeviceMajor:0 DeviceMinor:223 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-353 DeviceMajor:0 DeviceMinor:353 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f707b282-e507-475d-a82a-923c9bd1ff01/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:204 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/06456fd9d542589957a0a372e8152f9ad26074321460971ecd510d034d5ec1ba/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/08647e20db8ce95dd4fb26e2abb25c8b6667307bd738215b3b50be2d36b4e463/userdata/shm DeviceMajor:0 DeviceMinor:100 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a4bd1bb2-25ae-423c-ba05-08eb742d340e/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:203 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/ade75be2-e3bb-4b50-bed0-c9c257739707/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:210 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/083c70c2-0d5e-4876-9e88-10c6efbb681c/volumes/kubernetes.io~projected/kube-api-access-6p2xh DeviceMajor:0 DeviceMinor:224 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4f386b3d868858e3f05c1e4c51e74f8b9046128e43e4a207cd41e454a06e6f1f/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6b5e74da-5460-44ed-a599-b01620b64c92/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:303 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/aca47fe8-7068-4743-b6ab-283ee0874f66/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/ade75be2-e3bb-4b50-bed0-c9c257739707/volumes/kubernetes.io~projected/kube-api-access-tzwmc DeviceMajor:0 DeviceMinor:45 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1420d7fc54824d5882f58897ef950c1efcdbd4980a308175b1e490f5688683e2/userdata/shm DeviceMajor:0 DeviceMinor:328 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-108 DeviceMajor:0 DeviceMinor:108 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dc967875-1546-40bb-8376-fde09416c204/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:201 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/da694c74-3bce-4d95-bd30-12a1891ad5e6/volumes/kubernetes.io~projected/kube-api-access-bn2f4 DeviceMajor:0 DeviceMinor:67 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/347f2dc700ce6b660126fb5349d4f0be1ff9e5b6c7f0371edad033f971a5ee6e/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/8304f8da-d4a3-45c1-bfe6-f5d08bed17d7/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:197 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c02569bc0c161f51caedc865f18e9bfffc42568ce87441eb0ea60b1d4c09f4c6/userdata/shm DeviceMajor:0 DeviceMinor:305 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/7cd2b593-c362-43c2-b0bb-7e5021791214/volumes/kubernetes.io~projected/kube-api-access-srtk8 DeviceMajor:0 DeviceMinor:318 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582160384 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bf73153f-1421-44d6-abae-b76c3d90235e/volumes/kubernetes.io~projected/kube-api-access-mb8hg DeviceMajor:0 DeviceMinor:302 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-343 DeviceMajor:0 DeviceMinor:343 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/346de427586bad84a8f4fb0455636dda4d7688ea0d7137117cfecc01d588bf21/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/8304f8da-d4a3-45c1-bfe6-f5d08bed17d7/volumes/kubernetes.io~projected/kube-api-access-4tzg4 DeviceMajor:0 DeviceMinor:216 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/806dd609-9a2a-49c4-a656-e50437457fa9/volumes/kubernetes.io~projected/kube-api-access-gk8lh DeviceMajor:0 DeviceMinor:219 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7ac0fb5309e0357a2007b2d08f06eef95701e311031c68d8099504d151dc465d/userdata/shm DeviceMajor:0 DeviceMinor:304 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/a85af541-54ed-4d24-b4c5-d69244396157/volumes/kubernetes.io~projected/kube-api-access-gp4w5 DeviceMajor:0 DeviceMinor:60 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-71 DeviceMajor:0 DeviceMinor:71 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f707b282-e507-475d-a82a-923c9bd1ff01/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:214 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1c3374fc75c25c90d2209a22f831ad738912d8e0e51227dafebb8818b4a6ea8d/userdata/shm DeviceMajor:0 DeviceMinor:77 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/20a900a2eab78697079ac3639d0fcc4d077d0e5dcc9fd358b16c4b1215400dc0/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dd0e127ce14f31020f149eb6a7d374f34ef66f6316e05086093e2736a2578e63/userdata/shm DeviceMajor:0 DeviceMinor:243 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-279 DeviceMajor:0 DeviceMinor:279 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e16ea4af-ce12-442b-91df-092457a7e192/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:202 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/d7a64da6-7b20-4f20-90d7-bc23a71290e5/volumes/kubernetes.io~projected/kube-api-access-zkhbw DeviceMajor:0 DeviceMinor:226 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582164480 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d59463f528816d4b59dd570cc059bb929537bcfab0c33399f92624e5bb8edc20/userdata/shm DeviceMajor:0 DeviceMinor:69 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e9a035cc-e78b-4bdb-b0c6-ef13bc07249a/volumes/kubernetes.io~projected/kube-api-access-ltkwj DeviceMajor:0 DeviceMinor:64 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/d38c246c-86f8-4b57-89bd-db4cd0d3f6d1/volumes/kubernetes.io~projected/kube-api-access-zdt2n DeviceMajor:0 DeviceMinor:61 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/119282a48a7da81396d040950eab5be9eed0d50580fde28f1b5714cf9ac5d7f4/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/b0b2d906-7995-4769-8222-fa8ba2a30f3d/volumes/kubernetes.io~projected/kube-api-access-t5qnd DeviceMajor:0 DeviceMinor:213 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a7756810cb7a94ed1a71e06c72c0510df28c6247a04c0a76cce9756dea1229c5/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-138 DeviceMajor:0 DeviceMinor:138 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6d73c6c3-044d-4b88-a105-72bd5e4491cd/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:208 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/c06b4b47-6141-4263-94b3-803d9f28bb94/volumes/kubernetes.io~projected/kube-api-access-brjls DeviceMajor:0 DeviceMinor:65 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6ac8569a-95ae-421e-aee0-edd189e18d11/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:206 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b7fb2959d73fac358b070abb945abc1d6f0c75a2833392da16f4e23a1b623d0f/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-330 DeviceMajor:0 DeviceMinor:330 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c140c8c08bb48e03b8d7fcb1e70922a318eb16dcf7ad807e73a6f51047faab0f/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/e16ea4af-ce12-442b-91df-092457a7e192/volumes/kubernetes.io~projected/kube-api-access-5v9rm DeviceMajor:0 DeviceMinor:225 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6ea406e47e234617b3a8cbb33024b7dd98c009263a6260d4f172f05267af5052/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/34d6135192daf776d327c4e638d6497d104f8174e400915f3e099f255171a116/userdata/shm DeviceMajor:0 DeviceMinor:241 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f1116d41e0e81d8d5834fa70a3fbaebbc48e81b5ec63a46345ed6b7875bdd5dd/userdata/shm DeviceMajor:0 DeviceMinor:333 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/0dda4c8a-827d-428f-a6a2-5e810210662b/volumes/kubernetes.io~projected/kube-api-access-gt65k DeviceMajor:0 DeviceMinor:66 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e16ea4af-ce12-442b-91df-092457a7e192/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:222 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-326 DeviceMajor:0 DeviceMinor:326 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5e13fec44995927b6b7363560b232626b27499eec9ba59e9958238bd861f3dd1/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/f707b282-e507-475d-a82a-923c9bd1ff01/volumes/kubernetes.io~projected/kube-api-access-vm9mr DeviceMajor:0 DeviceMinor:220 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/dc967875-1546-40bb-8376-fde09416c204/volumes/kubernetes.io~projected/kube-api-access-n98n9 DeviceMajor:0 DeviceMinor:227 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/acfa03e8884898362b46b545016cc54a3e8c6767802f494b08ccd9e8d9a7b45e/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6ac8569a-95ae-421e-aee0-edd189e18d11/volumes/kubernetes.io~projected/kube-api-access-69bm9 DeviceMajor:0 DeviceMinor:217 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:06456fd9d542589 MacAddress:86:43:3d:df:dc:44 Speed:10000 Mtu:8901} {Name:1420d7fc54824d5 MacAddress:3e:e8:f6:0f:3c:4f Speed:10000 Mtu:8901} {Name:34d6135192daf77 MacAddress:16:6e:ee:6b:da:a9 Speed:10000 Mtu:8901} {Name:4f386b3d868858e MacAddress:36:59:de:b6:a0:69 Speed:10000 Mtu:8901} {Name:5e0a45a241a6763 MacAddress:66:a0:70:2d:ed:47 Speed:10000 Mtu:8901} {Name:6ea406e47e23461 MacAddress:e6:90:dc:01:72:c6 Speed:10000 Mtu:8901} {Name:7ac0fb5309e0357 MacAddress:5a:76:46:55:f8:9b Speed:10000 Mtu:8901} {Name:a7756810cb7a94e MacAddress:82:a3:85:4a:21:9a Speed:10000 Mtu:8901} {Name:b7fb2959d73fac3 MacAddress:86:92:88:8b:67:8c Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:f1:78:7b:9f Speed:0 Mtu:9001} {Name:br-int MacAddress:52:5e:57:d1:9c:60 Speed:0 Mtu:8901} {Name:c02569bc0c161f5 MacAddress:c2:a4:b5:42:ef:7e Speed:10000 Mtu:8901} {Name:c140c8c08bb48e0 MacAddress:1e:a3:1c:93:ab:3c Speed:10000 Mtu:8901} {Name:dd0e127ce14f310 MacAddress:aa:1b:a1:a5:6a:ef Speed:10000 Mtu:8901} {Name:e6bed7b02e02be3 MacAddress:56:0c:73:9a:e2:50 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:f1:78:7b:9f Speed:0 Mtu:9001} {Name:f1116d41e0e81d8 MacAddress:c6:62:04:fb:ca:63 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:9a:60:69:60:50:90 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:4e:bd:48:89:d4:82 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164324864 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 13:48:50.349550 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.345939 4982 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 13:48:50.349550 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.345998 4982 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 13:48:50.349550 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.346311 4982 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 13:48:50.349550 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.346349 4982 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-105.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 13:48:50.349550 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.346869 4982 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 13:48:50.349550 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.346880 4982 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 13:48:50.349550 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.346895 4982 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 13:48:50.349550 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.346907 4982 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 13:48:50.349550 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.346966 4982 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 13:48:50.349550 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.347120 4982 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 13:48:50.349550 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.347229 4982 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 13:48:50.349550 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.347261 4982 kubelet.go:493] "Attempting to sync node with API server" Oct 30 13:48:50.349550 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.347270 4982 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 13:48:50.349550 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.347278 4982 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 13:48:50.349550 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.347287 4982 kubelet.go:405] "Adding apiserver pod source" Oct 30 13:48:50.349550 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.347294 4982 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 13:48:50.349550 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.347884 4982 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 13:48:50.349550 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.347893 4982 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 13:48:50.349550 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.348372 4982 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 13:48:50.350436 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.350411 4982 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 13:48:50.350436 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.350426 4982 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 13:48:50.350535 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.350524 4982 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 13:48:50.350711 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.350699 4982 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 13:48:50.350742 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.350728 4982 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 13:48:50.350817 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.350809 4982 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 13:48:50.350846 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.350822 4982 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 13:48:50.350846 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.350828 4982 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 13:48:50.350846 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.350834 4982 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 13:48:50.350846 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.350840 4982 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 13:48:50.350846 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.350846 4982 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 13:48:50.350968 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.350852 4982 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 13:48:50.350968 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.350858 4982 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 13:48:50.350968 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.350870 4982 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 13:48:50.350968 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.350876 4982 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 13:48:50.350968 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.350888 4982 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 13:48:50.350968 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.350897 4982 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 13:48:50.350968 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.350920 4982 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 13:48:50.350968 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.350926 4982 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 13:48:50.353959 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.353946 4982 server.go:1263] "Started kubelet" Oct 30 13:48:50.354060 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.354011 4982 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 13:48:50.354117 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.354042 4982 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 13:48:50.354117 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.354105 4982 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 13:48:50.354377 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.354357 4982 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 13:48:50.354433 ip-10-0-140-105 systemd[1]: Started Kubernetes Kubelet. Oct 30 13:48:50.355691 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.355675 4982 server.go:317] "Adding debug handlers to kubelet server" Oct 30 13:48:50.357713 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.357457 4982 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 13:48:50.357713 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.357470 4982 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 13:48:50.357713 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.357521 4982 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 13:43:12 +0000 UTC" deadline="2028-05-16 06:28:08.694854074 +0000 UTC" Oct 30 13:48:50.357713 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.357551 4982 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13528h39m18.337306354s" Oct 30 13:48:50.358286 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.358059 4982 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 13:48:50.358286 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.358070 4982 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 13:48:50.358286 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.358141 4982 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 13:48:50.358847 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:48:50.358816 4982 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-105.ec2.internal\" not found" Oct 30 13:48:50.362368 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.362334 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f707b282-e507-475d-a82a-923c9bd1ff01" volumeName="kubernetes.io/secret/f707b282-e507-475d-a82a-923c9bd1ff01-installation-pull-secrets" seLinuxMountContext="" Oct 30 13:48:50.362456 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.362369 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ac8569a-95ae-421e-aee0-edd189e18d11" volumeName="kubernetes.io/secret/6ac8569a-95ae-421e-aee0-edd189e18d11-cert" seLinuxMountContext="" Oct 30 13:48:50.362456 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.362383 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7a64da6-7b20-4f20-90d7-bc23a71290e5" volumeName="kubernetes.io/projected/d7a64da6-7b20-4f20-90d7-bc23a71290e5-kube-api-access-zkhbw" seLinuxMountContext="" Oct 30 13:48:50.362456 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.362394 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b5e74da-5460-44ed-a599-b01620b64c92" volumeName="kubernetes.io/secret/6b5e74da-5460-44ed-a599-b01620b64c92-original-pull-secret" seLinuxMountContext="" Oct 30 13:48:50.362456 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.362401 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4bd1bb2-25ae-423c-ba05-08eb742d340e" volumeName="kubernetes.io/configmap/a4bd1bb2-25ae-423c-ba05-08eb742d340e-trusted-ca-bundle" seLinuxMountContext="" Oct 30 13:48:50.362456 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.362408 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4bd1bb2-25ae-423c-ba05-08eb742d340e" volumeName="kubernetes.io/empty-dir/a4bd1bb2-25ae-423c-ba05-08eb742d340e-tmp" seLinuxMountContext="" Oct 30 13:48:50.362456 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.362415 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ade75be2-e3bb-4b50-bed0-c9c257739707" volumeName="kubernetes.io/projected/ade75be2-e3bb-4b50-bed0-c9c257739707-kube-api-access-tzwmc" seLinuxMountContext="" Oct 30 13:48:50.362456 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.362422 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c06b4b47-6141-4263-94b3-803d9f28bb94" volumeName="kubernetes.io/projected/c06b4b47-6141-4263-94b3-803d9f28bb94-kube-api-access-brjls" seLinuxMountContext="" Oct 30 13:48:50.362456 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.362431 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="806dd609-9a2a-49c4-a656-e50437457fa9" volumeName="kubernetes.io/projected/806dd609-9a2a-49c4-a656-e50437457fa9-kube-api-access-gk8lh" seLinuxMountContext="" Oct 30 13:48:50.362456 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.362438 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81dcad0b-d124-41b8-8b4e-d41049de217a" volumeName="kubernetes.io/projected/81dcad0b-d124-41b8-8b4e-d41049de217a-kube-api-access-spnhx" seLinuxMountContext="" Oct 30 13:48:50.362456 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.362446 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d9908f6-aefa-4bde-9182-2c199f08a24a" volumeName="kubernetes.io/projected/8d9908f6-aefa-4bde-9182-2c199f08a24a-kube-api-access-fx49w" seLinuxMountContext="" Oct 30 13:48:50.362456 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.362452 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4bd1bb2-25ae-423c-ba05-08eb742d340e" volumeName="kubernetes.io/projected/a4bd1bb2-25ae-423c-ba05-08eb742d340e-kube-api-access-pwd54" seLinuxMountContext="" Oct 30 13:48:50.362456 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.362460 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dad9381c-aebb-448d-80b1-e2b65dd8ac80" volumeName="kubernetes.io/projected/dad9381c-aebb-448d-80b1-e2b65dd8ac80-kube-api-access-rwgrr" seLinuxMountContext="" Oct 30 13:48:50.362879 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.362467 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e16ea4af-ce12-442b-91df-092457a7e192" volumeName="kubernetes.io/secret/e16ea4af-ce12-442b-91df-092457a7e192-installation-pull-secrets" seLinuxMountContext="" Oct 30 13:48:50.362879 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.362476 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ade75be2-e3bb-4b50-bed0-c9c257739707" volumeName="kubernetes.io/configmap/ade75be2-e3bb-4b50-bed0-c9c257739707-config" seLinuxMountContext="" Oct 30 13:48:50.362879 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.362485 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8304f8da-d4a3-45c1-bfe6-f5d08bed17d7" volumeName="kubernetes.io/projected/8304f8da-d4a3-45c1-bfe6-f5d08bed17d7-kube-api-access-4tzg4" seLinuxMountContext="" Oct 30 13:48:50.362879 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.362491 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="083c70c2-0d5e-4876-9e88-10c6efbb681c" volumeName="kubernetes.io/empty-dir/083c70c2-0d5e-4876-9e88-10c6efbb681c-tmp-dir" seLinuxMountContext="" Oct 30 13:48:50.362879 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.362497 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a5f2fbe-141f-4beb-a5b5-0e2ba4686203" volumeName="kubernetes.io/empty-dir/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-etc-tuned" seLinuxMountContext="" Oct 30 13:48:50.362879 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.362503 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0dda4c8a-827d-428f-a6a2-5e810210662b" volumeName="kubernetes.io/configmap/0dda4c8a-827d-428f-a6a2-5e810210662b-cni-binary-copy" seLinuxMountContext="" Oct 30 13:48:50.362879 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.362510 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0dda4c8a-827d-428f-a6a2-5e810210662b" volumeName="kubernetes.io/configmap/0dda4c8a-827d-428f-a6a2-5e810210662b-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 13:48:50.362879 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.362515 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81dcad0b-d124-41b8-8b4e-d41049de217a" volumeName="kubernetes.io/configmap/81dcad0b-d124-41b8-8b4e-d41049de217a-samples-operator-config" seLinuxMountContext="" Oct 30 13:48:50.362879 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.362522 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0dda4c8a-827d-428f-a6a2-5e810210662b" volumeName="kubernetes.io/configmap/0dda4c8a-827d-428f-a6a2-5e810210662b-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 13:48:50.362879 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.362529 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b536198-f804-4868-beb9-123229c3ae51" volumeName="kubernetes.io/empty-dir/6b536198-f804-4868-beb9-123229c3ae51-data-volume" seLinuxMountContext="" Oct 30 13:48:50.362879 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.362535 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4bd1bb2-25ae-423c-ba05-08eb742d340e" volumeName="kubernetes.io/configmap/a4bd1bb2-25ae-423c-ba05-08eb742d340e-service-ca-bundle" seLinuxMountContext="" Oct 30 13:48:50.362879 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.362542 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a85af541-54ed-4d24-b4c5-d69244396157" volumeName="kubernetes.io/configmap/a85af541-54ed-4d24-b4c5-d69244396157-cni-binary-copy" seLinuxMountContext="" Oct 30 13:48:50.362879 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.362549 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ccafe9ae-44d7-4c31-a6a5-ba8506bc0cea" volumeName="kubernetes.io/projected/ccafe9ae-44d7-4c31-a6a5-ba8506bc0cea-kube-api-access-xhdhm" seLinuxMountContext="" Oct 30 13:48:50.362879 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.362556 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da694c74-3bce-4d95-bd30-12a1891ad5e6" volumeName="kubernetes.io/projected/da694c74-3bce-4d95-bd30-12a1891ad5e6-kube-api-access-bn2f4" seLinuxMountContext="" Oct 30 13:48:50.362879 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.362563 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0dda4c8a-827d-428f-a6a2-5e810210662b" volumeName="kubernetes.io/projected/0dda4c8a-827d-428f-a6a2-5e810210662b-kube-api-access-gt65k" seLinuxMountContext="" Oct 30 13:48:50.362879 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.362573 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6d73c6c3-044d-4b88-a105-72bd5e4491cd" volumeName="kubernetes.io/secret/6d73c6c3-044d-4b88-a105-72bd5e4491cd-stats-auth" seLinuxMountContext="" Oct 30 13:48:50.362879 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.362579 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0b2d906-7995-4769-8222-fa8ba2a30f3d" volumeName="kubernetes.io/configmap/b0b2d906-7995-4769-8222-fa8ba2a30f3d-telemetry-config" seLinuxMountContext="" Oct 30 13:48:50.362879 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.362586 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ac8569a-95ae-421e-aee0-edd189e18d11" volumeName="kubernetes.io/projected/6ac8569a-95ae-421e-aee0-edd189e18d11-kube-api-access-69bm9" seLinuxMountContext="" Oct 30 13:48:50.362879 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.362593 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4bd1bb2-25ae-423c-ba05-08eb742d340e" volumeName="kubernetes.io/empty-dir/a4bd1bb2-25ae-423c-ba05-08eb742d340e-snapshots" seLinuxMountContext="" Oct 30 13:48:50.362879 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.362600 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc967875-1546-40bb-8376-fde09416c204" volumeName="kubernetes.io/projected/dc967875-1546-40bb-8376-fde09416c204-kube-api-access-n98n9" seLinuxMountContext="" Oct 30 13:48:50.362879 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.362607 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9a035cc-e78b-4bdb-b0c6-ef13bc07249a" volumeName="kubernetes.io/empty-dir/e9a035cc-e78b-4bdb-b0c6-ef13bc07249a-tmp-dir" seLinuxMountContext="" Oct 30 13:48:50.362879 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.362613 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f707b282-e507-475d-a82a-923c9bd1ff01" volumeName="kubernetes.io/configmap/f707b282-e507-475d-a82a-923c9bd1ff01-trusted-ca" seLinuxMountContext="" Oct 30 13:48:50.362879 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.362620 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b536198-f804-4868-beb9-123229c3ae51" volumeName="kubernetes.io/projected/6b536198-f804-4868-beb9-123229c3ae51-kube-api-access-5r9cl" seLinuxMountContext="" Oct 30 13:48:50.362879 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.362626 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4bd1bb2-25ae-423c-ba05-08eb742d340e" volumeName="kubernetes.io/secret/a4bd1bb2-25ae-423c-ba05-08eb742d340e-serving-cert" seLinuxMountContext="" Oct 30 13:48:50.362879 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.362633 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aca47fe8-7068-4743-b6ab-283ee0874f66" volumeName="kubernetes.io/configmap/aca47fe8-7068-4743-b6ab-283ee0874f66-ovnkube-config" seLinuxMountContext="" Oct 30 13:48:50.362879 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.362751 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e16ea4af-ce12-442b-91df-092457a7e192" volumeName="kubernetes.io/empty-dir/e16ea4af-ce12-442b-91df-092457a7e192-ca-trust-extracted" seLinuxMountContext="" Oct 30 13:48:50.362879 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.362759 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9a035cc-e78b-4bdb-b0c6-ef13bc07249a" volumeName="kubernetes.io/projected/e9a035cc-e78b-4bdb-b0c6-ef13bc07249a-kube-api-access-ltkwj" seLinuxMountContext="" Oct 30 13:48:50.362879 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.362766 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a5f2fbe-141f-4beb-a5b5-0e2ba4686203" volumeName="kubernetes.io/projected/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-kube-api-access-xfktw" seLinuxMountContext="" Oct 30 13:48:50.362879 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.362777 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aca47fe8-7068-4743-b6ab-283ee0874f66" volumeName="kubernetes.io/configmap/aca47fe8-7068-4743-b6ab-283ee0874f66-env-overrides" seLinuxMountContext="" Oct 30 13:48:50.362879 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.362786 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aca47fe8-7068-4743-b6ab-283ee0874f66" volumeName="kubernetes.io/secret/aca47fe8-7068-4743-b6ab-283ee0874f66-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 13:48:50.362879 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.362793 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c06b4b47-6141-4263-94b3-803d9f28bb94" volumeName="kubernetes.io/configmap/c06b4b47-6141-4263-94b3-803d9f28bb94-serviceca" seLinuxMountContext="" Oct 30 13:48:50.362879 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.362804 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb90396e-d4e9-4eb0-888f-64c59d492fd4" volumeName="kubernetes.io/projected/cb90396e-d4e9-4eb0-888f-64c59d492fd4-kube-api-access-l99vz" seLinuxMountContext="" Oct 30 13:48:50.362879 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.362812 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d38c246c-86f8-4b57-89bd-db4cd0d3f6d1" volumeName="kubernetes.io/projected/d38c246c-86f8-4b57-89bd-db4cd0d3f6d1-kube-api-access-zdt2n" seLinuxMountContext="" Oct 30 13:48:50.362879 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.362822 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a85af541-54ed-4d24-b4c5-d69244396157" volumeName="kubernetes.io/projected/a85af541-54ed-4d24-b4c5-d69244396157-kube-api-access-gp4w5" seLinuxMountContext="" Oct 30 13:48:50.362879 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.362831 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aca47fe8-7068-4743-b6ab-283ee0874f66" volumeName="kubernetes.io/projected/aca47fe8-7068-4743-b6ab-283ee0874f66-kube-api-access-b6xq2" seLinuxMountContext="" Oct 30 13:48:50.362879 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.362840 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad85cf2a-6c44-4e85-8677-8b07cb9f78a4" volumeName="kubernetes.io/configmap/ad85cf2a-6c44-4e85-8677-8b07cb9f78a4-konnectivity-ca" seLinuxMountContext="" Oct 30 13:48:50.362879 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.362848 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf73153f-1421-44d6-abae-b76c3d90235e" volumeName="kubernetes.io/projected/bf73153f-1421-44d6-abae-b76c3d90235e-kube-api-access-mb8hg" seLinuxMountContext="" Oct 30 13:48:50.362879 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.362855 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8309f073-dcb0-4971-9eb1-8a9ee0eb68c6" volumeName="kubernetes.io/configmap/8309f073-dcb0-4971-9eb1-8a9ee0eb68c6-nginx-conf" seLinuxMountContext="" Oct 30 13:48:50.362879 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.362863 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06f40dde-8b71-41e0-9acb-cede4f1d4347" volumeName="kubernetes.io/projected/06f40dde-8b71-41e0-9acb-cede4f1d4347-kube-api-access-7kdz4" seLinuxMountContext="" Oct 30 13:48:50.362879 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.362870 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc967875-1546-40bb-8376-fde09416c204" volumeName="kubernetes.io/secret/dc967875-1546-40bb-8376-fde09416c204-serving-cert" seLinuxMountContext="" Oct 30 13:48:50.362879 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.362877 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="083c70c2-0d5e-4876-9e88-10c6efbb681c" volumeName="kubernetes.io/configmap/083c70c2-0d5e-4876-9e88-10c6efbb681c-config-volume" seLinuxMountContext="" Oct 30 13:48:50.362879 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.362887 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="083c70c2-0d5e-4876-9e88-10c6efbb681c" volumeName="kubernetes.io/projected/083c70c2-0d5e-4876-9e88-10c6efbb681c-kube-api-access-6p2xh" seLinuxMountContext="" Oct 30 13:48:50.362879 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.362897 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8304f8da-d4a3-45c1-bfe6-f5d08bed17d7" volumeName="kubernetes.io/configmap/8304f8da-d4a3-45c1-bfe6-f5d08bed17d7-config" seLinuxMountContext="" Oct 30 13:48:50.364835 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.362916 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aca47fe8-7068-4743-b6ab-283ee0874f66" volumeName="kubernetes.io/configmap/aca47fe8-7068-4743-b6ab-283ee0874f66-ovnkube-script-lib" seLinuxMountContext="" Oct 30 13:48:50.364835 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.362926 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad85cf2a-6c44-4e85-8677-8b07cb9f78a4" volumeName="kubernetes.io/secret/ad85cf2a-6c44-4e85-8677-8b07cb9f78a4-agent-certs" seLinuxMountContext="" Oct 30 13:48:50.364835 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.362933 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d38c246c-86f8-4b57-89bd-db4cd0d3f6d1" volumeName="kubernetes.io/configmap/d38c246c-86f8-4b57-89bd-db4cd0d3f6d1-iptables-alerter-script" seLinuxMountContext="" Oct 30 13:48:50.364835 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.362940 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc967875-1546-40bb-8376-fde09416c204" volumeName="kubernetes.io/configmap/dc967875-1546-40bb-8376-fde09416c204-trusted-ca" seLinuxMountContext="" Oct 30 13:48:50.364835 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.362963 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e16ea4af-ce12-442b-91df-092457a7e192" volumeName="kubernetes.io/configmap/e16ea4af-ce12-442b-91df-092457a7e192-registry-certificates" seLinuxMountContext="" Oct 30 13:48:50.364835 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.362971 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a5f2fbe-141f-4beb-a5b5-0e2ba4686203" volumeName="kubernetes.io/empty-dir/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-tmp" seLinuxMountContext="" Oct 30 13:48:50.364835 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.362980 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8304f8da-d4a3-45c1-bfe6-f5d08bed17d7" volumeName="kubernetes.io/secret/8304f8da-d4a3-45c1-bfe6-f5d08bed17d7-serving-cert" seLinuxMountContext="" Oct 30 13:48:50.364835 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.362988 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ade75be2-e3bb-4b50-bed0-c9c257739707" volumeName="kubernetes.io/secret/ade75be2-e3bb-4b50-bed0-c9c257739707-serving-cert" seLinuxMountContext="" Oct 30 13:48:50.364835 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.362995 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0b2d906-7995-4769-8222-fa8ba2a30f3d" volumeName="kubernetes.io/projected/b0b2d906-7995-4769-8222-fa8ba2a30f3d-kube-api-access-t5qnd" seLinuxMountContext="" Oct 30 13:48:50.364835 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.363001 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e16ea4af-ce12-442b-91df-092457a7e192" volumeName="kubernetes.io/projected/e16ea4af-ce12-442b-91df-092457a7e192-kube-api-access-5v9rm" seLinuxMountContext="" Oct 30 13:48:50.364835 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.363009 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e16ea4af-ce12-442b-91df-092457a7e192" volumeName="kubernetes.io/secret/e16ea4af-ce12-442b-91df-092457a7e192-image-registry-private-configuration" seLinuxMountContext="" Oct 30 13:48:50.364835 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.363020 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f707b282-e507-475d-a82a-923c9bd1ff01" volumeName="kubernetes.io/projected/f707b282-e507-475d-a82a-923c9bd1ff01-bound-sa-token" seLinuxMountContext="" Oct 30 13:48:50.364835 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.363030 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f707b282-e507-475d-a82a-923c9bd1ff01" volumeName="kubernetes.io/projected/f707b282-e507-475d-a82a-923c9bd1ff01-kube-api-access-vm9mr" seLinuxMountContext="" Oct 30 13:48:50.364835 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.363043 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7cd2b593-c362-43c2-b0bb-7e5021791214" volumeName="kubernetes.io/projected/7cd2b593-c362-43c2-b0bb-7e5021791214-kube-api-access-srtk8" seLinuxMountContext="" Oct 30 13:48:50.364835 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.363054 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a85af541-54ed-4d24-b4c5-d69244396157" volumeName="kubernetes.io/configmap/a85af541-54ed-4d24-b4c5-d69244396157-multus-daemon-config" seLinuxMountContext="" Oct 30 13:48:50.364835 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.363065 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc967875-1546-40bb-8376-fde09416c204" volumeName="kubernetes.io/configmap/dc967875-1546-40bb-8376-fde09416c204-config" seLinuxMountContext="" Oct 30 13:48:50.364835 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.363076 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e16ea4af-ce12-442b-91df-092457a7e192" volumeName="kubernetes.io/configmap/e16ea4af-ce12-442b-91df-092457a7e192-trusted-ca" seLinuxMountContext="" Oct 30 13:48:50.364835 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.363086 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e16ea4af-ce12-442b-91df-092457a7e192" volumeName="kubernetes.io/projected/e16ea4af-ce12-442b-91df-092457a7e192-bound-sa-token" seLinuxMountContext="" Oct 30 13:48:50.364835 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.363097 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f707b282-e507-475d-a82a-923c9bd1ff01" volumeName="kubernetes.io/configmap/f707b282-e507-475d-a82a-923c9bd1ff01-registry-certificates" seLinuxMountContext="" Oct 30 13:48:50.364835 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.363108 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f707b282-e507-475d-a82a-923c9bd1ff01" volumeName="kubernetes.io/empty-dir/f707b282-e507-475d-a82a-923c9bd1ff01-ca-trust-extracted" seLinuxMountContext="" Oct 30 13:48:50.364835 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.363120 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6d73c6c3-044d-4b88-a105-72bd5e4491cd" volumeName="kubernetes.io/projected/6d73c6c3-044d-4b88-a105-72bd5e4491cd-kube-api-access-tfgzb" seLinuxMountContext="" Oct 30 13:48:50.364835 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.363139 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6d73c6c3-044d-4b88-a105-72bd5e4491cd" volumeName="kubernetes.io/secret/6d73c6c3-044d-4b88-a105-72bd5e4491cd-default-certificate" seLinuxMountContext="" Oct 30 13:48:50.364835 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.363150 4982 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f707b282-e507-475d-a82a-923c9bd1ff01" volumeName="kubernetes.io/secret/f707b282-e507-475d-a82a-923c9bd1ff01-image-registry-private-configuration" seLinuxMountContext="" Oct 30 13:48:50.364835 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.363160 4982 reconstruct.go:97] "Volume reconstruction finished" Oct 30 13:48:50.364835 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.363168 4982 reconciler.go:29] "Reconciler: start to sync state" Oct 30 13:48:50.364835 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.363228 4982 factory.go:55] Registering systemd factory Oct 30 13:48:50.364835 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.363243 4982 factory.go:223] Registration of the systemd container factory successfully Oct 30 13:48:50.364835 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.363461 4982 factory.go:153] Registering CRI-O factory Oct 30 13:48:50.364835 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.363475 4982 factory.go:223] Registration of the crio container factory successfully Oct 30 13:48:50.364835 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.363513 4982 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 13:48:50.364835 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.363525 4982 factory.go:103] Registering Raw factory Oct 30 13:48:50.364835 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.363538 4982 manager.go:1199] Started watching for new ooms in manager Oct 30 13:48:50.364835 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.363889 4982 manager.go:319] Starting recovery of all containers Oct 30 13:48:50.389498 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.389447 4982 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 13:48:50.390344 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.390321 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ntlqq_dc967875-1546-40bb-8376-fde09416c204/console-operator/1.log" Oct 30 13:48:50.391679 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.391038 4982 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 13:48:50.391679 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.391062 4982 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 13:48:50.391679 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.391085 4982 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 13:48:50.391679 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:48:50.391123 4982 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 13:48:50.391679 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.391219 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ntlqq_dc967875-1546-40bb-8376-fde09416c204/console-operator/1.log" Oct 30 13:48:50.397700 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.397679 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhxjf_aca47fe8-7068-4743-b6ab-283ee0874f66/ovn-acl-logging/0.log" Oct 30 13:48:50.399075 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.399058 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhxjf_aca47fe8-7068-4743-b6ab-283ee0874f66/ovn-acl-logging/0.log" Oct 30 13:48:50.404115 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.404083 4982 generic.go:356] "Generic (PLEG): container finished" podID="0dda4c8a-827d-428f-a6a2-5e810210662b" containerID="85e5d87cab66a35b7dbf08ab96bc34e3f943ff7f6e478882ba29c0a44f7b0fb8" exitCode=0 Oct 30 13:48:50.404115 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.404114 4982 generic.go:356] "Generic (PLEG): container finished" podID="0dda4c8a-827d-428f-a6a2-5e810210662b" containerID="c3ec9ef7c73e9f870bf3caa22448c73c3073cc854248a974ffe2e795228e3781" exitCode=0 Oct 30 13:48:50.404282 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.404125 4982 generic.go:356] "Generic (PLEG): container finished" podID="0dda4c8a-827d-428f-a6a2-5e810210662b" containerID="44b719f9964236c03a9d2376a362899e228ac68027da40d28da79315f475e998" exitCode=0 Oct 30 13:48:50.404282 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.404134 4982 generic.go:356] "Generic (PLEG): container finished" podID="0dda4c8a-827d-428f-a6a2-5e810210662b" containerID="5c8d71f8e893bf7624b4b799e3b9279d85c5ffdbaa72b569f690327f4cf272c0" exitCode=0 Oct 30 13:48:50.404282 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.404144 4982 generic.go:356] "Generic (PLEG): container finished" podID="0dda4c8a-827d-428f-a6a2-5e810210662b" containerID="54ed10d44823e3a86c29639af52022900c5ad5e88201a263375aa69610464b3f" exitCode=0 Oct 30 13:48:50.404282 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.404153 4982 generic.go:356] "Generic (PLEG): container finished" podID="0dda4c8a-827d-428f-a6a2-5e810210662b" containerID="2fdc6ec6c6a17d45c7745fd442c8f3c89921da01a9c4577bc4d57b92d1ba5194" exitCode=0 Oct 30 13:48:50.415376 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.415351 4982 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 13:48:50.415532 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.415516 4982 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 13:48:50.418072 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.418056 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ntlqq_dc967875-1546-40bb-8376-fde09416c204/console-operator/1.log" Oct 30 13:48:50.418171 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.418077 4982 generic.go:356] "Generic (PLEG): container finished" podID="dc967875-1546-40bb-8376-fde09416c204" containerID="08dafe630716a5a619c9abe02a35bcce9e3b7cb581311f535ccb7aff3f8803cb" exitCode=255 Oct 30 13:48:50.421024 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.420972 4982 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 13:48:50.434746 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.434721 4982 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 13:48:50.434965 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.434944 4982 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 13:48:50.436445 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.436423 4982 generic.go:356] "Generic (PLEG): container finished" podID="18c4e529019bfc1529902a8bff559ebf" containerID="9b348ccc05a60fc27fd2d38cb5b50d635360ecbf5b23aef06c8e01b4cf5f3a98" exitCode=0 Oct 30 13:48:50.444621 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.444602 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhxjf_aca47fe8-7068-4743-b6ab-283ee0874f66/ovn-acl-logging/0.log" Oct 30 13:48:50.444999 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.444978 4982 generic.go:356] "Generic (PLEG): container finished" podID="aca47fe8-7068-4743-b6ab-283ee0874f66" containerID="8ae5eba6d805209a9813e5b8cb249e019de8c8ff2a821150567b953c36724d38" exitCode=1 Oct 30 13:48:50.457862 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.457845 4982 manager.go:324] Recovery completed Oct 30 13:48:50.479154 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.479129 4982 cpu_manager.go:226] "Starting" policy="none" Oct 30 13:48:50.479154 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.479142 4982 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 13:48:50.479247 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.479162 4982 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 13:48:50.479297 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.479282 4982 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 13:48:50.479327 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.479299 4982 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 13:48:50.479327 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.479320 4982 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 13:48:50.479393 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.479329 4982 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 13:48:50.479393 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.479337 4982 policy_none.go:50] "Start" Oct 30 13:48:50.479393 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.479348 4982 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 13:48:50.479393 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.479358 4982 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 13:48:50.479518 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.479508 4982 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 13:48:50.479547 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.479519 4982 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 13:48:50.479547 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.479524 4982 policy_none.go:44] "Start" Oct 30 13:48:50.483747 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.483723 4982 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 13:48:50.486623 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.486609 4982 manager.go:341] "Starting Device Plugin manager" Oct 30 13:48:50.487086 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:48:50.486658 4982 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 13:48:50.487159 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.487090 4982 server.go:87] "Starting device plugin registration server" Oct 30 13:48:50.487366 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.487352 4982 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 13:48:50.487442 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.487366 4982 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 13:48:50.487492 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.487457 4982 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 13:48:50.487596 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.487559 4982 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 13:48:50.487680 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.487604 4982 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 13:48:50.487680 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.487611 4982 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 13:48:50.487782 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.487681 4982 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T13:48:50.487594505Z","UUID":"befdb872-10cd-45f5-9a2b-b6f71e9e3b70","Handler":null,"Name":"","Endpoint":""} Oct 30 13:48:50.488919 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.488900 4982 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 13:48:50.489016 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:48:50.488914 4982 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 13:48:50.489016 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.488924 4982 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 13:48:50.491244 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.491206 4982 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-105.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-105.ec2.internal"] Oct 30 13:48:50.491932 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.491893 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-105.ec2.internal" event={"ID":"7f6d989ed16482ea69070d9acfe2e5f0","Type":"ContainerStarted","Data":"373a818ad40af0f606177e9696274dc40302e998ad58f78f704925d34417e144"} Oct 30 13:48:50.492017 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.491940 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-105.ec2.internal" event={"ID":"7f6d989ed16482ea69070d9acfe2e5f0","Type":"ContainerStarted","Data":"5e13fec44995927b6b7363560b232626b27499eec9ba59e9958238bd861f3dd1"} Oct 30 13:48:50.492017 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.491998 4982 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-105.ec2.internal" Oct 30 13:48:50.492130 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.492016 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-105.ec2.internal" event={"ID":"18c4e529019bfc1529902a8bff559ebf","Type":"ContainerStarted","Data":"9685a3b604fe44912c985b9919798a50746a4f2d5cfd957adb49525a10338ff4"} Oct 30 13:48:50.492130 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.492035 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-105.ec2.internal" event={"ID":"18c4e529019bfc1529902a8bff559ebf","Type":"ContainerDied","Data":"9b348ccc05a60fc27fd2d38cb5b50d635360ecbf5b23aef06c8e01b4cf5f3a98"} Oct 30 13:48:50.492130 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.492046 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-105.ec2.internal" event={"ID":"18c4e529019bfc1529902a8bff559ebf","Type":"ContainerStarted","Data":"20a900a2eab78697079ac3639d0fcc4d077d0e5dcc9fd358b16c4b1215400dc0"} Oct 30 13:48:50.492130 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.492125 4982 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-105.ec2.internal" Oct 30 13:48:50.518916 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.518896 4982 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 13:48:50.519008 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:48:50.518933 4982 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-140-105.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-105.ec2.internal" Oct 30 13:48:50.520063 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.520052 4982 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 13:48:50.520117 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:48:50.520099 4982 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-140-105.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-105.ec2.internal" Oct 30 13:48:50.587678 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.587661 4982 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 13:48:50.588863 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.588849 4982 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-105.ec2.internal" event="NodeHasSufficientMemory" Oct 30 13:48:50.588925 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.588872 4982 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-105.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 13:48:50.588925 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.588882 4982 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-105.ec2.internal" event="NodeHasSufficientPID" Oct 30 13:48:50.588925 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.588914 4982 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-105.ec2.internal" Oct 30 13:48:50.616337 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.616303 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/18c4e529019bfc1529902a8bff559ebf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-105.ec2.internal\" (UID: \"18c4e529019bfc1529902a8bff559ebf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-105.ec2.internal" Oct 30 13:48:50.616337 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.616338 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/18c4e529019bfc1529902a8bff559ebf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-105.ec2.internal\" (UID: \"18c4e529019bfc1529902a8bff559ebf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-105.ec2.internal" Oct 30 13:48:50.616479 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.616355 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7f6d989ed16482ea69070d9acfe2e5f0-config\") pod \"kube-apiserver-proxy-ip-10-0-140-105.ec2.internal\" (UID: \"7f6d989ed16482ea69070d9acfe2e5f0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-105.ec2.internal" Oct 30 13:48:50.633924 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.633907 4982 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-140-105.ec2.internal" Oct 30 13:48:50.633968 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.633950 4982 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-105.ec2.internal" Oct 30 13:48:50.716992 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.716944 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/18c4e529019bfc1529902a8bff559ebf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-105.ec2.internal\" (UID: \"18c4e529019bfc1529902a8bff559ebf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-105.ec2.internal" Oct 30 13:48:50.716992 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.716970 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/18c4e529019bfc1529902a8bff559ebf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-105.ec2.internal\" (UID: \"18c4e529019bfc1529902a8bff559ebf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-105.ec2.internal" Oct 30 13:48:50.716992 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.716985 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7f6d989ed16482ea69070d9acfe2e5f0-config\") pod \"kube-apiserver-proxy-ip-10-0-140-105.ec2.internal\" (UID: \"7f6d989ed16482ea69070d9acfe2e5f0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-105.ec2.internal" Oct 30 13:48:50.717154 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.717028 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7f6d989ed16482ea69070d9acfe2e5f0-config\") pod \"kube-apiserver-proxy-ip-10-0-140-105.ec2.internal\" (UID: \"7f6d989ed16482ea69070d9acfe2e5f0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-105.ec2.internal" Oct 30 13:48:50.717154 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.717037 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/18c4e529019bfc1529902a8bff559ebf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-105.ec2.internal\" (UID: \"18c4e529019bfc1529902a8bff559ebf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-105.ec2.internal" Oct 30 13:48:50.717154 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:50.717067 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/18c4e529019bfc1529902a8bff559ebf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-105.ec2.internal\" (UID: \"18c4e529019bfc1529902a8bff559ebf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-105.ec2.internal" Oct 30 13:48:51.348416 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.348395 4982 apiserver.go:52] "Watching apiserver" Oct 30 13:48:51.407324 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.407304 4982 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 13:48:51.408037 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.408016 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-bhw8w","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-105.ec2.internal","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2nsk6","openshift-image-registry/image-registry-6bc4d9765d-dp7xb","openshift-dns/dns-default-7lx4t","openshift-network-operator/iptables-alerter-qf6mp","kube-system/global-pull-secret-syncer-bdmgn","kube-system/kas-connection-checker-685895c7c9-4h2sq","openshift-kube-storage-version-migrator/migrator-66df5d496b-whnh9","openshift-dns/node-resolver-bjswt","openshift-image-registry/image-registry-87f7c8d8c-mdvsm","kube-system/konnectivity-agent-b622r","openshift-insights/insights-operator-7c4986d449-kcgbg","openshift-ovn-kubernetes/ovnkube-node-bhxjf","openshift-kube-storage-version-migrator/migrator-66df5d496b-v2x2w","kube-system/kas-connection-checker-685895c7c9-smvmg","openshift-network-console/networking-console-plugin-86c478bb88-2bnbl","openshift-insights/insights-runtime-extractor-87kd9","openshift-multus/network-metrics-daemon-pfg7x","openshift-network-diagnostics/network-check-target-dx94q","openshift-cluster-node-tuning-operator/tuned-tqst8","openshift-ingress/router-default-659879fb99-pjvcq","openshift-multus/multus-rvt5b","openshift-console-operator/console-operator-7cd56b8dbc-ntlqq","openshift-image-registry/node-ca-llx79","openshift-monitoring/cluster-monitoring-operator-884576c57-9r9zr","openshift-service-ca-operator/service-ca-operator-544d749c8c-tn69c","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qstzd","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9bxl","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-brd96","openshift-multus/multus-additional-cni-plugins-94cdn","openshift-ingress-canary/ingress-canary-xbkzq","kube-system/kas-connection-checker-685895c7c9-plvgd","kube-system/kube-apiserver-proxy-ip-10-0-140-105.ec2.internal"] Oct 30 13:48:51.412367 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.412345 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qstzd" Oct 30 13:48:51.412688 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.412668 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bc4d9765d-dp7xb" Oct 30 13:48:51.413336 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.413317 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9r9zr" Oct 30 13:48:51.414113 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.414097 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-659879fb99-pjvcq" Oct 30 13:48:51.415068 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.415052 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-2bnbl" Oct 30 13:48:51.417111 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.417093 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfg7x" Oct 30 13:48:51.421269 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.421252 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-87f7c8d8c-mdvsm" Oct 30 13:48:51.421706 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.421695 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7lx4t" Oct 30 13:48:51.426884 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.426862 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-87kd9" Oct 30 13:48:51.428167 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.428146 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 13:48:51.428261 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.428153 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 13:48:51.428261 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.428188 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 13:48:51.428261 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.428200 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 13:48:51.428261 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.428156 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 13:48:51.428261 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.428156 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 13:48:51.428726 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.428712 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 13:48:51.428798 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.428739 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-5d5mh\"" Oct 30 13:48:51.428798 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.428754 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-9vpkx\"" Oct 30 13:48:51.435216 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.435201 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 13:48:51.435631 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.435619 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 13:48:51.437771 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.437757 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 13:48:51.438204 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.438189 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 13:48:51.438302 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.438225 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 13:48:51.438302 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.438295 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 13:48:51.438387 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.438335 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 13:48:51.438387 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.438350 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 13:48:51.438387 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.438365 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 13:48:51.438387 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.438383 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 13:48:51.438575 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.438400 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 13:48:51.438575 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.438414 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 13:48:51.438575 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.438415 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 13:48:51.438575 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.438436 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 13:48:51.438575 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.438416 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 13:48:51.438575 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.438452 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 13:48:51.438575 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.438466 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 13:48:51.438575 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.438472 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 13:48:51.438575 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.438459 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-j65x4\"" Oct 30 13:48:51.438933 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.438616 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 13:48:51.438933 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.438698 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-w8k82\"" Oct 30 13:48:51.438933 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.438628 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 13:48:51.438933 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.438727 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 13:48:51.438933 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.438795 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 13:48:51.438933 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.438868 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 13:48:51.438933 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.438923 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 13:48:51.439142 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.438984 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-qtrmb\"" Oct 30 13:48:51.439142 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.439007 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 13:48:51.439142 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.439046 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 13:48:51.439264 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.439251 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 13:48:51.439349 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.439335 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 13:48:51.439400 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.439366 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 13:48:51.439400 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.439370 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 13:48:51.439462 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.439399 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 13:48:51.439462 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.439437 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 13:48:51.439555 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.439543 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 13:48:51.448616 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.448600 4982 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-105.ec2.internal" Oct 30 13:48:51.448699 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.448686 4982 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-105.ec2.internal" Oct 30 13:48:51.448778 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.448761 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 13:48:51.450444 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.450429 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bjswt_e9a035cc-e78b-4bdb-b0c6-ef13bc07249a/dns-node-resolver/0.log" Oct 30 13:48:51.451513 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.451494 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 13:48:51.452081 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.452064 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 13:48:51.452081 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.452070 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-wstqj\"" Oct 30 13:48:51.452194 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.452176 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 13:48:51.452231 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.452221 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 13:48:51.452278 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.452230 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 13:48:51.452278 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.452251 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 13:48:51.452440 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.452424 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7j7wh\"" Oct 30 13:48:51.452514 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.452499 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 13:48:51.452702 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.452688 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 13:48:51.453009 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.452995 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 13:48:51.453079 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.453065 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 13:48:51.453162 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.453151 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-b7cls\"" Oct 30 13:48:51.453193 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.453156 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 13:48:51.453238 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.453190 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 13:48:51.453340 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.453327 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-shzzf\"" Oct 30 13:48:51.453385 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.453367 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 13:48:51.453385 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.453373 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-d6wsl\"" Oct 30 13:48:51.453385 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.453380 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-4xx27\"" Oct 30 13:48:51.453514 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.453430 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jc894\"" Oct 30 13:48:51.453514 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.453447 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-8strq\"" Oct 30 13:48:51.453680 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.453530 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 13:48:51.453680 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.453546 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 13:48:51.453680 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.453555 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 13:48:51.453680 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.453560 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 13:48:51.453680 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.453602 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-smj8j\"" Oct 30 13:48:51.453680 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.453629 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-s5pv6\"" Oct 30 13:48:51.453680 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.453667 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-697m8\"" Oct 30 13:48:51.454023 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.453742 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 13:48:51.454023 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.453762 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 13:48:51.454023 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.453766 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-w8shz\"" Oct 30 13:48:51.454023 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.453862 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-k4gxw\"" Oct 30 13:48:51.454023 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.453894 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5r2q6\"" Oct 30 13:48:51.454023 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.453912 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-w6ztc\"" Oct 30 13:48:51.454023 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.453937 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 13:48:51.454023 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.453983 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 13:48:51.454023 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.454012 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 13:48:51.454406 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.454049 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-t9cdd\"" Oct 30 13:48:51.454406 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.454057 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-gw59s\"" Oct 30 13:48:51.454406 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.454088 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-7kqhp\"" Oct 30 13:48:51.454406 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.454126 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 13:48:51.454406 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.454147 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2dbfk\"" Oct 30 13:48:51.454406 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.454155 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 13:48:51.454406 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.454164 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 13:48:51.454406 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.454226 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 13:48:51.454406 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.454241 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 13:48:51.454406 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.454245 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-mxshj\"" Oct 30 13:48:51.454406 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.454276 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-snt8q\"" Oct 30 13:48:51.454406 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.454248 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 13:48:51.454406 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.454228 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 13:48:51.454406 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.454299 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 13:48:51.454406 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.454308 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 13:48:51.454406 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.454323 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 13:48:51.454406 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.454358 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 13:48:51.454406 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.454357 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xn5x9\"" Oct 30 13:48:51.454406 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.454391 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 13:48:51.454939 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.454459 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 13:48:51.455589 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.455564 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 13:48:51.455657 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.455592 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 13:48:51.455657 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.455577 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 13:48:51.455657 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.455602 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 13:48:51.456081 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.456062 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 13:48:51.456145 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.456070 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 13:48:51.460407 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.460391 4982 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 13:48:51.474678 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.474655 4982 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 13:48:51.474745 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:48:51.474694 4982 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-140-105.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-105.ec2.internal" Oct 30 13:48:51.475191 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.475179 4982 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 13:48:51.475241 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:48:51.475211 4982 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-140-105.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-105.ec2.internal" Oct 30 13:48:51.506585 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.506557 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:51.519554 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.519522 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-node-log\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:51.519554 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.519553 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b5e74da-5460-44ed-a599-b01620b64c92-original-pull-secret\") pod \"global-pull-secret-syncer-bdmgn\" (UID: \"6b5e74da-5460-44ed-a599-b01620b64c92\") " pod="kube-system/global-pull-secret-syncer-bdmgn" Oct 30 13:48:51.519714 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.519662 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5qnd\" (UniqueName: \"kubernetes.io/projected/b0b2d906-7995-4769-8222-fa8ba2a30f3d-kube-api-access-t5qnd\") pod \"cluster-monitoring-operator-884576c57-9r9zr\" (UID: \"b0b2d906-7995-4769-8222-fa8ba2a30f3d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9r9zr" Oct 30 13:48:51.519775 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.519718 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0dda4c8a-827d-428f-a6a2-5e810210662b-system-cni-dir\") pod \"multus-additional-cni-plugins-94cdn\" (UID: \"0dda4c8a-827d-428f-a6a2-5e810210662b\") " pod="openshift-multus/multus-additional-cni-plugins-94cdn" Oct 30 13:48:51.519775 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.519750 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n98n9\" (UniqueName: \"kubernetes.io/projected/dc967875-1546-40bb-8376-fde09416c204-kube-api-access-n98n9\") pod \"console-operator-7cd56b8dbc-ntlqq\" (UID: \"dc967875-1546-40bb-8376-fde09416c204\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ntlqq" Oct 30 13:48:51.519880 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.519777 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-run-systemd\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:51.519880 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.519797 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/06f40dde-8b71-41e0-9acb-cede4f1d4347-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h9bxl\" (UID: \"06f40dde-8b71-41e0-9acb-cede4f1d4347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9bxl" Oct 30 13:48:51.519880 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.519813 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn2f4\" (UniqueName: \"kubernetes.io/projected/da694c74-3bce-4d95-bd30-12a1891ad5e6-kube-api-access-bn2f4\") pod \"network-metrics-daemon-pfg7x\" (UID: \"da694c74-3bce-4d95-bd30-12a1891ad5e6\") " pod="openshift-multus/network-metrics-daemon-pfg7x" Oct 30 13:48:51.519880 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.519837 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e16ea4af-ce12-442b-91df-092457a7e192-image-registry-private-configuration\") pod \"image-registry-87f7c8d8c-mdvsm\" (UID: \"e16ea4af-ce12-442b-91df-092457a7e192\") " pod="openshift-image-registry/image-registry-87f7c8d8c-mdvsm" Oct 30 13:48:51.519880 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.519869 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b5e74da-5460-44ed-a599-b01620b64c92-original-pull-secret\") pod \"global-pull-secret-syncer-bdmgn\" (UID: \"6b5e74da-5460-44ed-a599-b01620b64c92\") " pod="kube-system/global-pull-secret-syncer-bdmgn" Oct 30 13:48:51.520162 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.519886 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-sys\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:51.520162 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.519913 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tzg4\" (UniqueName: \"kubernetes.io/projected/8304f8da-d4a3-45c1-bfe6-f5d08bed17d7-kube-api-access-4tzg4\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2nsk6\" (UID: \"8304f8da-d4a3-45c1-bfe6-f5d08bed17d7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2nsk6" Oct 30 13:48:51.520162 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.520049 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/083c70c2-0d5e-4876-9e88-10c6efbb681c-config-volume\") pod \"dns-default-7lx4t\" (UID: \"083c70c2-0d5e-4876-9e88-10c6efbb681c\") " pod="openshift-dns/dns-default-7lx4t" Oct 30 13:48:51.520162 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.520084 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f707b282-e507-475d-a82a-923c9bd1ff01-installation-pull-secrets\") pod \"image-registry-6bc4d9765d-dp7xb\" (UID: \"f707b282-e507-475d-a82a-923c9bd1ff01\") " pod="openshift-image-registry/image-registry-6bc4d9765d-dp7xb" Oct 30 13:48:51.520162 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.520111 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6d73c6c3-044d-4b88-a105-72bd5e4491cd-default-certificate\") pod \"router-default-659879fb99-pjvcq\" (UID: \"6d73c6c3-044d-4b88-a105-72bd5e4491cd\") " pod="openshift-ingress/router-default-659879fb99-pjvcq" Oct 30 13:48:51.520162 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.520139 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da694c74-3bce-4d95-bd30-12a1891ad5e6-metrics-certs\") pod \"network-metrics-daemon-pfg7x\" (UID: \"da694c74-3bce-4d95-bd30-12a1891ad5e6\") " pod="openshift-multus/network-metrics-daemon-pfg7x" Oct 30 13:48:51.520442 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.520164 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-run-openvswitch\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:51.520442 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.520187 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-host-run-ovn-kubernetes\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:51.520442 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.520214 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kdz4\" (UniqueName: \"kubernetes.io/projected/06f40dde-8b71-41e0-9acb-cede4f1d4347-kube-api-access-7kdz4\") pod \"aws-ebs-csi-driver-node-h9bxl\" (UID: \"06f40dde-8b71-41e0-9acb-cede4f1d4347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9bxl" Oct 30 13:48:51.520442 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.520251 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e16ea4af-ce12-442b-91df-092457a7e192-image-registry-private-configuration\") pod \"image-registry-87f7c8d8c-mdvsm\" (UID: \"e16ea4af-ce12-442b-91df-092457a7e192\") " pod="openshift-image-registry/image-registry-87f7c8d8c-mdvsm" Oct 30 13:48:51.520442 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.520256 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6d73c6c3-044d-4b88-a105-72bd5e4491cd-stats-auth\") pod \"router-default-659879fb99-pjvcq\" (UID: \"6d73c6c3-044d-4b88-a105-72bd5e4491cd\") " pod="openshift-ingress/router-default-659879fb99-pjvcq" Oct 30 13:48:51.520442 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.520293 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/083c70c2-0d5e-4876-9e88-10c6efbb681c-config-volume\") pod \"dns-default-7lx4t\" (UID: \"083c70c2-0d5e-4876-9e88-10c6efbb681c\") " pod="openshift-dns/dns-default-7lx4t" Oct 30 13:48:51.520442 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.520304 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-host-run-k8s-cni-cncf-io\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:51.520442 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.520332 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f707b282-e507-475d-a82a-923c9bd1ff01-installation-pull-secrets\") pod \"image-registry-6bc4d9765d-dp7xb\" (UID: \"f707b282-e507-475d-a82a-923c9bd1ff01\") " pod="openshift-image-registry/image-registry-6bc4d9765d-dp7xb" Oct 30 13:48:51.520442 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.520340 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltkwj\" (UniqueName: \"kubernetes.io/projected/e9a035cc-e78b-4bdb-b0c6-ef13bc07249a-kube-api-access-ltkwj\") pod \"node-resolver-bjswt\" (UID: \"e9a035cc-e78b-4bdb-b0c6-ef13bc07249a\") " pod="openshift-dns/node-resolver-bjswt" Oct 30 13:48:51.520442 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.520372 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0dda4c8a-827d-428f-a6a2-5e810210662b-cnibin\") pod \"multus-additional-cni-plugins-94cdn\" (UID: \"0dda4c8a-827d-428f-a6a2-5e810210662b\") " pod="openshift-multus/multus-additional-cni-plugins-94cdn" Oct 30 13:48:51.520442 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.520388 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6d73c6c3-044d-4b88-a105-72bd5e4491cd-default-certificate\") pod \"router-default-659879fb99-pjvcq\" (UID: \"6d73c6c3-044d-4b88-a105-72bd5e4491cd\") " pod="openshift-ingress/router-default-659879fb99-pjvcq" Oct 30 13:48:51.520442 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.520394 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8309f073-dcb0-4971-9eb1-8a9ee0eb68c6-nginx-conf\") pod \"networking-console-plugin-86c478bb88-2bnbl\" (UID: \"8309f073-dcb0-4971-9eb1-8a9ee0eb68c6\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2bnbl" Oct 30 13:48:51.520442 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.520424 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dc967875-1546-40bb-8376-fde09416c204-serving-cert\") pod \"console-operator-7cd56b8dbc-ntlqq\" (UID: \"dc967875-1546-40bb-8376-fde09416c204\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ntlqq" Oct 30 13:48:51.521025 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.520469 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a4bd1bb2-25ae-423c-ba05-08eb742d340e-serving-cert\") pod \"insights-operator-7c4986d449-kcgbg\" (UID: \"a4bd1bb2-25ae-423c-ba05-08eb742d340e\") " pod="openshift-insights/insights-operator-7c4986d449-kcgbg" Oct 30 13:48:51.521025 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.520525 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdt2n\" (UniqueName: \"kubernetes.io/projected/d38c246c-86f8-4b57-89bd-db4cd0d3f6d1-kube-api-access-zdt2n\") pod \"iptables-alerter-qf6mp\" (UID: \"d38c246c-86f8-4b57-89bd-db4cd0d3f6d1\") " pod="openshift-network-operator/iptables-alerter-qf6mp" Oct 30 13:48:51.521025 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.520528 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6d73c6c3-044d-4b88-a105-72bd5e4491cd-stats-auth\") pod \"router-default-659879fb99-pjvcq\" (UID: \"6d73c6c3-044d-4b88-a105-72bd5e4491cd\") " pod="openshift-ingress/router-default-659879fb99-pjvcq" Oct 30 13:48:51.521025 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.520546 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/083c70c2-0d5e-4876-9e88-10c6efbb681c-tmp-dir\") pod \"dns-default-7lx4t\" (UID: \"083c70c2-0d5e-4876-9e88-10c6efbb681c\") " pod="openshift-dns/dns-default-7lx4t" Oct 30 13:48:51.521025 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.520564 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f707b282-e507-475d-a82a-923c9bd1ff01-ca-trust-extracted\") pod \"image-registry-6bc4d9765d-dp7xb\" (UID: \"f707b282-e507-475d-a82a-923c9bd1ff01\") " pod="openshift-image-registry/image-registry-6bc4d9765d-dp7xb" Oct 30 13:48:51.521025 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.520570 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8309f073-dcb0-4971-9eb1-8a9ee0eb68c6-nginx-conf\") pod \"networking-console-plugin-86c478bb88-2bnbl\" (UID: \"8309f073-dcb0-4971-9eb1-8a9ee0eb68c6\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2bnbl" Oct 30 13:48:51.521025 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.520583 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/06f40dde-8b71-41e0-9acb-cede4f1d4347-etc-selinux\") pod \"aws-ebs-csi-driver-node-h9bxl\" (UID: \"06f40dde-8b71-41e0-9acb-cede4f1d4347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9bxl" Oct 30 13:48:51.521025 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.520625 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dc967875-1546-40bb-8376-fde09416c204-serving-cert\") pod \"console-operator-7cd56b8dbc-ntlqq\" (UID: \"dc967875-1546-40bb-8376-fde09416c204\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ntlqq" Oct 30 13:48:51.521025 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.520630 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a4bd1bb2-25ae-423c-ba05-08eb742d340e-serving-cert\") pod \"insights-operator-7c4986d449-kcgbg\" (UID: \"a4bd1bb2-25ae-423c-ba05-08eb742d340e\") " pod="openshift-insights/insights-operator-7c4986d449-kcgbg" Oct 30 13:48:51.521025 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.520668 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f707b282-e507-475d-a82a-923c9bd1ff01-ca-trust-extracted\") pod \"image-registry-6bc4d9765d-dp7xb\" (UID: \"f707b282-e507-475d-a82a-923c9bd1ff01\") " pod="openshift-image-registry/image-registry-6bc4d9765d-dp7xb" Oct 30 13:48:51.521025 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.520656 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwgrr\" (UniqueName: \"kubernetes.io/projected/dad9381c-aebb-448d-80b1-e2b65dd8ac80-kube-api-access-rwgrr\") pod \"kas-connection-checker-685895c7c9-4h2sq\" (UID: \"dad9381c-aebb-448d-80b1-e2b65dd8ac80\") " pod="kube-system/kas-connection-checker-685895c7c9-4h2sq" Oct 30 13:48:51.521025 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.520681 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/083c70c2-0d5e-4876-9e88-10c6efbb681c-tmp-dir\") pod \"dns-default-7lx4t\" (UID: \"083c70c2-0d5e-4876-9e88-10c6efbb681c\") " pod="openshift-dns/dns-default-7lx4t" Oct 30 13:48:51.521025 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.520705 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0dda4c8a-827d-428f-a6a2-5e810210662b-cni-binary-copy\") pod \"multus-additional-cni-plugins-94cdn\" (UID: \"0dda4c8a-827d-428f-a6a2-5e810210662b\") " pod="openshift-multus/multus-additional-cni-plugins-94cdn" Oct 30 13:48:51.521025 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.520778 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0dda4c8a-827d-428f-a6a2-5e810210662b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-94cdn\" (UID: \"0dda4c8a-827d-428f-a6a2-5e810210662b\") " pod="openshift-multus/multus-additional-cni-plugins-94cdn" Oct 30 13:48:51.521025 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.520886 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69bm9\" (UniqueName: \"kubernetes.io/projected/6ac8569a-95ae-421e-aee0-edd189e18d11-kube-api-access-69bm9\") pod \"ingress-canary-xbkzq\" (UID: \"6ac8569a-95ae-421e-aee0-edd189e18d11\") " pod="openshift-ingress-canary/ingress-canary-xbkzq" Oct 30 13:48:51.521025 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.520932 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ad85cf2a-6c44-4e85-8677-8b07cb9f78a4-agent-certs\") pod \"konnectivity-agent-b622r\" (UID: \"ad85cf2a-6c44-4e85-8677-8b07cb9f78a4\") " pod="kube-system/konnectivity-agent-b622r" Oct 30 13:48:51.521025 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.520948 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0dda4c8a-827d-428f-a6a2-5e810210662b-cni-binary-copy\") pod \"multus-additional-cni-plugins-94cdn\" (UID: \"0dda4c8a-827d-428f-a6a2-5e810210662b\") " pod="openshift-multus/multus-additional-cni-plugins-94cdn" Oct 30 13:48:51.521697 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.521039 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8309f073-dcb0-4971-9eb1-8a9ee0eb68c6-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2bnbl\" (UID: \"8309f073-dcb0-4971-9eb1-8a9ee0eb68c6\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2bnbl" Oct 30 13:48:51.521697 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.521068 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8304f8da-d4a3-45c1-bfe6-f5d08bed17d7-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2nsk6\" (UID: \"8304f8da-d4a3-45c1-bfe6-f5d08bed17d7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2nsk6" Oct 30 13:48:51.521697 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.521088 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a4bd1bb2-25ae-423c-ba05-08eb742d340e-snapshots\") pod \"insights-operator-7c4986d449-kcgbg\" (UID: \"a4bd1bb2-25ae-423c-ba05-08eb742d340e\") " pod="openshift-insights/insights-operator-7c4986d449-kcgbg" Oct 30 13:48:51.521697 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.521114 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-host-kubelet\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:51.521697 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.521132 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-host-run-netns\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:51.521697 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.521147 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-host-cni-bin\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:51.521697 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.521158 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a4bd1bb2-25ae-423c-ba05-08eb742d340e-snapshots\") pod \"insights-operator-7c4986d449-kcgbg\" (UID: \"a4bd1bb2-25ae-423c-ba05-08eb742d340e\") " pod="openshift-insights/insights-operator-7c4986d449-kcgbg" Oct 30 13:48:51.521697 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.521167 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:51.521697 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.521214 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f707b282-e507-475d-a82a-923c9bd1ff01-bound-sa-token\") pod \"image-registry-6bc4d9765d-dp7xb\" (UID: \"f707b282-e507-475d-a82a-923c9bd1ff01\") " pod="openshift-image-registry/image-registry-6bc4d9765d-dp7xb" Oct 30 13:48:51.521697 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.521246 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e16ea4af-ce12-442b-91df-092457a7e192-installation-pull-secrets\") pod \"image-registry-87f7c8d8c-mdvsm\" (UID: \"e16ea4af-ce12-442b-91df-092457a7e192\") " pod="openshift-image-registry/image-registry-87f7c8d8c-mdvsm" Oct 30 13:48:51.521697 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.521249 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ad85cf2a-6c44-4e85-8677-8b07cb9f78a4-agent-certs\") pod \"konnectivity-agent-b622r\" (UID: \"ad85cf2a-6c44-4e85-8677-8b07cb9f78a4\") " pod="kube-system/konnectivity-agent-b622r" Oct 30 13:48:51.521697 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.521282 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-etc-kubernetes\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:51.521697 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.521297 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8304f8da-d4a3-45c1-bfe6-f5d08bed17d7-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2nsk6\" (UID: \"8304f8da-d4a3-45c1-bfe6-f5d08bed17d7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2nsk6" Oct 30 13:48:51.521697 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.521331 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dc967875-1546-40bb-8376-fde09416c204-config\") pod \"console-operator-7cd56b8dbc-ntlqq\" (UID: \"dc967875-1546-40bb-8376-fde09416c204\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ntlqq" Oct 30 13:48:51.521697 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.521427 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-host-slash\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:51.521697 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.521447 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/aca47fe8-7068-4743-b6ab-283ee0874f66-ovnkube-script-lib\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:51.521697 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.521489 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm9mr\" (UniqueName: \"kubernetes.io/projected/f707b282-e507-475d-a82a-923c9bd1ff01-kube-api-access-vm9mr\") pod \"image-registry-6bc4d9765d-dp7xb\" (UID: \"f707b282-e507-475d-a82a-923c9bd1ff01\") " pod="openshift-image-registry/image-registry-6bc4d9765d-dp7xb" Oct 30 13:48:51.521697 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.521515 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e16ea4af-ce12-442b-91df-092457a7e192-installation-pull-secrets\") pod \"image-registry-87f7c8d8c-mdvsm\" (UID: \"e16ea4af-ce12-442b-91df-092457a7e192\") " pod="openshift-image-registry/image-registry-87f7c8d8c-mdvsm" Oct 30 13:48:51.521697 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.521516 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb8hg\" (UniqueName: \"kubernetes.io/projected/bf73153f-1421-44d6-abae-b76c3d90235e-kube-api-access-mb8hg\") pod \"network-check-target-dx94q\" (UID: \"bf73153f-1421-44d6-abae-b76c3d90235e\") " pod="openshift-network-diagnostics/network-check-target-dx94q" Oct 30 13:48:51.521697 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.521577 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ade75be2-e3bb-4b50-bed0-c9c257739707-serving-cert\") pod \"service-ca-operator-544d749c8c-tn69c\" (UID: \"ade75be2-e3bb-4b50-bed0-c9c257739707\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tn69c" Oct 30 13:48:51.521697 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.521596 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e16ea4af-ce12-442b-91df-092457a7e192-bound-sa-token\") pod \"image-registry-87f7c8d8c-mdvsm\" (UID: \"e16ea4af-ce12-442b-91df-092457a7e192\") " pod="openshift-image-registry/image-registry-87f7c8d8c-mdvsm" Oct 30 13:48:51.521697 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.521610 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp4w5\" (UniqueName: \"kubernetes.io/projected/a85af541-54ed-4d24-b4c5-d69244396157-kube-api-access-gp4w5\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:51.521697 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.521620 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dc967875-1546-40bb-8376-fde09416c204-config\") pod \"console-operator-7cd56b8dbc-ntlqq\" (UID: \"dc967875-1546-40bb-8376-fde09416c204\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ntlqq" Oct 30 13:48:51.521697 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.521626 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk8lh\" (UniqueName: \"kubernetes.io/projected/806dd609-9a2a-49c4-a656-e50437457fa9-kube-api-access-gk8lh\") pod \"network-check-source-6f78d59f99-bhw8w\" (UID: \"806dd609-9a2a-49c4-a656-e50437457fa9\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bhw8w" Oct 30 13:48:51.521697 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.521668 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/aca47fe8-7068-4743-b6ab-283ee0874f66-ovnkube-script-lib\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:51.522549 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.521773 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0dda4c8a-827d-428f-a6a2-5e810210662b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-94cdn\" (UID: \"0dda4c8a-827d-428f-a6a2-5e810210662b\") " pod="openshift-multus/multus-additional-cni-plugins-94cdn" Oct 30 13:48:51.522549 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.521789 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ade75be2-e3bb-4b50-bed0-c9c257739707-serving-cert\") pod \"service-ca-operator-544d749c8c-tn69c\" (UID: \"ade75be2-e3bb-4b50-bed0-c9c257739707\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tn69c" Oct 30 13:48:51.522549 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.521803 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b0b2d906-7995-4769-8222-fa8ba2a30f3d-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-9r9zr\" (UID: \"b0b2d906-7995-4769-8222-fa8ba2a30f3d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9r9zr" Oct 30 13:48:51.522549 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.521829 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ad85cf2a-6c44-4e85-8677-8b07cb9f78a4-konnectivity-ca\") pod \"konnectivity-agent-b622r\" (UID: \"ad85cf2a-6c44-4e85-8677-8b07cb9f78a4\") " pod="kube-system/konnectivity-agent-b622r" Oct 30 13:48:51.522549 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.521857 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-system-cni-dir\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:51.522549 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.521883 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-var-lib-openvswitch\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:51.522549 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.521906 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-run-ovn\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:51.522549 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.521937 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0dda4c8a-827d-428f-a6a2-5e810210662b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-94cdn\" (UID: \"0dda4c8a-827d-428f-a6a2-5e810210662b\") " pod="openshift-multus/multus-additional-cni-plugins-94cdn" Oct 30 13:48:51.522549 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.522005 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/083c70c2-0d5e-4876-9e88-10c6efbb681c-metrics-tls\") pod \"dns-default-7lx4t\" (UID: \"083c70c2-0d5e-4876-9e88-10c6efbb681c\") " pod="openshift-dns/dns-default-7lx4t" Oct 30 13:48:51.522549 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.522036 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b0b2d906-7995-4769-8222-fa8ba2a30f3d-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-9r9zr\" (UID: \"b0b2d906-7995-4769-8222-fa8ba2a30f3d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9r9zr" Oct 30 13:48:51.522549 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.522049 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfgzb\" (UniqueName: \"kubernetes.io/projected/6d73c6c3-044d-4b88-a105-72bd5e4491cd-kube-api-access-tfgzb\") pod \"router-default-659879fb99-pjvcq\" (UID: \"6d73c6c3-044d-4b88-a105-72bd5e4491cd\") " pod="openshift-ingress/router-default-659879fb99-pjvcq" Oct 30 13:48:51.522549 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.522060 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ad85cf2a-6c44-4e85-8677-8b07cb9f78a4-konnectivity-ca\") pod \"konnectivity-agent-b622r\" (UID: \"ad85cf2a-6c44-4e85-8677-8b07cb9f78a4\") " pod="kube-system/konnectivity-agent-b622r" Oct 30 13:48:51.522549 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.522161 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e16ea4af-ce12-442b-91df-092457a7e192-registry-certificates\") pod \"image-registry-87f7c8d8c-mdvsm\" (UID: \"e16ea4af-ce12-442b-91df-092457a7e192\") " pod="openshift-image-registry/image-registry-87f7c8d8c-mdvsm" Oct 30 13:48:51.522549 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.522192 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-cnibin\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:51.522549 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.522206 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-hostroot\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:51.522549 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.522236 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-etc-sysctl-d\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:51.522549 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.522320 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e16ea4af-ce12-442b-91df-092457a7e192-registry-certificates\") pod \"image-registry-87f7c8d8c-mdvsm\" (UID: \"e16ea4af-ce12-442b-91df-092457a7e192\") " pod="openshift-image-registry/image-registry-87f7c8d8c-mdvsm" Oct 30 13:48:51.522549 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.522331 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p2xh\" (UniqueName: \"kubernetes.io/projected/083c70c2-0d5e-4876-9e88-10c6efbb681c-kube-api-access-6p2xh\") pod \"dns-default-7lx4t\" (UID: \"083c70c2-0d5e-4876-9e88-10c6efbb681c\") " pod="openshift-dns/dns-default-7lx4t" Oct 30 13:48:51.522549 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.522357 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-tmp\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:51.522549 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.522374 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c06b4b47-6141-4263-94b3-803d9f28bb94-serviceca\") pod \"node-ca-llx79\" (UID: \"c06b4b47-6141-4263-94b3-803d9f28bb94\") " pod="openshift-image-registry/node-ca-llx79" Oct 30 13:48:51.522549 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.522407 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d38c246c-86f8-4b57-89bd-db4cd0d3f6d1-host-slash\") pod \"iptables-alerter-qf6mp\" (UID: \"d38c246c-86f8-4b57-89bd-db4cd0d3f6d1\") " pod="openshift-network-operator/iptables-alerter-qf6mp" Oct 30 13:48:51.522549 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.522418 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-tmp\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:51.522549 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.522498 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6b5e74da-5460-44ed-a599-b01620b64c92-dbus\") pod \"global-pull-secret-syncer-bdmgn\" (UID: \"6b5e74da-5460-44ed-a599-b01620b64c92\") " pod="kube-system/global-pull-secret-syncer-bdmgn" Oct 30 13:48:51.522549 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.522563 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-host-var-lib-kubelet\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:51.523595 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.522572 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c06b4b47-6141-4263-94b3-803d9f28bb94-serviceca\") pod \"node-ca-llx79\" (UID: \"c06b4b47-6141-4263-94b3-803d9f28bb94\") " pod="openshift-image-registry/node-ca-llx79" Oct 30 13:48:51.523595 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.522581 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6b536198-f804-4868-beb9-123229c3ae51-crio-socket\") pod \"insights-runtime-extractor-87kd9\" (UID: \"6b536198-f804-4868-beb9-123229c3ae51\") " pod="openshift-insights/insights-runtime-extractor-87kd9" Oct 30 13:48:51.523595 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.522610 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6b536198-f804-4868-beb9-123229c3ae51-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-87kd9\" (UID: \"6b536198-f804-4868-beb9-123229c3ae51\") " pod="openshift-insights/insights-runtime-extractor-87kd9" Oct 30 13:48:51.523595 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.522628 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l99vz\" (UniqueName: \"kubernetes.io/projected/cb90396e-d4e9-4eb0-888f-64c59d492fd4-kube-api-access-l99vz\") pod \"kas-connection-checker-685895c7c9-smvmg\" (UID: \"cb90396e-d4e9-4eb0-888f-64c59d492fd4\") " pod="kube-system/kas-connection-checker-685895c7c9-smvmg" Oct 30 13:48:51.523595 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.522664 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e16ea4af-ce12-442b-91df-092457a7e192-registry-tls\") pod \"image-registry-87f7c8d8c-mdvsm\" (UID: \"e16ea4af-ce12-442b-91df-092457a7e192\") " pod="openshift-image-registry/image-registry-87f7c8d8c-mdvsm" Oct 30 13:48:51.523595 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.522744 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-etc-modprobe-d\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:51.523595 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.522791 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-var-lib-kubelet\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:51.523595 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.522826 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwd54\" (UniqueName: \"kubernetes.io/projected/a4bd1bb2-25ae-423c-ba05-08eb742d340e-kube-api-access-pwd54\") pod \"insights-operator-7c4986d449-kcgbg\" (UID: \"a4bd1bb2-25ae-423c-ba05-08eb742d340e\") " pod="openshift-insights/insights-operator-7c4986d449-kcgbg" Oct 30 13:48:51.523595 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.522867 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/aca47fe8-7068-4743-b6ab-283ee0874f66-ovn-node-metrics-cert\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:51.523595 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.522909 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx49w\" (UniqueName: \"kubernetes.io/projected/8d9908f6-aefa-4bde-9182-2c199f08a24a-kube-api-access-fx49w\") pod \"volume-data-source-validator-58c9f5f46d-brd96\" (UID: \"8d9908f6-aefa-4bde-9182-2c199f08a24a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-brd96" Oct 30 13:48:51.523595 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.523027 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-etc-kubernetes\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:51.523595 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.523059 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a85af541-54ed-4d24-b4c5-d69244396157-multus-daemon-config\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:51.523595 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.523096 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/aca47fe8-7068-4743-b6ab-283ee0874f66-ovn-node-metrics-cert\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:51.523595 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.523134 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6b536198-f804-4868-beb9-123229c3ae51-data-volume\") pod \"insights-runtime-extractor-87kd9\" (UID: \"6b536198-f804-4868-beb9-123229c3ae51\") " pod="openshift-insights/insights-runtime-extractor-87kd9" Oct 30 13:48:51.523595 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.523177 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6xq2\" (UniqueName: \"kubernetes.io/projected/aca47fe8-7068-4743-b6ab-283ee0874f66-kube-api-access-b6xq2\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:51.523595 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.523240 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-etc-sysconfig\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:51.523595 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.523257 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-host-run-multus-certs\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:51.523595 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.523256 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6b536198-f804-4868-beb9-123229c3ae51-data-volume\") pod \"insights-runtime-extractor-87kd9\" (UID: \"6b536198-f804-4868-beb9-123229c3ae51\") " pod="openshift-insights/insights-runtime-extractor-87kd9" Oct 30 13:48:51.523595 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.523284 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/aca47fe8-7068-4743-b6ab-283ee0874f66-ovnkube-config\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:51.523595 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.523304 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/06f40dde-8b71-41e0-9acb-cede4f1d4347-registration-dir\") pod \"aws-ebs-csi-driver-node-h9bxl\" (UID: \"06f40dde-8b71-41e0-9acb-cede4f1d4347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9bxl" Oct 30 13:48:51.523595 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.523321 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-multus-cni-dir\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:51.523595 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.523348 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-host-var-lib-cni-bin\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:51.523595 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.523374 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a85af541-54ed-4d24-b4c5-d69244396157-multus-daemon-config\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:51.523595 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.523379 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srtk8\" (UniqueName: \"kubernetes.io/projected/7cd2b593-c362-43c2-b0bb-7e5021791214-kube-api-access-srtk8\") pod \"migrator-66df5d496b-whnh9\" (UID: \"7cd2b593-c362-43c2-b0bb-7e5021791214\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-whnh9" Oct 30 13:48:51.523595 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.523416 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r9cl\" (UniqueName: \"kubernetes.io/projected/6b536198-f804-4868-beb9-123229c3ae51-kube-api-access-5r9cl\") pod \"insights-runtime-extractor-87kd9\" (UID: \"6b536198-f804-4868-beb9-123229c3ae51\") " pod="openshift-insights/insights-runtime-extractor-87kd9" Oct 30 13:48:51.523595 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.523435 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/aca47fe8-7068-4743-b6ab-283ee0874f66-env-overrides\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:51.523595 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.523451 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f707b282-e507-475d-a82a-923c9bd1ff01-registry-certificates\") pod \"image-registry-6bc4d9765d-dp7xb\" (UID: \"f707b282-e507-475d-a82a-923c9bd1ff01\") " pod="openshift-image-registry/image-registry-6bc4d9765d-dp7xb" Oct 30 13:48:51.523595 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.523417 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/aca47fe8-7068-4743-b6ab-283ee0874f66-ovnkube-config\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:51.523595 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.523475 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d73c6c3-044d-4b88-a105-72bd5e4491cd-metrics-certs\") pod \"router-default-659879fb99-pjvcq\" (UID: \"6d73c6c3-044d-4b88-a105-72bd5e4491cd\") " pod="openshift-ingress/router-default-659879fb99-pjvcq" Oct 30 13:48:51.523595 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.523499 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c06b4b47-6141-4263-94b3-803d9f28bb94-host\") pod \"node-ca-llx79\" (UID: \"c06b4b47-6141-4263-94b3-803d9f28bb94\") " pod="openshift-image-registry/node-ca-llx79" Oct 30 13:48:51.523595 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.523527 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8304f8da-d4a3-45c1-bfe6-f5d08bed17d7-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2nsk6\" (UID: \"8304f8da-d4a3-45c1-bfe6-f5d08bed17d7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2nsk6" Oct 30 13:48:51.523595 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.523552 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt65k\" (UniqueName: \"kubernetes.io/projected/0dda4c8a-827d-428f-a6a2-5e810210662b-kube-api-access-gt65k\") pod \"multus-additional-cni-plugins-94cdn\" (UID: \"0dda4c8a-827d-428f-a6a2-5e810210662b\") " pod="openshift-multus/multus-additional-cni-plugins-94cdn" Oct 30 13:48:51.523595 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.523581 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f707b282-e507-475d-a82a-923c9bd1ff01-trusted-ca\") pod \"image-registry-6bc4d9765d-dp7xb\" (UID: \"f707b282-e507-475d-a82a-923c9bd1ff01\") " pod="openshift-image-registry/image-registry-6bc4d9765d-dp7xb" Oct 30 13:48:51.523595 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.523598 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/aca47fe8-7068-4743-b6ab-283ee0874f66-env-overrides\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:51.524691 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.523635 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f707b282-e507-475d-a82a-923c9bd1ff01-registry-certificates\") pod \"image-registry-6bc4d9765d-dp7xb\" (UID: \"f707b282-e507-475d-a82a-923c9bd1ff01\") " pod="openshift-image-registry/image-registry-6bc4d9765d-dp7xb" Oct 30 13:48:51.524691 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.523737 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brjls\" (UniqueName: \"kubernetes.io/projected/c06b4b47-6141-4263-94b3-803d9f28bb94-kube-api-access-brjls\") pod \"node-ca-llx79\" (UID: \"c06b4b47-6141-4263-94b3-803d9f28bb94\") " pod="openshift-image-registry/node-ca-llx79" Oct 30 13:48:51.524691 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.523771 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spnhx\" (UniqueName: \"kubernetes.io/projected/81dcad0b-d124-41b8-8b4e-d41049de217a-kube-api-access-spnhx\") pod \"cluster-samples-operator-6ddb6c8b94-qstzd\" (UID: \"81dcad0b-d124-41b8-8b4e-d41049de217a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qstzd" Oct 30 13:48:51.524691 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.523825 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v9rm\" (UniqueName: \"kubernetes.io/projected/e16ea4af-ce12-442b-91df-092457a7e192-kube-api-access-5v9rm\") pod \"image-registry-87f7c8d8c-mdvsm\" (UID: \"e16ea4af-ce12-442b-91df-092457a7e192\") " pod="openshift-image-registry/image-registry-87f7c8d8c-mdvsm" Oct 30 13:48:51.524691 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.523852 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-run\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:51.524691 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.523860 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8304f8da-d4a3-45c1-bfe6-f5d08bed17d7-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2nsk6\" (UID: \"8304f8da-d4a3-45c1-bfe6-f5d08bed17d7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2nsk6" Oct 30 13:48:51.524691 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.523877 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-lib-modules\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:51.524691 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.523930 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f707b282-e507-475d-a82a-923c9bd1ff01-registry-tls\") pod \"image-registry-6bc4d9765d-dp7xb\" (UID: \"f707b282-e507-475d-a82a-923c9bd1ff01\") " pod="openshift-image-registry/image-registry-6bc4d9765d-dp7xb" Oct 30 13:48:51.524691 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.523963 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/06f40dde-8b71-41e0-9acb-cede4f1d4347-device-dir\") pod \"aws-ebs-csi-driver-node-h9bxl\" (UID: \"06f40dde-8b71-41e0-9acb-cede4f1d4347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9bxl" Oct 30 13:48:51.524691 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.523999 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzwmc\" (UniqueName: \"kubernetes.io/projected/ade75be2-e3bb-4b50-bed0-c9c257739707-kube-api-access-tzwmc\") pod \"service-ca-operator-544d749c8c-tn69c\" (UID: \"ade75be2-e3bb-4b50-bed0-c9c257739707\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tn69c" Oct 30 13:48:51.524691 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.524013 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f707b282-e507-475d-a82a-923c9bd1ff01-trusted-ca\") pod \"image-registry-6bc4d9765d-dp7xb\" (UID: \"f707b282-e507-475d-a82a-923c9bd1ff01\") " pod="openshift-image-registry/image-registry-6bc4d9765d-dp7xb" Oct 30 13:48:51.524691 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.524026 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a85af541-54ed-4d24-b4c5-d69244396157-cni-binary-copy\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:51.524691 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.524051 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkhbw\" (UniqueName: \"kubernetes.io/projected/d7a64da6-7b20-4f20-90d7-bc23a71290e5-kube-api-access-zkhbw\") pod \"kas-connection-checker-685895c7c9-plvgd\" (UID: \"d7a64da6-7b20-4f20-90d7-bc23a71290e5\") " pod="kube-system/kas-connection-checker-685895c7c9-plvgd" Oct 30 13:48:51.524691 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.524092 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9a035cc-e78b-4bdb-b0c6-ef13bc07249a-tmp-dir\") pod \"node-resolver-bjswt\" (UID: \"e9a035cc-e78b-4bdb-b0c6-ef13bc07249a\") " pod="openshift-dns/node-resolver-bjswt" Oct 30 13:48:51.524691 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.524113 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/81dcad0b-d124-41b8-8b4e-d41049de217a-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-qstzd\" (UID: \"81dcad0b-d124-41b8-8b4e-d41049de217a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qstzd" Oct 30 13:48:51.524691 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.524131 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b0b2d906-7995-4769-8222-fa8ba2a30f3d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-9r9zr\" (UID: \"b0b2d906-7995-4769-8222-fa8ba2a30f3d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9r9zr" Oct 30 13:48:51.524691 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.524148 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-host-run-netns\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:51.524691 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.524174 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f707b282-e507-475d-a82a-923c9bd1ff01-image-registry-private-configuration\") pod \"image-registry-6bc4d9765d-dp7xb\" (UID: \"f707b282-e507-475d-a82a-923c9bd1ff01\") " pod="openshift-image-registry/image-registry-6bc4d9765d-dp7xb" Oct 30 13:48:51.524691 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.524200 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a85af541-54ed-4d24-b4c5-d69244396157-cni-binary-copy\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:51.524691 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.524214 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6b5e74da-5460-44ed-a599-b01620b64c92-kubelet-config\") pod \"global-pull-secret-syncer-bdmgn\" (UID: \"6b5e74da-5460-44ed-a599-b01620b64c92\") " pod="kube-system/global-pull-secret-syncer-bdmgn" Oct 30 13:48:51.524691 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.524248 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9a035cc-e78b-4bdb-b0c6-ef13bc07249a-tmp-dir\") pod \"node-resolver-bjswt\" (UID: \"e9a035cc-e78b-4bdb-b0c6-ef13bc07249a\") " pod="openshift-dns/node-resolver-bjswt" Oct 30 13:48:51.524691 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.524300 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/81dcad0b-d124-41b8-8b4e-d41049de217a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-qstzd\" (UID: \"81dcad0b-d124-41b8-8b4e-d41049de217a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qstzd" Oct 30 13:48:51.524691 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.524329 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/81dcad0b-d124-41b8-8b4e-d41049de217a-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-qstzd\" (UID: \"81dcad0b-d124-41b8-8b4e-d41049de217a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qstzd" Oct 30 13:48:51.524691 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.524339 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f707b282-e507-475d-a82a-923c9bd1ff01-image-registry-private-configuration\") pod \"image-registry-6bc4d9765d-dp7xb\" (UID: \"f707b282-e507-475d-a82a-923c9bd1ff01\") " pod="openshift-image-registry/image-registry-6bc4d9765d-dp7xb" Oct 30 13:48:51.524691 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.524341 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/06f40dde-8b71-41e0-9acb-cede4f1d4347-sys-fs\") pod \"aws-ebs-csi-driver-node-h9bxl\" (UID: \"06f40dde-8b71-41e0-9acb-cede4f1d4347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9bxl" Oct 30 13:48:51.524691 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.524379 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-etc-systemd\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:51.524691 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.524411 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-multus-conf-dir\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:51.524691 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.524431 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e9a035cc-e78b-4bdb-b0c6-ef13bc07249a-hosts-file\") pod \"node-resolver-bjswt\" (UID: \"e9a035cc-e78b-4bdb-b0c6-ef13bc07249a\") " pod="openshift-dns/node-resolver-bjswt" Oct 30 13:48:51.524691 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.524448 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4bd1bb2-25ae-423c-ba05-08eb742d340e-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-kcgbg\" (UID: \"a4bd1bb2-25ae-423c-ba05-08eb742d340e\") " pod="openshift-insights/insights-operator-7c4986d449-kcgbg" Oct 30 13:48:51.524691 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.524470 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6ac8569a-95ae-421e-aee0-edd189e18d11-cert\") pod \"ingress-canary-xbkzq\" (UID: \"6ac8569a-95ae-421e-aee0-edd189e18d11\") " pod="openshift-ingress-canary/ingress-canary-xbkzq" Oct 30 13:48:51.524691 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.524488 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ade75be2-e3bb-4b50-bed0-c9c257739707-config\") pod \"service-ca-operator-544d749c8c-tn69c\" (UID: \"ade75be2-e3bb-4b50-bed0-c9c257739707\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tn69c" Oct 30 13:48:51.524691 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.524535 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-host-cni-netd\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:51.524691 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.524651 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhdhm\" (UniqueName: \"kubernetes.io/projected/ccafe9ae-44d7-4c31-a6a5-ba8506bc0cea-kube-api-access-xhdhm\") pod \"migrator-66df5d496b-v2x2w\" (UID: \"ccafe9ae-44d7-4c31-a6a5-ba8506bc0cea\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-v2x2w" Oct 30 13:48:51.524691 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.524687 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6ac8569a-95ae-421e-aee0-edd189e18d11-cert\") pod \"ingress-canary-xbkzq\" (UID: \"6ac8569a-95ae-421e-aee0-edd189e18d11\") " pod="openshift-ingress-canary/ingress-canary-xbkzq" Oct 30 13:48:51.524691 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.524700 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/06f40dde-8b71-41e0-9acb-cede4f1d4347-socket-dir\") pod \"aws-ebs-csi-driver-node-h9bxl\" (UID: \"06f40dde-8b71-41e0-9acb-cede4f1d4347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9bxl" Oct 30 13:48:51.524691 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.524705 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ade75be2-e3bb-4b50-bed0-c9c257739707-config\") pod \"service-ca-operator-544d749c8c-tn69c\" (UID: \"ade75be2-e3bb-4b50-bed0-c9c257739707\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tn69c" Oct 30 13:48:51.525800 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.524723 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6b536198-f804-4868-beb9-123229c3ae51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-87kd9\" (UID: \"6b536198-f804-4868-beb9-123229c3ae51\") " pod="openshift-insights/insights-runtime-extractor-87kd9" Oct 30 13:48:51.525800 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.524739 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4bd1bb2-25ae-423c-ba05-08eb742d340e-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-kcgbg\" (UID: \"a4bd1bb2-25ae-423c-ba05-08eb742d340e\") " pod="openshift-insights/insights-operator-7c4986d449-kcgbg" Oct 30 13:48:51.525800 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.524774 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0dda4c8a-827d-428f-a6a2-5e810210662b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-94cdn\" (UID: \"0dda4c8a-827d-428f-a6a2-5e810210662b\") " pod="openshift-multus/multus-additional-cni-plugins-94cdn" Oct 30 13:48:51.525800 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.524793 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4bd1bb2-25ae-423c-ba05-08eb742d340e-tmp\") pod \"insights-operator-7c4986d449-kcgbg\" (UID: \"a4bd1bb2-25ae-423c-ba05-08eb742d340e\") " pod="openshift-insights/insights-operator-7c4986d449-kcgbg" Oct 30 13:48:51.525800 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.524808 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d38c246c-86f8-4b57-89bd-db4cd0d3f6d1-iptables-alerter-script\") pod \"iptables-alerter-qf6mp\" (UID: \"d38c246c-86f8-4b57-89bd-db4cd0d3f6d1\") " pod="openshift-network-operator/iptables-alerter-qf6mp" Oct 30 13:48:51.525800 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.524836 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d73c6c3-044d-4b88-a105-72bd5e4491cd-service-ca-bundle\") pod \"router-default-659879fb99-pjvcq\" (UID: \"6d73c6c3-044d-4b88-a105-72bd5e4491cd\") " pod="openshift-ingress/router-default-659879fb99-pjvcq" Oct 30 13:48:51.525800 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.524862 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e16ea4af-ce12-442b-91df-092457a7e192-ca-trust-extracted\") pod \"image-registry-87f7c8d8c-mdvsm\" (UID: \"e16ea4af-ce12-442b-91df-092457a7e192\") " pod="openshift-image-registry/image-registry-87f7c8d8c-mdvsm" Oct 30 13:48:51.525800 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.524887 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e16ea4af-ce12-442b-91df-092457a7e192-trusted-ca\") pod \"image-registry-87f7c8d8c-mdvsm\" (UID: \"e16ea4af-ce12-442b-91df-092457a7e192\") " pod="openshift-image-registry/image-registry-87f7c8d8c-mdvsm" Oct 30 13:48:51.525800 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.524948 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-host\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:51.525800 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.524970 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e16ea4af-ce12-442b-91df-092457a7e192-ca-trust-extracted\") pod \"image-registry-87f7c8d8c-mdvsm\" (UID: \"e16ea4af-ce12-442b-91df-092457a7e192\") " pod="openshift-image-registry/image-registry-87f7c8d8c-mdvsm" Oct 30 13:48:51.525800 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.524980 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0dda4c8a-827d-428f-a6a2-5e810210662b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-94cdn\" (UID: \"0dda4c8a-827d-428f-a6a2-5e810210662b\") " pod="openshift-multus/multus-additional-cni-plugins-94cdn" Oct 30 13:48:51.525800 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.524888 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4bd1bb2-25ae-423c-ba05-08eb742d340e-tmp\") pod \"insights-operator-7c4986d449-kcgbg\" (UID: \"a4bd1bb2-25ae-423c-ba05-08eb742d340e\") " pod="openshift-insights/insights-operator-7c4986d449-kcgbg" Oct 30 13:48:51.525800 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.524974 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-host-var-lib-cni-multus\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:51.525800 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.525017 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4bd1bb2-25ae-423c-ba05-08eb742d340e-service-ca-bundle\") pod \"insights-operator-7c4986d449-kcgbg\" (UID: \"a4bd1bb2-25ae-423c-ba05-08eb742d340e\") " pod="openshift-insights/insights-operator-7c4986d449-kcgbg" Oct 30 13:48:51.525800 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.525045 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d38c246c-86f8-4b57-89bd-db4cd0d3f6d1-iptables-alerter-script\") pod \"iptables-alerter-qf6mp\" (UID: \"d38c246c-86f8-4b57-89bd-db4cd0d3f6d1\") " pod="openshift-network-operator/iptables-alerter-qf6mp" Oct 30 13:48:51.525800 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.525046 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-systemd-units\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:51.525800 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.525138 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-log-socket\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:51.525800 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.525160 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfktw\" (UniqueName: \"kubernetes.io/projected/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-kube-api-access-xfktw\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:51.525800 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.525172 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e16ea4af-ce12-442b-91df-092457a7e192-trusted-ca\") pod \"image-registry-87f7c8d8c-mdvsm\" (UID: \"e16ea4af-ce12-442b-91df-092457a7e192\") " pod="openshift-image-registry/image-registry-87f7c8d8c-mdvsm" Oct 30 13:48:51.525800 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.525176 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dc967875-1546-40bb-8376-fde09416c204-trusted-ca\") pod \"console-operator-7cd56b8dbc-ntlqq\" (UID: \"dc967875-1546-40bb-8376-fde09416c204\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ntlqq" Oct 30 13:48:51.525800 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.525174 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4bd1bb2-25ae-423c-ba05-08eb742d340e-service-ca-bundle\") pod \"insights-operator-7c4986d449-kcgbg\" (UID: \"a4bd1bb2-25ae-423c-ba05-08eb742d340e\") " pod="openshift-insights/insights-operator-7c4986d449-kcgbg" Oct 30 13:48:51.525800 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.525277 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-etc-sysctl-conf\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:51.525800 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.525310 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-etc-tuned\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:51.525800 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.525350 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-multus-socket-dir-parent\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:51.525800 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.525374 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-etc-openvswitch\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:51.525800 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.525402 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-etc-tuned\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:51.526458 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.525924 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dc967875-1546-40bb-8376-fde09416c204-trusted-ca\") pod \"console-operator-7cd56b8dbc-ntlqq\" (UID: \"dc967875-1546-40bb-8376-fde09416c204\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ntlqq" Oct 30 13:48:51.543476 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.543454 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5qnd\" (UniqueName: \"kubernetes.io/projected/b0b2d906-7995-4769-8222-fa8ba2a30f3d-kube-api-access-t5qnd\") pod \"cluster-monitoring-operator-884576c57-9r9zr\" (UID: \"b0b2d906-7995-4769-8222-fa8ba2a30f3d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9r9zr" Oct 30 13:48:51.543724 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.543708 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tzg4\" (UniqueName: \"kubernetes.io/projected/8304f8da-d4a3-45c1-bfe6-f5d08bed17d7-kube-api-access-4tzg4\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2nsk6\" (UID: \"8304f8da-d4a3-45c1-bfe6-f5d08bed17d7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2nsk6" Oct 30 13:48:51.544193 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.544173 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwgrr\" (UniqueName: \"kubernetes.io/projected/dad9381c-aebb-448d-80b1-e2b65dd8ac80-kube-api-access-rwgrr\") pod \"kas-connection-checker-685895c7c9-4h2sq\" (UID: \"dad9381c-aebb-448d-80b1-e2b65dd8ac80\") " pod="kube-system/kas-connection-checker-685895c7c9-4h2sq" Oct 30 13:48:51.544510 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.544348 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kdz4\" (UniqueName: \"kubernetes.io/projected/06f40dde-8b71-41e0-9acb-cede4f1d4347-kube-api-access-7kdz4\") pod \"aws-ebs-csi-driver-node-h9bxl\" (UID: \"06f40dde-8b71-41e0-9acb-cede4f1d4347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9bxl" Oct 30 13:48:51.544510 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.544355 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdt2n\" (UniqueName: \"kubernetes.io/projected/d38c246c-86f8-4b57-89bd-db4cd0d3f6d1-kube-api-access-zdt2n\") pod \"iptables-alerter-qf6mp\" (UID: \"d38c246c-86f8-4b57-89bd-db4cd0d3f6d1\") " pod="openshift-network-operator/iptables-alerter-qf6mp" Oct 30 13:48:51.544595 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.544567 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n98n9\" (UniqueName: \"kubernetes.io/projected/dc967875-1546-40bb-8376-fde09416c204-kube-api-access-n98n9\") pod \"console-operator-7cd56b8dbc-ntlqq\" (UID: \"dc967875-1546-40bb-8376-fde09416c204\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ntlqq" Oct 30 13:48:51.544908 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.544889 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69bm9\" (UniqueName: \"kubernetes.io/projected/6ac8569a-95ae-421e-aee0-edd189e18d11-kube-api-access-69bm9\") pod \"ingress-canary-xbkzq\" (UID: \"6ac8569a-95ae-421e-aee0-edd189e18d11\") " pod="openshift-ingress-canary/ingress-canary-xbkzq" Oct 30 13:48:51.544993 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.544981 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb8hg\" (UniqueName: \"kubernetes.io/projected/bf73153f-1421-44d6-abae-b76c3d90235e-kube-api-access-mb8hg\") pod \"network-check-target-dx94q\" (UID: \"bf73153f-1421-44d6-abae-b76c3d90235e\") " pod="openshift-network-diagnostics/network-check-target-dx94q" Oct 30 13:48:51.545341 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.545321 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f707b282-e507-475d-a82a-923c9bd1ff01-bound-sa-token\") pod \"image-registry-6bc4d9765d-dp7xb\" (UID: \"f707b282-e507-475d-a82a-923c9bd1ff01\") " pod="openshift-image-registry/image-registry-6bc4d9765d-dp7xb" Oct 30 13:48:51.545580 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.545560 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltkwj\" (UniqueName: \"kubernetes.io/projected/e9a035cc-e78b-4bdb-b0c6-ef13bc07249a-kube-api-access-ltkwj\") pod \"node-resolver-bjswt\" (UID: \"e9a035cc-e78b-4bdb-b0c6-ef13bc07249a\") " pod="openshift-dns/node-resolver-bjswt" Oct 30 13:48:51.545785 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.545765 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfgzb\" (UniqueName: \"kubernetes.io/projected/6d73c6c3-044d-4b88-a105-72bd5e4491cd-kube-api-access-tfgzb\") pod \"router-default-659879fb99-pjvcq\" (UID: \"6d73c6c3-044d-4b88-a105-72bd5e4491cd\") " pod="openshift-ingress/router-default-659879fb99-pjvcq" Oct 30 13:48:51.546547 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.546528 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp4w5\" (UniqueName: \"kubernetes.io/projected/a85af541-54ed-4d24-b4c5-d69244396157-kube-api-access-gp4w5\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:51.549387 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.548753 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn2f4\" (UniqueName: \"kubernetes.io/projected/da694c74-3bce-4d95-bd30-12a1891ad5e6-kube-api-access-bn2f4\") pod \"network-metrics-daemon-pfg7x\" (UID: \"da694c74-3bce-4d95-bd30-12a1891ad5e6\") " pod="openshift-multus/network-metrics-daemon-pfg7x" Oct 30 13:48:51.549387 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.549064 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p2xh\" (UniqueName: \"kubernetes.io/projected/083c70c2-0d5e-4876-9e88-10c6efbb681c-kube-api-access-6p2xh\") pod \"dns-default-7lx4t\" (UID: \"083c70c2-0d5e-4876-9e88-10c6efbb681c\") " pod="openshift-dns/dns-default-7lx4t" Oct 30 13:48:51.549387 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.549349 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk8lh\" (UniqueName: \"kubernetes.io/projected/806dd609-9a2a-49c4-a656-e50437457fa9-kube-api-access-gk8lh\") pod \"network-check-source-6f78d59f99-bhw8w\" (UID: \"806dd609-9a2a-49c4-a656-e50437457fa9\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bhw8w" Oct 30 13:48:51.550216 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.550194 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e16ea4af-ce12-442b-91df-092457a7e192-bound-sa-token\") pod \"image-registry-87f7c8d8c-mdvsm\" (UID: \"e16ea4af-ce12-442b-91df-092457a7e192\") " pod="openshift-image-registry/image-registry-87f7c8d8c-mdvsm" Oct 30 13:48:51.550464 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.550442 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm9mr\" (UniqueName: \"kubernetes.io/projected/f707b282-e507-475d-a82a-923c9bd1ff01-kube-api-access-vm9mr\") pod \"image-registry-6bc4d9765d-dp7xb\" (UID: \"f707b282-e507-475d-a82a-923c9bd1ff01\") " pod="openshift-image-registry/image-registry-6bc4d9765d-dp7xb" Oct 30 13:48:51.562795 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.562775 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brjls\" (UniqueName: \"kubernetes.io/projected/c06b4b47-6141-4263-94b3-803d9f28bb94-kube-api-access-brjls\") pod \"node-ca-llx79\" (UID: \"c06b4b47-6141-4263-94b3-803d9f28bb94\") " pod="openshift-image-registry/node-ca-llx79" Oct 30 13:48:51.562891 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.562801 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhdhm\" (UniqueName: \"kubernetes.io/projected/ccafe9ae-44d7-4c31-a6a5-ba8506bc0cea-kube-api-access-xhdhm\") pod \"migrator-66df5d496b-v2x2w\" (UID: \"ccafe9ae-44d7-4c31-a6a5-ba8506bc0cea\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-v2x2w" Oct 30 13:48:51.563157 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.563139 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwd54\" (UniqueName: \"kubernetes.io/projected/a4bd1bb2-25ae-423c-ba05-08eb742d340e-kube-api-access-pwd54\") pod \"insights-operator-7c4986d449-kcgbg\" (UID: \"a4bd1bb2-25ae-423c-ba05-08eb742d340e\") " pod="openshift-insights/insights-operator-7c4986d449-kcgbg" Oct 30 13:48:51.563492 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.563474 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r9cl\" (UniqueName: \"kubernetes.io/projected/6b536198-f804-4868-beb9-123229c3ae51-kube-api-access-5r9cl\") pod \"insights-runtime-extractor-87kd9\" (UID: \"6b536198-f804-4868-beb9-123229c3ae51\") " pod="openshift-insights/insights-runtime-extractor-87kd9" Oct 30 13:48:51.563825 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.563808 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l99vz\" (UniqueName: \"kubernetes.io/projected/cb90396e-d4e9-4eb0-888f-64c59d492fd4-kube-api-access-l99vz\") pod \"kas-connection-checker-685895c7c9-smvmg\" (UID: \"cb90396e-d4e9-4eb0-888f-64c59d492fd4\") " pod="kube-system/kas-connection-checker-685895c7c9-smvmg" Oct 30 13:48:51.564247 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.564232 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkhbw\" (UniqueName: \"kubernetes.io/projected/d7a64da6-7b20-4f20-90d7-bc23a71290e5-kube-api-access-zkhbw\") pod \"kas-connection-checker-685895c7c9-plvgd\" (UID: \"d7a64da6-7b20-4f20-90d7-bc23a71290e5\") " pod="kube-system/kas-connection-checker-685895c7c9-plvgd" Oct 30 13:48:51.566816 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.566794 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzwmc\" (UniqueName: \"kubernetes.io/projected/ade75be2-e3bb-4b50-bed0-c9c257739707-kube-api-access-tzwmc\") pod \"service-ca-operator-544d749c8c-tn69c\" (UID: \"ade75be2-e3bb-4b50-bed0-c9c257739707\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tn69c" Oct 30 13:48:51.567037 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.567019 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srtk8\" (UniqueName: \"kubernetes.io/projected/7cd2b593-c362-43c2-b0bb-7e5021791214-kube-api-access-srtk8\") pod \"migrator-66df5d496b-whnh9\" (UID: \"7cd2b593-c362-43c2-b0bb-7e5021791214\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-whnh9" Oct 30 13:48:51.567188 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.567170 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt65k\" (UniqueName: \"kubernetes.io/projected/0dda4c8a-827d-428f-a6a2-5e810210662b-kube-api-access-gt65k\") pod \"multus-additional-cni-plugins-94cdn\" (UID: \"0dda4c8a-827d-428f-a6a2-5e810210662b\") " pod="openshift-multus/multus-additional-cni-plugins-94cdn" Oct 30 13:48:51.567244 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.567223 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfktw\" (UniqueName: \"kubernetes.io/projected/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-kube-api-access-xfktw\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:51.567920 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.567900 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spnhx\" (UniqueName: \"kubernetes.io/projected/81dcad0b-d124-41b8-8b4e-d41049de217a-kube-api-access-spnhx\") pod \"cluster-samples-operator-6ddb6c8b94-qstzd\" (UID: \"81dcad0b-d124-41b8-8b4e-d41049de217a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qstzd" Oct 30 13:48:51.567980 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.567957 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6xq2\" (UniqueName: \"kubernetes.io/projected/aca47fe8-7068-4743-b6ab-283ee0874f66-kube-api-access-b6xq2\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:51.568131 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.568116 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx49w\" (UniqueName: \"kubernetes.io/projected/8d9908f6-aefa-4bde-9182-2c199f08a24a-kube-api-access-fx49w\") pod \"volume-data-source-validator-58c9f5f46d-brd96\" (UID: \"8d9908f6-aefa-4bde-9182-2c199f08a24a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-brd96" Oct 30 13:48:51.568328 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.568314 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v9rm\" (UniqueName: \"kubernetes.io/projected/e16ea4af-ce12-442b-91df-092457a7e192-kube-api-access-5v9rm\") pod \"image-registry-87f7c8d8c-mdvsm\" (UID: \"e16ea4af-ce12-442b-91df-092457a7e192\") " pod="openshift-image-registry/image-registry-87f7c8d8c-mdvsm" Oct 30 13:48:51.626230 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.626176 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-multus-cni-dir\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:51.626230 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.626209 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-host-var-lib-cni-bin\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:51.626230 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.626228 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d73c6c3-044d-4b88-a105-72bd5e4491cd-metrics-certs\") pod \"router-default-659879fb99-pjvcq\" (UID: \"6d73c6c3-044d-4b88-a105-72bd5e4491cd\") " pod="openshift-ingress/router-default-659879fb99-pjvcq" Oct 30 13:48:51.626413 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.626242 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c06b4b47-6141-4263-94b3-803d9f28bb94-host\") pod \"node-ca-llx79\" (UID: \"c06b4b47-6141-4263-94b3-803d9f28bb94\") " pod="openshift-image-registry/node-ca-llx79" Oct 30 13:48:51.626413 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.626264 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-run\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:51.626413 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.626285 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-lib-modules\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:51.626413 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.626294 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-host-var-lib-cni-bin\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:51.626413 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.626305 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-multus-cni-dir\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:51.626413 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.626307 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f707b282-e507-475d-a82a-923c9bd1ff01-registry-tls\") pod \"image-registry-6bc4d9765d-dp7xb\" (UID: \"f707b282-e507-475d-a82a-923c9bd1ff01\") " pod="openshift-image-registry/image-registry-6bc4d9765d-dp7xb" Oct 30 13:48:51.626413 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.626345 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/06f40dde-8b71-41e0-9acb-cede4f1d4347-device-dir\") pod \"aws-ebs-csi-driver-node-h9bxl\" (UID: \"06f40dde-8b71-41e0-9acb-cede4f1d4347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9bxl" Oct 30 13:48:51.626413 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.626356 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-run\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:51.626413 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.626367 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b0b2d906-7995-4769-8222-fa8ba2a30f3d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-9r9zr\" (UID: \"b0b2d906-7995-4769-8222-fa8ba2a30f3d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9r9zr" Oct 30 13:48:51.626413 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:48:51.626356 4982 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 13:48:51.626413 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.626313 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c06b4b47-6141-4263-94b3-803d9f28bb94-host\") pod \"node-ca-llx79\" (UID: \"c06b4b47-6141-4263-94b3-803d9f28bb94\") " pod="openshift-image-registry/node-ca-llx79" Oct 30 13:48:51.626413 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.626391 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-host-run-netns\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:51.626413 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:48:51.626364 4982 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 13:48:51.626413 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:48:51.626412 4982 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bc4d9765d-dp7xb: secret "image-registry-tls" not found Oct 30 13:48:51.626413 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.626417 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6b5e74da-5460-44ed-a599-b01620b64c92-kubelet-config\") pod \"global-pull-secret-syncer-bdmgn\" (UID: \"6b5e74da-5460-44ed-a599-b01620b64c92\") " pod="kube-system/global-pull-secret-syncer-bdmgn" Oct 30 13:48:51.626413 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.626420 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/06f40dde-8b71-41e0-9acb-cede4f1d4347-device-dir\") pod \"aws-ebs-csi-driver-node-h9bxl\" (UID: \"06f40dde-8b71-41e0-9acb-cede4f1d4347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9bxl" Oct 30 13:48:51.627136 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.626433 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-lib-modules\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:51.627136 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:48:51.626455 4982 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 13:48:51.627136 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.626460 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6b5e74da-5460-44ed-a599-b01620b64c92-kubelet-config\") pod \"global-pull-secret-syncer-bdmgn\" (UID: \"6b5e74da-5460-44ed-a599-b01620b64c92\") " pod="kube-system/global-pull-secret-syncer-bdmgn" Oct 30 13:48:51.627136 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.626469 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-host-run-netns\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:51.627136 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:48:51.626475 4982 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d73c6c3-044d-4b88-a105-72bd5e4491cd-metrics-certs podName:6d73c6c3-044d-4b88-a105-72bd5e4491cd nodeName:}" failed. No retries permitted until 2026-10-30 13:48:52.126419413 +0000 UTC m=+1.870553457 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d73c6c3-044d-4b88-a105-72bd5e4491cd-metrics-certs") pod "router-default-659879fb99-pjvcq" (UID: "6d73c6c3-044d-4b88-a105-72bd5e4491cd") : secret "router-metrics-certs-default" not found Oct 30 13:48:51.627136 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:48:51.626536 4982 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f707b282-e507-475d-a82a-923c9bd1ff01-registry-tls podName:f707b282-e507-475d-a82a-923c9bd1ff01 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:52.126511444 +0000 UTC m=+1.870645479 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f707b282-e507-475d-a82a-923c9bd1ff01-registry-tls") pod "image-registry-6bc4d9765d-dp7xb" (UID: "f707b282-e507-475d-a82a-923c9bd1ff01") : secret "image-registry-tls" not found Oct 30 13:48:51.627136 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:48:51.626556 4982 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0b2d906-7995-4769-8222-fa8ba2a30f3d-cluster-monitoring-operator-tls podName:b0b2d906-7995-4769-8222-fa8ba2a30f3d nodeName:}" failed. No retries permitted until 2026-10-30 13:48:52.126546393 +0000 UTC m=+1.870680417 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b0b2d906-7995-4769-8222-fa8ba2a30f3d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-9r9zr" (UID: "b0b2d906-7995-4769-8222-fa8ba2a30f3d") : secret "cluster-monitoring-operator-tls" not found Oct 30 13:48:51.627136 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.626581 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/81dcad0b-d124-41b8-8b4e-d41049de217a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-qstzd\" (UID: \"81dcad0b-d124-41b8-8b4e-d41049de217a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qstzd" Oct 30 13:48:51.627136 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.626606 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/06f40dde-8b71-41e0-9acb-cede4f1d4347-sys-fs\") pod \"aws-ebs-csi-driver-node-h9bxl\" (UID: \"06f40dde-8b71-41e0-9acb-cede4f1d4347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9bxl" Oct 30 13:48:51.627136 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.626659 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-etc-systemd\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:51.627136 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:48:51.626677 4982 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 13:48:51.627136 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.626705 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-multus-conf-dir\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:51.627136 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:48:51.626725 4982 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81dcad0b-d124-41b8-8b4e-d41049de217a-samples-operator-tls podName:81dcad0b-d124-41b8-8b4e-d41049de217a nodeName:}" failed. No retries permitted until 2026-10-30 13:48:52.126712566 +0000 UTC m=+1.870846599 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/81dcad0b-d124-41b8-8b4e-d41049de217a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-qstzd" (UID: "81dcad0b-d124-41b8-8b4e-d41049de217a") : secret "samples-operator-tls" not found Oct 30 13:48:51.627136 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.626725 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/06f40dde-8b71-41e0-9acb-cede4f1d4347-sys-fs\") pod \"aws-ebs-csi-driver-node-h9bxl\" (UID: \"06f40dde-8b71-41e0-9acb-cede4f1d4347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9bxl" Oct 30 13:48:51.627136 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.626745 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-multus-conf-dir\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:51.627136 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.626755 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e9a035cc-e78b-4bdb-b0c6-ef13bc07249a-hosts-file\") pod \"node-resolver-bjswt\" (UID: \"e9a035cc-e78b-4bdb-b0c6-ef13bc07249a\") " pod="openshift-dns/node-resolver-bjswt" Oct 30 13:48:51.627136 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.626780 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-host-cni-netd\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:51.627136 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.626797 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e9a035cc-e78b-4bdb-b0c6-ef13bc07249a-hosts-file\") pod \"node-resolver-bjswt\" (UID: \"e9a035cc-e78b-4bdb-b0c6-ef13bc07249a\") " pod="openshift-dns/node-resolver-bjswt" Oct 30 13:48:51.627136 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.626802 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/06f40dde-8b71-41e0-9acb-cede4f1d4347-socket-dir\") pod \"aws-ebs-csi-driver-node-h9bxl\" (UID: \"06f40dde-8b71-41e0-9acb-cede4f1d4347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9bxl" Oct 30 13:48:51.627136 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.626756 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-etc-systemd\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:51.627136 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.626809 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-host-cni-netd\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:51.627136 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.626823 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6b536198-f804-4868-beb9-123229c3ae51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-87kd9\" (UID: \"6b536198-f804-4868-beb9-123229c3ae51\") " pod="openshift-insights/insights-runtime-extractor-87kd9" Oct 30 13:48:51.627136 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.626842 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d73c6c3-044d-4b88-a105-72bd5e4491cd-service-ca-bundle\") pod \"router-default-659879fb99-pjvcq\" (UID: \"6d73c6c3-044d-4b88-a105-72bd5e4491cd\") " pod="openshift-ingress/router-default-659879fb99-pjvcq" Oct 30 13:48:51.627136 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.626860 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/06f40dde-8b71-41e0-9acb-cede4f1d4347-socket-dir\") pod \"aws-ebs-csi-driver-node-h9bxl\" (UID: \"06f40dde-8b71-41e0-9acb-cede4f1d4347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9bxl" Oct 30 13:48:51.627136 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:48:51.626907 4982 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 13:48:51.627136 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:48:51.626954 4982 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b536198-f804-4868-beb9-123229c3ae51-insights-runtime-extractor-tls podName:6b536198-f804-4868-beb9-123229c3ae51 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:52.126942968 +0000 UTC m=+1.871077001 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6b536198-f804-4868-beb9-123229c3ae51-insights-runtime-extractor-tls") pod "insights-runtime-extractor-87kd9" (UID: "6b536198-f804-4868-beb9-123229c3ae51") : secret "insights-runtime-extractor-tls" not found Oct 30 13:48:51.627136 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.626969 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-host\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:51.627136 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.626993 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-host-var-lib-cni-multus\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:51.627136 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627018 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-systemd-units\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:51.627136 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627039 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-host\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:51.627136 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627040 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-log-socket\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:51.627136 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627048 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-host-var-lib-cni-multus\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:51.627136 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627074 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-log-socket\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:51.627136 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627074 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-etc-sysctl-conf\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:51.627136 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627094 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-systemd-units\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:51.627136 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627109 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-multus-socket-dir-parent\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627149 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-etc-openvswitch\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627163 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-etc-sysctl-conf\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627173 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-multus-socket-dir-parent\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627178 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-node-log\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627204 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-etc-openvswitch\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627204 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0dda4c8a-827d-428f-a6a2-5e810210662b-system-cni-dir\") pod \"multus-additional-cni-plugins-94cdn\" (UID: \"0dda4c8a-827d-428f-a6a2-5e810210662b\") " pod="openshift-multus/multus-additional-cni-plugins-94cdn" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627235 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0dda4c8a-827d-428f-a6a2-5e810210662b-system-cni-dir\") pod \"multus-additional-cni-plugins-94cdn\" (UID: \"0dda4c8a-827d-428f-a6a2-5e810210662b\") " pod="openshift-multus/multus-additional-cni-plugins-94cdn" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627244 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-run-systemd\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627255 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-node-log\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627271 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/06f40dde-8b71-41e0-9acb-cede4f1d4347-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h9bxl\" (UID: \"06f40dde-8b71-41e0-9acb-cede4f1d4347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9bxl" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627279 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-run-systemd\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627298 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-sys\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627326 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/06f40dde-8b71-41e0-9acb-cede4f1d4347-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h9bxl\" (UID: \"06f40dde-8b71-41e0-9acb-cede4f1d4347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9bxl" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627324 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da694c74-3bce-4d95-bd30-12a1891ad5e6-metrics-certs\") pod \"network-metrics-daemon-pfg7x\" (UID: \"da694c74-3bce-4d95-bd30-12a1891ad5e6\") " pod="openshift-multus/network-metrics-daemon-pfg7x" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627352 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-run-openvswitch\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627360 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-sys\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627366 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-host-run-ovn-kubernetes\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627382 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-host-run-ovn-kubernetes\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:48:51.627387 4982 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627400 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-host-run-k8s-cni-cncf-io\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627405 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-run-openvswitch\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627425 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0dda4c8a-827d-428f-a6a2-5e810210662b-cnibin\") pod \"multus-additional-cni-plugins-94cdn\" (UID: \"0dda4c8a-827d-428f-a6a2-5e810210662b\") " pod="openshift-multus/multus-additional-cni-plugins-94cdn" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627421 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-host-run-k8s-cni-cncf-io\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:48:51.627439 4982 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da694c74-3bce-4d95-bd30-12a1891ad5e6-metrics-certs podName:da694c74-3bce-4d95-bd30-12a1891ad5e6 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:52.127425591 +0000 UTC m=+1.871559616 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/da694c74-3bce-4d95-bd30-12a1891ad5e6-metrics-certs") pod "network-metrics-daemon-pfg7x" (UID: "da694c74-3bce-4d95-bd30-12a1891ad5e6") : secret "metrics-daemon-secret" not found Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627459 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0dda4c8a-827d-428f-a6a2-5e810210662b-cnibin\") pod \"multus-additional-cni-plugins-94cdn\" (UID: \"0dda4c8a-827d-428f-a6a2-5e810210662b\") " pod="openshift-multus/multus-additional-cni-plugins-94cdn" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627475 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d73c6c3-044d-4b88-a105-72bd5e4491cd-service-ca-bundle\") pod \"router-default-659879fb99-pjvcq\" (UID: \"6d73c6c3-044d-4b88-a105-72bd5e4491cd\") " pod="openshift-ingress/router-default-659879fb99-pjvcq" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627475 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/06f40dde-8b71-41e0-9acb-cede4f1d4347-etc-selinux\") pod \"aws-ebs-csi-driver-node-h9bxl\" (UID: \"06f40dde-8b71-41e0-9acb-cede4f1d4347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9bxl" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627513 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0dda4c8a-827d-428f-a6a2-5e810210662b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-94cdn\" (UID: \"0dda4c8a-827d-428f-a6a2-5e810210662b\") " pod="openshift-multus/multus-additional-cni-plugins-94cdn" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627520 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/06f40dde-8b71-41e0-9acb-cede4f1d4347-etc-selinux\") pod \"aws-ebs-csi-driver-node-h9bxl\" (UID: \"06f40dde-8b71-41e0-9acb-cede4f1d4347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9bxl" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627534 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8309f073-dcb0-4971-9eb1-8a9ee0eb68c6-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2bnbl\" (UID: \"8309f073-dcb0-4971-9eb1-8a9ee0eb68c6\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2bnbl" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627561 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-host-kubelet\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627580 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0dda4c8a-827d-428f-a6a2-5e810210662b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-94cdn\" (UID: \"0dda4c8a-827d-428f-a6a2-5e810210662b\") " pod="openshift-multus/multus-additional-cni-plugins-94cdn" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627585 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-host-run-netns\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627607 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-host-cni-bin\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627614 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-host-kubelet\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627619 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-host-run-netns\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627626 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627668 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627679 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-etc-kubernetes\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627670 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-host-cni-bin\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627704 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-host-slash\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627731 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-etc-kubernetes\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627734 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-system-cni-dir\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627758 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-var-lib-openvswitch\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627765 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-host-slash\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627774 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-run-ovn\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627794 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/083c70c2-0d5e-4876-9e88-10c6efbb681c-metrics-tls\") pod \"dns-default-7lx4t\" (UID: \"083c70c2-0d5e-4876-9e88-10c6efbb681c\") " pod="openshift-dns/dns-default-7lx4t" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627798 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-var-lib-openvswitch\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627776 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-system-cni-dir\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627808 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-cnibin\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627825 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-hostroot\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627828 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/aca47fe8-7068-4743-b6ab-283ee0874f66-run-ovn\") pod \"ovnkube-node-bhxjf\" (UID: \"aca47fe8-7068-4743-b6ab-283ee0874f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627834 4982 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627840 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-etc-sysctl-d\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627872 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-cnibin\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627878 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-hostroot\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627900 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d38c246c-86f8-4b57-89bd-db4cd0d3f6d1-host-slash\") pod \"iptables-alerter-qf6mp\" (UID: \"d38c246c-86f8-4b57-89bd-db4cd0d3f6d1\") " pod="openshift-network-operator/iptables-alerter-qf6mp" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627927 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6b5e74da-5460-44ed-a599-b01620b64c92-dbus\") pod \"global-pull-secret-syncer-bdmgn\" (UID: \"6b5e74da-5460-44ed-a599-b01620b64c92\") " pod="kube-system/global-pull-secret-syncer-bdmgn" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627932 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-etc-sysctl-d\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627956 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-host-var-lib-kubelet\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627966 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d38c246c-86f8-4b57-89bd-db4cd0d3f6d1-host-slash\") pod \"iptables-alerter-qf6mp\" (UID: \"d38c246c-86f8-4b57-89bd-db4cd0d3f6d1\") " pod="openshift-network-operator/iptables-alerter-qf6mp" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627973 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6b536198-f804-4868-beb9-123229c3ae51-crio-socket\") pod \"insights-runtime-extractor-87kd9\" (UID: \"6b536198-f804-4868-beb9-123229c3ae51\") " pod="openshift-insights/insights-runtime-extractor-87kd9" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627988 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6b536198-f804-4868-beb9-123229c3ae51-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-87kd9\" (UID: \"6b536198-f804-4868-beb9-123229c3ae51\") " pod="openshift-insights/insights-runtime-extractor-87kd9" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.627996 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6b5e74da-5460-44ed-a599-b01620b64c92-dbus\") pod \"global-pull-secret-syncer-bdmgn\" (UID: \"6b5e74da-5460-44ed-a599-b01620b64c92\") " pod="kube-system/global-pull-secret-syncer-bdmgn" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.628005 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-host-var-lib-kubelet\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.628006 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e16ea4af-ce12-442b-91df-092457a7e192-registry-tls\") pod \"image-registry-87f7c8d8c-mdvsm\" (UID: \"e16ea4af-ce12-442b-91df-092457a7e192\") " pod="openshift-image-registry/image-registry-87f7c8d8c-mdvsm" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.628045 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6b536198-f804-4868-beb9-123229c3ae51-crio-socket\") pod \"insights-runtime-extractor-87kd9\" (UID: \"6b536198-f804-4868-beb9-123229c3ae51\") " pod="openshift-insights/insights-runtime-extractor-87kd9" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.628043 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-etc-modprobe-d\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:48:51.628075 4982 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.628082 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-var-lib-kubelet\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.628110 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-etc-kubernetes\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:48:51.628087 4982 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-87f7c8d8c-mdvsm: secret "image-registry-tls" not found Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.628126 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-var-lib-kubelet\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.628136 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-etc-sysconfig\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.628111 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-etc-modprobe-d\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:48:51.628150 4982 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e16ea4af-ce12-442b-91df-092457a7e192-registry-tls podName:e16ea4af-ce12-442b-91df-092457a7e192 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:52.128137832 +0000 UTC m=+1.872271855 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e16ea4af-ce12-442b-91df-092457a7e192-registry-tls") pod "image-registry-87f7c8d8c-mdvsm" (UID: "e16ea4af-ce12-442b-91df-092457a7e192") : secret "image-registry-tls" not found Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.628177 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-etc-kubernetes\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.628185 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-host-run-multus-certs\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.628210 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a85af541-54ed-4d24-b4c5-d69244396157-host-run-multus-certs\") pod \"multus-rvt5b\" (UID: \"a85af541-54ed-4d24-b4c5-d69244396157\") " pod="openshift-multus/multus-rvt5b" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.628211 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/06f40dde-8b71-41e0-9acb-cede4f1d4347-registration-dir\") pod \"aws-ebs-csi-driver-node-h9bxl\" (UID: \"06f40dde-8b71-41e0-9acb-cede4f1d4347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9bxl" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.628182 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0a5f2fbe-141f-4beb-a5b5-0e2ba4686203-etc-sysconfig\") pod \"tuned-tqst8\" (UID: \"0a5f2fbe-141f-4beb-a5b5-0e2ba4686203\") " pod="openshift-cluster-node-tuning-operator/tuned-tqst8" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.628241 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/06f40dde-8b71-41e0-9acb-cede4f1d4347-registration-dir\") pod \"aws-ebs-csi-driver-node-h9bxl\" (UID: \"06f40dde-8b71-41e0-9acb-cede4f1d4347\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h9bxl" Oct 30 13:48:51.628720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.628457 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6b536198-f804-4868-beb9-123229c3ae51-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-87kd9\" (UID: \"6b536198-f804-4868-beb9-123229c3ae51\") " pod="openshift-insights/insights-runtime-extractor-87kd9" Oct 30 13:48:51.631437 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.629997 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8309f073-dcb0-4971-9eb1-8a9ee0eb68c6-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2bnbl\" (UID: \"8309f073-dcb0-4971-9eb1-8a9ee0eb68c6\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2bnbl" Oct 30 13:48:51.631437 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.630334 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/083c70c2-0d5e-4876-9e88-10c6efbb681c-metrics-tls\") pod \"dns-default-7lx4t\" (UID: \"083c70c2-0d5e-4876-9e88-10c6efbb681c\") " pod="openshift-dns/dns-default-7lx4t" Oct 30 13:48:51.714809 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.714780 4982 scope.go:122] "RemoveContainer" containerID="08dafe630716a5a619c9abe02a35bcce9e3b7cb581311f535ccb7aff3f8803cb" Oct 30 13:48:51.715263 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.715247 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-2bnbl" Oct 30 13:48:51.722305 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.722286 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7lx4t" Oct 30 13:48:51.744571 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:51.744549 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:52.032696 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:52.032668 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:52.056219 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:52.056197 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-2bnbl"] Oct 30 13:48:52.058500 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:52.058474 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8309f073_dcb0_4971_9eb1_8a9ee0eb68c6.slice/crio-a0e5cde7ca5774941b5022249f986ff0a2cd1ee63f6afa180b09595026a755ac WatchSource:0}: Error finding container a0e5cde7ca5774941b5022249f986ff0a2cd1ee63f6afa180b09595026a755ac: Status 404 returned error can't find the container with id a0e5cde7ca5774941b5022249f986ff0a2cd1ee63f6afa180b09595026a755ac Oct 30 13:48:52.060370 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:52.060355 4982 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 13:48:52.092884 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:52.092866 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7lx4t"] Oct 30 13:48:52.094306 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:52.094277 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod083c70c2_0d5e_4876_9e88_10c6efbb681c.slice/crio-a430d61b42c6e5724ababaa99bdc2dfb99ac6e173d0d4bded5b782371fd8b030 WatchSource:0}: Error finding container a430d61b42c6e5724ababaa99bdc2dfb99ac6e173d0d4bded5b782371fd8b030: Status 404 returned error can't find the container with id a430d61b42c6e5724ababaa99bdc2dfb99ac6e173d0d4bded5b782371fd8b030 Oct 30 13:48:52.109185 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:52.109151 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:52.132743 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:52.132695 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e16ea4af-ce12-442b-91df-092457a7e192-registry-tls\") pod \"image-registry-87f7c8d8c-mdvsm\" (UID: \"e16ea4af-ce12-442b-91df-092457a7e192\") " pod="openshift-image-registry/image-registry-87f7c8d8c-mdvsm" Oct 30 13:48:52.132743 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:52.132730 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d73c6c3-044d-4b88-a105-72bd5e4491cd-metrics-certs\") pod \"router-default-659879fb99-pjvcq\" (UID: \"6d73c6c3-044d-4b88-a105-72bd5e4491cd\") " pod="openshift-ingress/router-default-659879fb99-pjvcq" Oct 30 13:48:52.132877 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:52.132762 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f707b282-e507-475d-a82a-923c9bd1ff01-registry-tls\") pod \"image-registry-6bc4d9765d-dp7xb\" (UID: \"f707b282-e507-475d-a82a-923c9bd1ff01\") " pod="openshift-image-registry/image-registry-6bc4d9765d-dp7xb" Oct 30 13:48:52.132877 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:48:52.132867 4982 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 13:48:52.132963 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:48:52.132886 4982 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-87f7c8d8c-mdvsm: secret "image-registry-tls" not found Oct 30 13:48:52.132963 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:48:52.132920 4982 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 13:48:52.132963 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:52.132869 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b0b2d906-7995-4769-8222-fa8ba2a30f3d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-9r9zr\" (UID: \"b0b2d906-7995-4769-8222-fa8ba2a30f3d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9r9zr" Oct 30 13:48:52.132963 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:48:52.132935 4982 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e16ea4af-ce12-442b-91df-092457a7e192-registry-tls podName:e16ea4af-ce12-442b-91df-092457a7e192 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:53.132917661 +0000 UTC m=+2.877051685 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e16ea4af-ce12-442b-91df-092457a7e192-registry-tls") pod "image-registry-87f7c8d8c-mdvsm" (UID: "e16ea4af-ce12-442b-91df-092457a7e192") : secret "image-registry-tls" not found Oct 30 13:48:52.132963 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:48:52.132937 4982 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bc4d9765d-dp7xb: secret "image-registry-tls" not found Oct 30 13:48:52.132963 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:52.132962 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/81dcad0b-d124-41b8-8b4e-d41049de217a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-qstzd\" (UID: \"81dcad0b-d124-41b8-8b4e-d41049de217a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qstzd" Oct 30 13:48:52.133242 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:48:52.132996 4982 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f707b282-e507-475d-a82a-923c9bd1ff01-registry-tls podName:f707b282-e507-475d-a82a-923c9bd1ff01 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:53.132976513 +0000 UTC m=+2.877110538 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f707b282-e507-475d-a82a-923c9bd1ff01-registry-tls") pod "image-registry-6bc4d9765d-dp7xb" (UID: "f707b282-e507-475d-a82a-923c9bd1ff01") : secret "image-registry-tls" not found Oct 30 13:48:52.133242 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:48:52.132926 4982 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 13:48:52.133242 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:48:52.133018 4982 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 13:48:52.133242 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:52.133031 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6b536198-f804-4868-beb9-123229c3ae51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-87kd9\" (UID: \"6b536198-f804-4868-beb9-123229c3ae51\") " pod="openshift-insights/insights-runtime-extractor-87kd9" Oct 30 13:48:52.133242 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:48:52.133060 4982 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81dcad0b-d124-41b8-8b4e-d41049de217a-samples-operator-tls podName:81dcad0b-d124-41b8-8b4e-d41049de217a nodeName:}" failed. No retries permitted until 2026-10-30 13:48:53.13304551 +0000 UTC m=+2.877179529 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/81dcad0b-d124-41b8-8b4e-d41049de217a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-qstzd" (UID: "81dcad0b-d124-41b8-8b4e-d41049de217a") : secret "samples-operator-tls" not found Oct 30 13:48:52.133242 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:48:52.133080 4982 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0b2d906-7995-4769-8222-fa8ba2a30f3d-cluster-monitoring-operator-tls podName:b0b2d906-7995-4769-8222-fa8ba2a30f3d nodeName:}" failed. No retries permitted until 2026-10-30 13:48:53.133071453 +0000 UTC m=+2.877205472 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b0b2d906-7995-4769-8222-fa8ba2a30f3d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-9r9zr" (UID: "b0b2d906-7995-4769-8222-fa8ba2a30f3d") : secret "cluster-monitoring-operator-tls" not found Oct 30 13:48:52.133242 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:48:52.133092 4982 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 13:48:52.133242 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:48:52.133125 4982 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b536198-f804-4868-beb9-123229c3ae51-insights-runtime-extractor-tls podName:6b536198-f804-4868-beb9-123229c3ae51 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:53.133116592 +0000 UTC m=+2.877250611 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6b536198-f804-4868-beb9-123229c3ae51-insights-runtime-extractor-tls") pod "insights-runtime-extractor-87kd9" (UID: "6b536198-f804-4868-beb9-123229c3ae51") : secret "insights-runtime-extractor-tls" not found Oct 30 13:48:52.133242 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:52.133123 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da694c74-3bce-4d95-bd30-12a1891ad5e6-metrics-certs\") pod \"network-metrics-daemon-pfg7x\" (UID: \"da694c74-3bce-4d95-bd30-12a1891ad5e6\") " pod="openshift-multus/network-metrics-daemon-pfg7x" Oct 30 13:48:52.133242 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:48:52.133160 4982 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 13:48:52.133242 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:48:52.133187 4982 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da694c74-3bce-4d95-bd30-12a1891ad5e6-metrics-certs podName:da694c74-3bce-4d95-bd30-12a1891ad5e6 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:53.133177571 +0000 UTC m=+2.877311593 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/da694c74-3bce-4d95-bd30-12a1891ad5e6-metrics-certs") pod "network-metrics-daemon-pfg7x" (UID: "da694c74-3bce-4d95-bd30-12a1891ad5e6") : secret "metrics-daemon-secret" not found Oct 30 13:48:52.134919 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:52.134903 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d73c6c3-044d-4b88-a105-72bd5e4491cd-metrics-certs\") pod \"router-default-659879fb99-pjvcq\" (UID: \"6d73c6c3-044d-4b88-a105-72bd5e4491cd\") " pod="openshift-ingress/router-default-659879fb99-pjvcq" Oct 30 13:48:52.315161 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:52.315138 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-659879fb99-pjvcq" Oct 30 13:48:52.453553 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:52.453532 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ntlqq_dc967875-1546-40bb-8376-fde09416c204/console-operator/2.log" Oct 30 13:48:52.453974 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:52.453957 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ntlqq_dc967875-1546-40bb-8376-fde09416c204/console-operator/1.log" Oct 30 13:48:52.454040 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:52.453994 4982 generic.go:356] "Generic (PLEG): container finished" podID="dc967875-1546-40bb-8376-fde09416c204" containerID="34d4502189edf1503539ba2cbc4804cc8f474fe80abf87de7b0259d75495aacf" exitCode=255 Oct 30 13:48:52.454102 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:52.454069 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-ntlqq" event={"ID":"dc967875-1546-40bb-8376-fde09416c204","Type":"ContainerDied","Data":"34d4502189edf1503539ba2cbc4804cc8f474fe80abf87de7b0259d75495aacf"} Oct 30 13:48:52.454156 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:52.454120 4982 scope.go:122] "RemoveContainer" containerID="08dafe630716a5a619c9abe02a35bcce9e3b7cb581311f535ccb7aff3f8803cb" Oct 30 13:48:52.454470 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:52.454453 4982 scope.go:122] "RemoveContainer" containerID="34d4502189edf1503539ba2cbc4804cc8f474fe80abf87de7b0259d75495aacf" Oct 30 13:48:52.454699 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:48:52.454672 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-ntlqq_openshift-console-operator(dc967875-1546-40bb-8376-fde09416c204)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-ntlqq" podUID="dc967875-1546-40bb-8376-fde09416c204" Oct 30 13:48:52.455332 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:52.455314 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7lx4t" event={"ID":"083c70c2-0d5e-4876-9e88-10c6efbb681c","Type":"ContainerStarted","Data":"a430d61b42c6e5724ababaa99bdc2dfb99ac6e173d0d4bded5b782371fd8b030"} Oct 30 13:48:52.456464 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:52.456443 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-2bnbl" event={"ID":"8309f073-dcb0-4971-9eb1-8a9ee0eb68c6","Type":"ContainerStarted","Data":"a0e5cde7ca5774941b5022249f986ff0a2cd1ee63f6afa180b09595026a755ac"} Oct 30 13:48:52.635163 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:52.635064 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-659879fb99-pjvcq"] Oct 30 13:48:52.636957 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:52.636928 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d73c6c3_044d_4b88_a105_72bd5e4491cd.slice/crio-3ec67e5c0cbe527916ec938feb97fb31a48763dd82552b65057ce02e5307e56a WatchSource:0}: Error finding container 3ec67e5c0cbe527916ec938feb97fb31a48763dd82552b65057ce02e5307e56a: Status 404 returned error can't find the container with id 3ec67e5c0cbe527916ec938feb97fb31a48763dd82552b65057ce02e5307e56a Oct 30 13:48:52.644085 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:52.643080 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-llx79_c06b4b47-6141-4263-94b3-803d9f28bb94/node-ca/0.log" Oct 30 13:48:52.707150 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:52.707101 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:52.914384 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:52.914327 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:53.142583 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:53.142548 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e16ea4af-ce12-442b-91df-092457a7e192-registry-tls\") pod \"image-registry-87f7c8d8c-mdvsm\" (UID: \"e16ea4af-ce12-442b-91df-092457a7e192\") " pod="openshift-image-registry/image-registry-87f7c8d8c-mdvsm" Oct 30 13:48:53.142788 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:53.142619 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f707b282-e507-475d-a82a-923c9bd1ff01-registry-tls\") pod \"image-registry-6bc4d9765d-dp7xb\" (UID: \"f707b282-e507-475d-a82a-923c9bd1ff01\") " pod="openshift-image-registry/image-registry-6bc4d9765d-dp7xb" Oct 30 13:48:53.142788 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:53.142667 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b0b2d906-7995-4769-8222-fa8ba2a30f3d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-9r9zr\" (UID: \"b0b2d906-7995-4769-8222-fa8ba2a30f3d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9r9zr" Oct 30 13:48:53.142788 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:53.142698 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/81dcad0b-d124-41b8-8b4e-d41049de217a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-qstzd\" (UID: \"81dcad0b-d124-41b8-8b4e-d41049de217a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qstzd" Oct 30 13:48:53.142788 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:53.142733 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6b536198-f804-4868-beb9-123229c3ae51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-87kd9\" (UID: \"6b536198-f804-4868-beb9-123229c3ae51\") " pod="openshift-insights/insights-runtime-extractor-87kd9" Oct 30 13:48:53.142788 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:53.142782 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da694c74-3bce-4d95-bd30-12a1891ad5e6-metrics-certs\") pod \"network-metrics-daemon-pfg7x\" (UID: \"da694c74-3bce-4d95-bd30-12a1891ad5e6\") " pod="openshift-multus/network-metrics-daemon-pfg7x" Oct 30 13:48:53.142961 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:48:53.142906 4982 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 13:48:53.142961 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:48:53.142958 4982 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da694c74-3bce-4d95-bd30-12a1891ad5e6-metrics-certs podName:da694c74-3bce-4d95-bd30-12a1891ad5e6 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:55.142940684 +0000 UTC m=+4.887074706 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/da694c74-3bce-4d95-bd30-12a1891ad5e6-metrics-certs") pod "network-metrics-daemon-pfg7x" (UID: "da694c74-3bce-4d95-bd30-12a1891ad5e6") : secret "metrics-daemon-secret" not found Oct 30 13:48:53.143320 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:48:53.143297 4982 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 13:48:53.143320 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:48:53.143318 4982 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-87f7c8d8c-mdvsm: secret "image-registry-tls" not found Oct 30 13:48:53.143475 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:48:53.143361 4982 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e16ea4af-ce12-442b-91df-092457a7e192-registry-tls podName:e16ea4af-ce12-442b-91df-092457a7e192 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:55.143346659 +0000 UTC m=+4.887480693 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e16ea4af-ce12-442b-91df-092457a7e192-registry-tls") pod "image-registry-87f7c8d8c-mdvsm" (UID: "e16ea4af-ce12-442b-91df-092457a7e192") : secret "image-registry-tls" not found Oct 30 13:48:53.143475 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:48:53.143418 4982 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 13:48:53.143475 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:48:53.143428 4982 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bc4d9765d-dp7xb: secret "image-registry-tls" not found Oct 30 13:48:53.143475 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:48:53.143456 4982 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f707b282-e507-475d-a82a-923c9bd1ff01-registry-tls podName:f707b282-e507-475d-a82a-923c9bd1ff01 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:55.143446261 +0000 UTC m=+4.887580295 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f707b282-e507-475d-a82a-923c9bd1ff01-registry-tls") pod "image-registry-6bc4d9765d-dp7xb" (UID: "f707b282-e507-475d-a82a-923c9bd1ff01") : secret "image-registry-tls" not found Oct 30 13:48:53.143864 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:48:53.143839 4982 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 13:48:53.143955 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:48:53.143910 4982 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b536198-f804-4868-beb9-123229c3ae51-insights-runtime-extractor-tls podName:6b536198-f804-4868-beb9-123229c3ae51 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:55.143893471 +0000 UTC m=+4.888027500 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6b536198-f804-4868-beb9-123229c3ae51-insights-runtime-extractor-tls") pod "insights-runtime-extractor-87kd9" (UID: "6b536198-f804-4868-beb9-123229c3ae51") : secret "insights-runtime-extractor-tls" not found Oct 30 13:48:53.146282 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:53.146260 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b0b2d906-7995-4769-8222-fa8ba2a30f3d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-9r9zr\" (UID: \"b0b2d906-7995-4769-8222-fa8ba2a30f3d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9r9zr" Oct 30 13:48:53.146630 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:53.146612 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/81dcad0b-d124-41b8-8b4e-d41049de217a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-qstzd\" (UID: \"81dcad0b-d124-41b8-8b4e-d41049de217a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qstzd" Oct 30 13:48:53.210357 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:53.210331 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-ntlqq" Oct 30 13:48:53.213509 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:53.213487 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qstzd" Oct 30 13:48:53.213654 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:53.213619 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9r9zr" Oct 30 13:48:53.243185 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:53.242988 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xbkzq_6ac8569a-95ae-421e-aee0-edd189e18d11/serve-healthcheck-canary/0.log" Oct 30 13:48:53.301085 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:53.301059 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:53.461480 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:53.461404 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-2bnbl" event={"ID":"8309f073-dcb0-4971-9eb1-8a9ee0eb68c6","Type":"ContainerStarted","Data":"df4f2ea1873bd09afb2ea3cbe9677355a14a7ec26035290fdb7b1f5dc3a6c265"} Oct 30 13:48:53.463236 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:53.463209 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-659879fb99-pjvcq" event={"ID":"6d73c6c3-044d-4b88-a105-72bd5e4491cd","Type":"ContainerStarted","Data":"78c590bebabfaa34ad7aa78bd4d64b3bbb8e22ab33f93e890fd9525ea06ece1c"} Oct 30 13:48:53.463353 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:53.463262 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-659879fb99-pjvcq" event={"ID":"6d73c6c3-044d-4b88-a105-72bd5e4491cd","Type":"ContainerStarted","Data":"3ec67e5c0cbe527916ec938feb97fb31a48763dd82552b65057ce02e5307e56a"} Oct 30 13:48:53.468739 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:53.468719 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ntlqq_dc967875-1546-40bb-8376-fde09416c204/console-operator/2.log" Oct 30 13:48:53.469046 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:53.469028 4982 scope.go:122] "RemoveContainer" containerID="34d4502189edf1503539ba2cbc4804cc8f474fe80abf87de7b0259d75495aacf" Oct 30 13:48:53.469223 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:48:53.469201 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-ntlqq_openshift-console-operator(dc967875-1546-40bb-8376-fde09416c204)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-ntlqq" podUID="dc967875-1546-40bb-8376-fde09416c204" Oct 30 13:48:53.509171 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:53.509147 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:53.554596 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:53.554573 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-9r9zr"] Oct 30 13:48:53.555479 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:53.555461 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qstzd"] Oct 30 13:48:53.581697 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:53.581673 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0b2d906_7995_4769_8222_fa8ba2a30f3d.slice/crio-b2035be810d7e334eb961786f8346ca5c05bafdbcaa148069515f4b60803ff10 WatchSource:0}: Error finding container b2035be810d7e334eb961786f8346ca5c05bafdbcaa148069515f4b60803ff10: Status 404 returned error can't find the container with id b2035be810d7e334eb961786f8346ca5c05bafdbcaa148069515f4b60803ff10 Oct 30 13:48:53.643397 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:53.643371 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-v2x2w_ccafe9ae-44d7-4c31-a6a5-ba8506bc0cea/migrator/0.log" Oct 30 13:48:53.682326 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:53.682300 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:53.843052 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:53.843025 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-whnh9_7cd2b593-c362-43c2-b0bb-7e5021791214/migrator/0.log" Oct 30 13:48:54.043618 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:54.043595 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-2nsk6_8304f8da-d4a3-45c1-bfe6-f5d08bed17d7/kube-storage-version-migrator-operator/0.log" Oct 30 13:48:54.105707 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:54.105618 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:54.316112 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:54.316079 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-659879fb99-pjvcq" Oct 30 13:48:54.319310 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:54.319290 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-659879fb99-pjvcq" Oct 30 13:48:54.475685 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:54.475595 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7lx4t" event={"ID":"083c70c2-0d5e-4876-9e88-10c6efbb681c","Type":"ContainerStarted","Data":"8281f2ecfd2e497a89b9b1545c6312a3ac8f1a5e135fe19319f593af1ea9fdc8"} Oct 30 13:48:54.475685 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:54.475678 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7lx4t" event={"ID":"083c70c2-0d5e-4876-9e88-10c6efbb681c","Type":"ContainerStarted","Data":"c91a1c9643fd72144c5c5fa33831e10291891645cf90f5012da1eb5036220c7e"} Oct 30 13:48:54.476161 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:54.475725 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-7lx4t" Oct 30 13:48:54.477306 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:54.477259 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9r9zr" event={"ID":"b0b2d906-7995-4769-8222-fa8ba2a30f3d","Type":"ContainerStarted","Data":"b2035be810d7e334eb961786f8346ca5c05bafdbcaa148069515f4b60803ff10"} Oct 30 13:48:54.479015 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:54.478988 4982 generic.go:356] "Generic (PLEG): container finished" podID="8304f8da-d4a3-45c1-bfe6-f5d08bed17d7" containerID="cf977f7a9365f43c6bbab4365b6941118e3b94afb26b89a5a377f8b8f72c7f78" exitCode=0 Oct 30 13:48:54.479104 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:54.479053 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2nsk6" event={"ID":"8304f8da-d4a3-45c1-bfe6-f5d08bed17d7","Type":"ContainerDied","Data":"cf977f7a9365f43c6bbab4365b6941118e3b94afb26b89a5a377f8b8f72c7f78"} Oct 30 13:48:54.479314 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:54.479297 4982 scope.go:122] "RemoveContainer" containerID="cf977f7a9365f43c6bbab4365b6941118e3b94afb26b89a5a377f8b8f72c7f78" Oct 30 13:48:54.480721 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:54.480684 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qstzd" event={"ID":"81dcad0b-d124-41b8-8b4e-d41049de217a","Type":"ContainerStarted","Data":"ee52b05f72affbed713bcaa2ff2b6fff1b006140f18c7294c4fd4a3101468d27"} Oct 30 13:48:54.702250 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:54.702215 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:54.901914 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:54.901882 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:55.163466 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:55.163386 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e16ea4af-ce12-442b-91df-092457a7e192-registry-tls\") pod \"image-registry-87f7c8d8c-mdvsm\" (UID: \"e16ea4af-ce12-442b-91df-092457a7e192\") " pod="openshift-image-registry/image-registry-87f7c8d8c-mdvsm" Oct 30 13:48:55.163466 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:55.163446 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f707b282-e507-475d-a82a-923c9bd1ff01-registry-tls\") pod \"image-registry-6bc4d9765d-dp7xb\" (UID: \"f707b282-e507-475d-a82a-923c9bd1ff01\") " pod="openshift-image-registry/image-registry-6bc4d9765d-dp7xb" Oct 30 13:48:55.163710 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:55.163490 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6b536198-f804-4868-beb9-123229c3ae51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-87kd9\" (UID: \"6b536198-f804-4868-beb9-123229c3ae51\") " pod="openshift-insights/insights-runtime-extractor-87kd9" Oct 30 13:48:55.163710 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:55.163534 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da694c74-3bce-4d95-bd30-12a1891ad5e6-metrics-certs\") pod \"network-metrics-daemon-pfg7x\" (UID: \"da694c74-3bce-4d95-bd30-12a1891ad5e6\") " pod="openshift-multus/network-metrics-daemon-pfg7x" Oct 30 13:48:55.163710 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:48:55.163552 4982 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 13:48:55.163710 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:48:55.163570 4982 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-87f7c8d8c-mdvsm: secret "image-registry-tls" not found Oct 30 13:48:55.163710 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:48:55.163630 4982 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e16ea4af-ce12-442b-91df-092457a7e192-registry-tls podName:e16ea4af-ce12-442b-91df-092457a7e192 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:59.163612859 +0000 UTC m=+8.907746882 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e16ea4af-ce12-442b-91df-092457a7e192-registry-tls") pod "image-registry-87f7c8d8c-mdvsm" (UID: "e16ea4af-ce12-442b-91df-092457a7e192") : secret "image-registry-tls" not found Oct 30 13:48:55.163710 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:48:55.163655 4982 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 13:48:55.163710 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:48:55.163677 4982 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bc4d9765d-dp7xb: secret "image-registry-tls" not found Oct 30 13:48:55.163710 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:48:55.163683 4982 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 13:48:55.164074 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:48:55.163720 4982 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da694c74-3bce-4d95-bd30-12a1891ad5e6-metrics-certs podName:da694c74-3bce-4d95-bd30-12a1891ad5e6 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:59.163709677 +0000 UTC m=+8.907843711 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/da694c74-3bce-4d95-bd30-12a1891ad5e6-metrics-certs") pod "network-metrics-daemon-pfg7x" (UID: "da694c74-3bce-4d95-bd30-12a1891ad5e6") : secret "metrics-daemon-secret" not found Oct 30 13:48:55.164074 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:48:55.163735 4982 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f707b282-e507-475d-a82a-923c9bd1ff01-registry-tls podName:f707b282-e507-475d-a82a-923c9bd1ff01 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:59.163727409 +0000 UTC m=+8.907861428 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f707b282-e507-475d-a82a-923c9bd1ff01-registry-tls") pod "image-registry-6bc4d9765d-dp7xb" (UID: "f707b282-e507-475d-a82a-923c9bd1ff01") : secret "image-registry-tls" not found Oct 30 13:48:55.164074 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:48:55.163780 4982 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 13:48:55.164074 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:48:55.163824 4982 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b536198-f804-4868-beb9-123229c3ae51-insights-runtime-extractor-tls podName:6b536198-f804-4868-beb9-123229c3ae51 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:59.16380503 +0000 UTC m=+8.907939055 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6b536198-f804-4868-beb9-123229c3ae51-insights-runtime-extractor-tls") pod "insights-runtime-extractor-87kd9" (UID: "6b536198-f804-4868-beb9-123229c3ae51") : secret "insights-runtime-extractor-tls" not found Oct 30 13:48:55.307733 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:55.307708 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:55.486971 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:55.486940 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2nsk6" event={"ID":"8304f8da-d4a3-45c1-bfe6-f5d08bed17d7","Type":"ContainerStarted","Data":"91cbe2a2dead752a507bee6505fcc9a11274e8e81a92209c50ee4a97030455a8"} Oct 30 13:48:55.492196 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:55.489612 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qstzd" event={"ID":"81dcad0b-d124-41b8-8b4e-d41049de217a","Type":"ContainerStarted","Data":"ee4ea4fcfae64b82f3f744da8005fe7716f9fe60e86765f4b12aef415b48a9d5"} Oct 30 13:48:55.492196 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:55.489728 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qstzd" event={"ID":"81dcad0b-d124-41b8-8b4e-d41049de217a","Type":"ContainerStarted","Data":"708924e105202f00e824516bdc20cd7532550e197d6d980f61b614a09971340a"} Oct 30 13:48:55.495349 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:55.495322 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9r9zr" event={"ID":"b0b2d906-7995-4769-8222-fa8ba2a30f3d","Type":"ContainerStarted","Data":"138200aa5a799e80b604a451517a5ce515d23189cdced3976ea8d0a10cb71eee"} Oct 30 13:48:55.495584 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:55.495566 4982 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 13:48:55.510106 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:55.510078 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:55.699720 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:55.699627 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:48:55.699849 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:55.699825 4982 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 13:48:55.812730 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:55.812675 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:55.962623 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:55.962520 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:56.179768 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:56.179716 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:56.327764 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:56.327743 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:56.467370 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:56.467342 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:56.482511 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:56.482491 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-659879fb99-pjvcq" Oct 30 13:48:56.483692 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:56.483675 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-659879fb99-pjvcq" Oct 30 13:48:56.706915 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:56.706852 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:56.902462 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:56.902436 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:57.043237 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:57.043178 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:57.319545 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:57.319517 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:57.505449 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:57.505420 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:58.102852 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:58.102820 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:58.414116 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:58.414045 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:58.558909 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:58.558880 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:58.694905 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:58.694848 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:58.895386 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:58.895362 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:59.045108 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:59.045051 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:59.199328 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:59.199301 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da694c74-3bce-4d95-bd30-12a1891ad5e6-metrics-certs\") pod \"network-metrics-daemon-pfg7x\" (UID: \"da694c74-3bce-4d95-bd30-12a1891ad5e6\") " pod="openshift-multus/network-metrics-daemon-pfg7x" Oct 30 13:48:59.199679 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:59.199345 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e16ea4af-ce12-442b-91df-092457a7e192-registry-tls\") pod \"image-registry-87f7c8d8c-mdvsm\" (UID: \"e16ea4af-ce12-442b-91df-092457a7e192\") " pod="openshift-image-registry/image-registry-87f7c8d8c-mdvsm" Oct 30 13:48:59.199679 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:59.199374 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f707b282-e507-475d-a82a-923c9bd1ff01-registry-tls\") pod \"image-registry-6bc4d9765d-dp7xb\" (UID: \"f707b282-e507-475d-a82a-923c9bd1ff01\") " pod="openshift-image-registry/image-registry-6bc4d9765d-dp7xb" Oct 30 13:48:59.199679 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:59.199409 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6b536198-f804-4868-beb9-123229c3ae51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-87kd9\" (UID: \"6b536198-f804-4868-beb9-123229c3ae51\") " pod="openshift-insights/insights-runtime-extractor-87kd9" Oct 30 13:48:59.199679 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:48:59.199495 4982 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 13:48:59.199679 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:48:59.199555 4982 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b536198-f804-4868-beb9-123229c3ae51-insights-runtime-extractor-tls podName:6b536198-f804-4868-beb9-123229c3ae51 nodeName:}" failed. No retries permitted until 2026-10-30 13:49:07.199539545 +0000 UTC m=+16.943673566 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6b536198-f804-4868-beb9-123229c3ae51-insights-runtime-extractor-tls") pod "insights-runtime-extractor-87kd9" (UID: "6b536198-f804-4868-beb9-123229c3ae51") : secret "insights-runtime-extractor-tls" not found Oct 30 13:48:59.201595 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:59.201568 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da694c74-3bce-4d95-bd30-12a1891ad5e6-metrics-certs\") pod \"network-metrics-daemon-pfg7x\" (UID: \"da694c74-3bce-4d95-bd30-12a1891ad5e6\") " pod="openshift-multus/network-metrics-daemon-pfg7x" Oct 30 13:48:59.201741 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:59.201723 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f707b282-e507-475d-a82a-923c9bd1ff01-registry-tls\") pod \"image-registry-6bc4d9765d-dp7xb\" (UID: \"f707b282-e507-475d-a82a-923c9bd1ff01\") " pod="openshift-image-registry/image-registry-6bc4d9765d-dp7xb" Oct 30 13:48:59.201897 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:59.201880 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e16ea4af-ce12-442b-91df-092457a7e192-registry-tls\") pod \"image-registry-87f7c8d8c-mdvsm\" (UID: \"e16ea4af-ce12-442b-91df-092457a7e192\") " pod="openshift-image-registry/image-registry-87f7c8d8c-mdvsm" Oct 30 13:48:59.213801 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:59.213782 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bc4d9765d-dp7xb" Oct 30 13:48:59.217421 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:59.217403 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pfg7x" Oct 30 13:48:59.222073 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:59.222054 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-87f7c8d8c-mdvsm" Oct 30 13:48:59.299198 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:59.299103 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:59.497699 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:59.497635 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:48:59.546867 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:59.546843 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pfg7x"] Oct 30 13:48:59.548284 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:59.548242 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda694c74_3bce_4d95_bd30_12a1891ad5e6.slice/crio-cfc13eb678335494d36a1a3fa9279feb6f575cd1f700923e87882929f9cb1f5c WatchSource:0}: Error finding container cfc13eb678335494d36a1a3fa9279feb6f575cd1f700923e87882929f9cb1f5c: Status 404 returned error can't find the container with id cfc13eb678335494d36a1a3fa9279feb6f575cd1f700923e87882929f9cb1f5c Oct 30 13:48:59.560287 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:59.560251 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf707b282_e507_475d_a82a_923c9bd1ff01.slice/crio-8e885dc12b5153c5ae65bf07e8743fca2344b10e63e599ddfae5985778e33b1b WatchSource:0}: Error finding container 8e885dc12b5153c5ae65bf07e8743fca2344b10e63e599ddfae5985778e33b1b: Status 404 returned error can't find the container with id 8e885dc12b5153c5ae65bf07e8743fca2344b10e63e599ddfae5985778e33b1b Oct 30 13:48:59.560612 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:59.560594 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6bc4d9765d-dp7xb"] Oct 30 13:48:59.564682 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:59.564661 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-87f7c8d8c-mdvsm"] Oct 30 13:48:59.564867 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:48:59.564851 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode16ea4af_ce12_442b_91df_092457a7e192.slice/crio-45eb573d8df770fe31bcfc61dd7653472f085c3bb13d8f14ad02c5ee0f6fff53 WatchSource:0}: Error finding container 45eb573d8df770fe31bcfc61dd7653472f085c3bb13d8f14ad02c5ee0f6fff53: Status 404 returned error can't find the container with id 45eb573d8df770fe31bcfc61dd7653472f085c3bb13d8f14ad02c5ee0f6fff53 Oct 30 13:48:59.649348 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:48:59.649313 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:00.107123 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:00.107090 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:00.143117 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:00.143091 4982 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-ntlqq" Oct 30 13:49:00.143500 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:00.143464 4982 scope.go:122] "RemoveContainer" containerID="34d4502189edf1503539ba2cbc4804cc8f474fe80abf87de7b0259d75495aacf" Oct 30 13:49:00.143717 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:49:00.143697 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-ntlqq_openshift-console-operator(dc967875-1546-40bb-8376-fde09416c204)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-ntlqq" podUID="dc967875-1546-40bb-8376-fde09416c204" Oct 30 13:49:00.513074 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:00.513003 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pfg7x" event={"ID":"da694c74-3bce-4d95-bd30-12a1891ad5e6","Type":"ContainerStarted","Data":"cfc13eb678335494d36a1a3fa9279feb6f575cd1f700923e87882929f9cb1f5c"} Oct 30 13:49:00.514359 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:00.514329 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6bc4d9765d-dp7xb" event={"ID":"f707b282-e507-475d-a82a-923c9bd1ff01","Type":"ContainerStarted","Data":"5d3d70bed58f65f77da5f578f9bf9538e3d2dff7e46bf2ef4cfbf9fc298d1ad3"} Oct 30 13:49:00.514470 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:00.514374 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6bc4d9765d-dp7xb" event={"ID":"f707b282-e507-475d-a82a-923c9bd1ff01","Type":"ContainerStarted","Data":"8e885dc12b5153c5ae65bf07e8743fca2344b10e63e599ddfae5985778e33b1b"} Oct 30 13:49:00.514527 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:00.514495 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6bc4d9765d-dp7xb" Oct 30 13:49:00.515806 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:00.515782 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-87f7c8d8c-mdvsm" event={"ID":"e16ea4af-ce12-442b-91df-092457a7e192","Type":"ContainerStarted","Data":"8d94a200c729d6c50f0f023b5beb8c1bee09c9b6218464967652d63185b89f16"} Oct 30 13:49:00.515897 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:00.515821 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-87f7c8d8c-mdvsm" event={"ID":"e16ea4af-ce12-442b-91df-092457a7e192","Type":"ContainerStarted","Data":"45eb573d8df770fe31bcfc61dd7653472f085c3bb13d8f14ad02c5ee0f6fff53"} Oct 30 13:49:00.704021 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:00.704000 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:00.902191 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:00.902160 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:01.320547 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:01.320519 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:01.503462 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:01.503434 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:01.523935 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:01.523911 4982 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 13:49:01.523935 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:01.523923 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pfg7x" event={"ID":"da694c74-3bce-4d95-bd30-12a1891ad5e6","Type":"ContainerStarted","Data":"1fd7f3295f8b7df2d3014f2f0b97ae91e00d1fac637235eae6d2f6227b20337b"} Oct 30 13:49:01.524278 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:01.523968 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pfg7x" event={"ID":"da694c74-3bce-4d95-bd30-12a1891ad5e6","Type":"ContainerStarted","Data":"1bf76497dd4a0a14acd1d78da51eee13396669232370dcd6ee7d91c4476ef663"} Oct 30 13:49:01.608186 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:01.608118 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-dx94q" Oct 30 13:49:01.703430 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:01.703408 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:02.392321 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:02.392292 4982 scope.go:122] "RemoveContainer" containerID="34d4502189edf1503539ba2cbc4804cc8f474fe80abf87de7b0259d75495aacf" Oct 30 13:49:02.530551 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:02.530524 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ntlqq_dc967875-1546-40bb-8376-fde09416c204/console-operator/2.log" Oct 30 13:49:02.530952 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:02.530635 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-ntlqq" event={"ID":"dc967875-1546-40bb-8376-fde09416c204","Type":"ContainerStarted","Data":"02b248a92d66b27ed2466a63fa4e21e62435da6732c337d834344f8e0379349d"} Oct 30 13:49:03.066407 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:03.066373 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:03.219804 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:03.219773 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:03.365813 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:03.365736 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:03.503386 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:03.503357 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:03.665107 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:03.665018 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:04.497669 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:04.497631 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7lx4t" Oct 30 13:49:05.186028 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:05.185999 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-b622r" Oct 30 13:49:05.431631 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:05.431604 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bhxjf" Oct 30 13:49:05.890852 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:05.890826 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:06.046713 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:06.046688 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:06.423416 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:06.423389 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6bc4d9765d-dp7xb"] Oct 30 13:49:06.504211 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:06.504183 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:06.701076 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:06.701011 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:06.845011 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:06.844983 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:07.260432 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:07.260398 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6b536198-f804-4868-beb9-123229c3ae51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-87kd9\" (UID: \"6b536198-f804-4868-beb9-123229c3ae51\") " pod="openshift-insights/insights-runtime-extractor-87kd9" Oct 30 13:49:07.262738 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:07.262720 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6b536198-f804-4868-beb9-123229c3ae51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-87kd9\" (UID: \"6b536198-f804-4868-beb9-123229c3ae51\") " pod="openshift-insights/insights-runtime-extractor-87kd9" Oct 30 13:49:07.327843 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:07.327816 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-87kd9" Oct 30 13:49:07.626037 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:07.625985 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-87kd9"] Oct 30 13:49:07.628176 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:49:07.628151 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b536198_f804_4868_beb9_123229c3ae51.slice/crio-1911035927b1583b293b36f1964ad5ca3a49799854fe20c39afc9dae2d7affe4 WatchSource:0}: Error finding container 1911035927b1583b293b36f1964ad5ca3a49799854fe20c39afc9dae2d7affe4: Status 404 returned error can't find the container with id 1911035927b1583b293b36f1964ad5ca3a49799854fe20c39afc9dae2d7affe4 Oct 30 13:49:08.073828 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:08.073798 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-87f7c8d8c-mdvsm" Oct 30 13:49:08.555593 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:08.555561 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-87kd9" event={"ID":"6b536198-f804-4868-beb9-123229c3ae51","Type":"ContainerStarted","Data":"bb9dbee030e7b12843202da80d010e4f3a17627b13ffdcbe4594e6db1062d11c"} Oct 30 13:49:08.555731 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:08.555610 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-87kd9" event={"ID":"6b536198-f804-4868-beb9-123229c3ae51","Type":"ContainerStarted","Data":"38a9a149bab94317600e3bdd7f50735f242dbb005a6ee25e8880bacb38ae41e6"} Oct 30 13:49:08.555731 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:08.555630 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-87kd9" event={"ID":"6b536198-f804-4868-beb9-123229c3ae51","Type":"ContainerStarted","Data":"1911035927b1583b293b36f1964ad5ca3a49799854fe20c39afc9dae2d7affe4"} Oct 30 13:49:09.333443 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:09.333411 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:09.476216 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:09.476181 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:09.626276 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:09.626251 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:10.564800 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:10.564766 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-87kd9" event={"ID":"6b536198-f804-4868-beb9-123229c3ae51","Type":"ContainerStarted","Data":"d2ed035bdf4929c6327bdf1936e9ae2a5c5268698832b6a4f83c877e1a3816a9"} Oct 30 13:49:12.318994 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:12.318960 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:12.531416 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:12.531382 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-ntlqq" Oct 30 13:49:12.535166 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:12.535147 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-ntlqq" Oct 30 13:49:13.430510 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:13.430476 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:13.581248 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:13.581216 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:13.752297 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:13.752227 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:13.954531 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:13.954495 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:14.014438 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:14.014355 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8qgb5"] Oct 30 13:49:14.048228 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:14.048199 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8qgb5" Oct 30 13:49:14.103715 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:14.103665 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 13:49:14.105691 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:14.105671 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hdbtp\"" Oct 30 13:49:14.105758 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:14.105671 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 13:49:14.106906 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:14.106887 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/518f84a1-ae8d-4fcd-8c5b-1eeb820fac19-node-exporter-wtmp\") pod \"node-exporter-8qgb5\" (UID: \"518f84a1-ae8d-4fcd-8c5b-1eeb820fac19\") " pod="openshift-monitoring/node-exporter-8qgb5" Oct 30 13:49:14.106956 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:14.106918 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnkc5\" (UniqueName: \"kubernetes.io/projected/518f84a1-ae8d-4fcd-8c5b-1eeb820fac19-kube-api-access-lnkc5\") pod \"node-exporter-8qgb5\" (UID: \"518f84a1-ae8d-4fcd-8c5b-1eeb820fac19\") " pod="openshift-monitoring/node-exporter-8qgb5" Oct 30 13:49:14.106956 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:14.106936 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/518f84a1-ae8d-4fcd-8c5b-1eeb820fac19-root\") pod \"node-exporter-8qgb5\" (UID: \"518f84a1-ae8d-4fcd-8c5b-1eeb820fac19\") " pod="openshift-monitoring/node-exporter-8qgb5" Oct 30 13:49:14.107039 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:14.107025 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/518f84a1-ae8d-4fcd-8c5b-1eeb820fac19-sys\") pod \"node-exporter-8qgb5\" (UID: \"518f84a1-ae8d-4fcd-8c5b-1eeb820fac19\") " pod="openshift-monitoring/node-exporter-8qgb5" Oct 30 13:49:14.107095 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:14.107080 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/518f84a1-ae8d-4fcd-8c5b-1eeb820fac19-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8qgb5\" (UID: \"518f84a1-ae8d-4fcd-8c5b-1eeb820fac19\") " pod="openshift-monitoring/node-exporter-8qgb5" Oct 30 13:49:14.107155 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:14.107122 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/518f84a1-ae8d-4fcd-8c5b-1eeb820fac19-node-exporter-textfile\") pod \"node-exporter-8qgb5\" (UID: \"518f84a1-ae8d-4fcd-8c5b-1eeb820fac19\") " pod="openshift-monitoring/node-exporter-8qgb5" Oct 30 13:49:14.107206 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:14.107192 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/518f84a1-ae8d-4fcd-8c5b-1eeb820fac19-metrics-client-ca\") pod \"node-exporter-8qgb5\" (UID: \"518f84a1-ae8d-4fcd-8c5b-1eeb820fac19\") " pod="openshift-monitoring/node-exporter-8qgb5" Oct 30 13:49:14.107240 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:14.107215 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/518f84a1-ae8d-4fcd-8c5b-1eeb820fac19-node-exporter-tls\") pod \"node-exporter-8qgb5\" (UID: \"518f84a1-ae8d-4fcd-8c5b-1eeb820fac19\") " pod="openshift-monitoring/node-exporter-8qgb5" Oct 30 13:49:14.107280 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:14.107267 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/518f84a1-ae8d-4fcd-8c5b-1eeb820fac19-node-exporter-accelerators-collector-config\") pod \"node-exporter-8qgb5\" (UID: \"518f84a1-ae8d-4fcd-8c5b-1eeb820fac19\") " pod="openshift-monitoring/node-exporter-8qgb5" Oct 30 13:49:14.114978 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:14.114959 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 13:49:14.115098 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:14.115069 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 13:49:14.132077 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:14.132056 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:14.208227 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:14.208201 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/518f84a1-ae8d-4fcd-8c5b-1eeb820fac19-sys\") pod \"node-exporter-8qgb5\" (UID: \"518f84a1-ae8d-4fcd-8c5b-1eeb820fac19\") " pod="openshift-monitoring/node-exporter-8qgb5" Oct 30 13:49:14.208332 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:14.208237 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/518f84a1-ae8d-4fcd-8c5b-1eeb820fac19-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8qgb5\" (UID: \"518f84a1-ae8d-4fcd-8c5b-1eeb820fac19\") " pod="openshift-monitoring/node-exporter-8qgb5" Oct 30 13:49:14.208332 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:14.208304 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/518f84a1-ae8d-4fcd-8c5b-1eeb820fac19-sys\") pod \"node-exporter-8qgb5\" (UID: \"518f84a1-ae8d-4fcd-8c5b-1eeb820fac19\") " pod="openshift-monitoring/node-exporter-8qgb5" Oct 30 13:49:14.208448 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:14.208357 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/518f84a1-ae8d-4fcd-8c5b-1eeb820fac19-node-exporter-textfile\") pod \"node-exporter-8qgb5\" (UID: \"518f84a1-ae8d-4fcd-8c5b-1eeb820fac19\") " pod="openshift-monitoring/node-exporter-8qgb5" Oct 30 13:49:14.208448 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:14.208400 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/518f84a1-ae8d-4fcd-8c5b-1eeb820fac19-metrics-client-ca\") pod \"node-exporter-8qgb5\" (UID: \"518f84a1-ae8d-4fcd-8c5b-1eeb820fac19\") " pod="openshift-monitoring/node-exporter-8qgb5" Oct 30 13:49:14.208566 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:14.208500 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/518f84a1-ae8d-4fcd-8c5b-1eeb820fac19-node-exporter-tls\") pod \"node-exporter-8qgb5\" (UID: \"518f84a1-ae8d-4fcd-8c5b-1eeb820fac19\") " pod="openshift-monitoring/node-exporter-8qgb5" Oct 30 13:49:14.208566 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:14.208540 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/518f84a1-ae8d-4fcd-8c5b-1eeb820fac19-node-exporter-accelerators-collector-config\") pod \"node-exporter-8qgb5\" (UID: \"518f84a1-ae8d-4fcd-8c5b-1eeb820fac19\") " pod="openshift-monitoring/node-exporter-8qgb5" Oct 30 13:49:14.208701 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:14.208581 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/518f84a1-ae8d-4fcd-8c5b-1eeb820fac19-node-exporter-wtmp\") pod \"node-exporter-8qgb5\" (UID: \"518f84a1-ae8d-4fcd-8c5b-1eeb820fac19\") " pod="openshift-monitoring/node-exporter-8qgb5" Oct 30 13:49:14.208701 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:14.208611 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnkc5\" (UniqueName: \"kubernetes.io/projected/518f84a1-ae8d-4fcd-8c5b-1eeb820fac19-kube-api-access-lnkc5\") pod \"node-exporter-8qgb5\" (UID: \"518f84a1-ae8d-4fcd-8c5b-1eeb820fac19\") " pod="openshift-monitoring/node-exporter-8qgb5" Oct 30 13:49:14.208701 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:14.208652 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/518f84a1-ae8d-4fcd-8c5b-1eeb820fac19-root\") pod \"node-exporter-8qgb5\" (UID: \"518f84a1-ae8d-4fcd-8c5b-1eeb820fac19\") " pod="openshift-monitoring/node-exporter-8qgb5" Oct 30 13:49:14.208858 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:14.208722 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/518f84a1-ae8d-4fcd-8c5b-1eeb820fac19-node-exporter-textfile\") pod \"node-exporter-8qgb5\" (UID: \"518f84a1-ae8d-4fcd-8c5b-1eeb820fac19\") " pod="openshift-monitoring/node-exporter-8qgb5" Oct 30 13:49:14.208858 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:14.208756 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/518f84a1-ae8d-4fcd-8c5b-1eeb820fac19-root\") pod \"node-exporter-8qgb5\" (UID: \"518f84a1-ae8d-4fcd-8c5b-1eeb820fac19\") " pod="openshift-monitoring/node-exporter-8qgb5" Oct 30 13:49:14.208858 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:14.208804 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/518f84a1-ae8d-4fcd-8c5b-1eeb820fac19-node-exporter-wtmp\") pod \"node-exporter-8qgb5\" (UID: \"518f84a1-ae8d-4fcd-8c5b-1eeb820fac19\") " pod="openshift-monitoring/node-exporter-8qgb5" Oct 30 13:49:14.209009 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:14.208985 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/518f84a1-ae8d-4fcd-8c5b-1eeb820fac19-metrics-client-ca\") pod \"node-exporter-8qgb5\" (UID: \"518f84a1-ae8d-4fcd-8c5b-1eeb820fac19\") " pod="openshift-monitoring/node-exporter-8qgb5" Oct 30 13:49:14.209070 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:14.209036 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/518f84a1-ae8d-4fcd-8c5b-1eeb820fac19-node-exporter-accelerators-collector-config\") pod \"node-exporter-8qgb5\" (UID: \"518f84a1-ae8d-4fcd-8c5b-1eeb820fac19\") " pod="openshift-monitoring/node-exporter-8qgb5" Oct 30 13:49:14.210522 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:14.210497 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/518f84a1-ae8d-4fcd-8c5b-1eeb820fac19-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8qgb5\" (UID: \"518f84a1-ae8d-4fcd-8c5b-1eeb820fac19\") " pod="openshift-monitoring/node-exporter-8qgb5" Oct 30 13:49:14.210735 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:14.210715 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/518f84a1-ae8d-4fcd-8c5b-1eeb820fac19-node-exporter-tls\") pod \"node-exporter-8qgb5\" (UID: \"518f84a1-ae8d-4fcd-8c5b-1eeb820fac19\") " pod="openshift-monitoring/node-exporter-8qgb5" Oct 30 13:49:14.234745 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:14.234724 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnkc5\" (UniqueName: \"kubernetes.io/projected/518f84a1-ae8d-4fcd-8c5b-1eeb820fac19-kube-api-access-lnkc5\") pod \"node-exporter-8qgb5\" (UID: \"518f84a1-ae8d-4fcd-8c5b-1eeb820fac19\") " pod="openshift-monitoring/node-exporter-8qgb5" Oct 30 13:49:14.321965 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:14.321937 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:14.398348 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:14.398327 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8qgb5" Oct 30 13:49:14.428266 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:49:14.428246 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod518f84a1_ae8d_4fcd_8c5b_1eeb820fac19.slice/crio-4735100d250ad299612c20556af0ea25c8822405f8f21b4cd16f4b814433583c WatchSource:0}: Error finding container 4735100d250ad299612c20556af0ea25c8822405f8f21b4cd16f4b814433583c: Status 404 returned error can't find the container with id 4735100d250ad299612c20556af0ea25c8822405f8f21b4cd16f4b814433583c Oct 30 13:49:14.544574 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:14.544544 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:14.583018 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:14.582959 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8qgb5" event={"ID":"518f84a1-ae8d-4fcd-8c5b-1eeb820fac19","Type":"ContainerStarted","Data":"4735100d250ad299612c20556af0ea25c8822405f8f21b4cd16f4b814433583c"} Oct 30 13:49:14.688522 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:14.688497 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:15.206129 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:15.206085 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:15.423634 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:15.423599 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:15.674938 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:15.674907 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:16.428373 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:16.428348 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6bc4d9765d-dp7xb" Oct 30 13:49:16.592132 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:16.592099 4982 generic.go:356] "Generic (PLEG): container finished" podID="518f84a1-ae8d-4fcd-8c5b-1eeb820fac19" containerID="fd5bf911cbb14a9c1a87cb87aed202fe4a1f6cf340c85aef075c8724933d143e" exitCode=0 Oct 30 13:49:16.592268 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:16.592138 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8qgb5" event={"ID":"518f84a1-ae8d-4fcd-8c5b-1eeb820fac19","Type":"ContainerDied","Data":"fd5bf911cbb14a9c1a87cb87aed202fe4a1f6cf340c85aef075c8724933d143e"} Oct 30 13:49:17.337358 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:17.337279 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:17.485872 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:17.485840 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:17.597556 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:17.597497 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8qgb5" event={"ID":"518f84a1-ae8d-4fcd-8c5b-1eeb820fac19","Type":"ContainerStarted","Data":"141abcf9d685f8e6341afb38c14958500bc17c5c8a41193bf6acfea0dd8ff00b"} Oct 30 13:49:17.597684 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:17.597569 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8qgb5" event={"ID":"518f84a1-ae8d-4fcd-8c5b-1eeb820fac19","Type":"ContainerStarted","Data":"8934993f4886ebba4dcc2b0253e5259e4d02c35c1b29835ff4d72b15e36f9dbc"} Oct 30 13:49:17.628656 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:17.628618 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:17.671060 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:17.671013 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8qgb5" podStartSLOduration=3.248788497 podStartE2EDuration="4.67100189s" totalImagesPullingTime="1.339308083s" totalInitContainerRuntime="82.90531ms" isStatefulPod=true podCreationTimestamp="2026-10-30 13:49:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:49:17.657567384 +0000 UTC m=+27.401701424" watchObservedRunningTime="2026-10-30 13:49:17.67100189 +0000 UTC m=+27.415135925" Oct 30 13:49:18.078721 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:18.078693 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-87f7c8d8c-mdvsm" Oct 30 13:49:18.459951 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:18.459873 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:18.603537 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:18.603512 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:18.817442 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:18.817413 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:18.962718 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:18.962692 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:20.092353 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:20.092308 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:20.299328 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:20.299299 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:20.446521 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:20.446456 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:21.136515 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:21.136485 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:21.611321 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:21.611292 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-dx94q" Oct 30 13:49:23.015022 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:23.014990 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:24.263543 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:24.263510 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:24.409979 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:24.409950 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:26.327562 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:26.327525 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:26.479273 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:26.479243 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:27.513994 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:27.513965 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:27.669504 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:27.669473 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:28.009579 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:49:28.009545 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 13:49:28.314987 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:28.314951 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-87f7c8d8c-mdvsm"] Oct 30 13:49:28.430393 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:28.430360 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:28.571022 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:28.570964 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:29.722188 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:29.722150 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:29.866909 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:29.866879 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:31.442845 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:31.442801 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6bc4d9765d-dp7xb" podUID="f707b282-e507-475d-a82a-923c9bd1ff01" containerName="registry" containerID="cri-o://5d3d70bed58f65f77da5f578f9bf9538e3d2dff7e46bf2ef4cfbf9fc298d1ad3" gracePeriod=30 Oct 30 13:49:31.656711 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:31.656675 4982 generic.go:356] "Generic (PLEG): container finished" podID="f707b282-e507-475d-a82a-923c9bd1ff01" containerID="5d3d70bed58f65f77da5f578f9bf9538e3d2dff7e46bf2ef4cfbf9fc298d1ad3" exitCode=0 Oct 30 13:49:31.656862 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:31.656741 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6bc4d9765d-dp7xb" event={"ID":"f707b282-e507-475d-a82a-923c9bd1ff01","Type":"ContainerDied","Data":"5d3d70bed58f65f77da5f578f9bf9538e3d2dff7e46bf2ef4cfbf9fc298d1ad3"} Oct 30 13:49:31.694168 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:31.694116 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bc4d9765d-dp7xb" Oct 30 13:49:31.738085 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:31.738056 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f707b282-e507-475d-a82a-923c9bd1ff01-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f707b282-e507-475d-a82a-923c9bd1ff01-installation-pull-secrets\") pod \"f707b282-e507-475d-a82a-923c9bd1ff01\" (UID: \"f707b282-e507-475d-a82a-923c9bd1ff01\") " Oct 30 13:49:31.738192 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:31.738092 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f707b282-e507-475d-a82a-923c9bd1ff01-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f707b282-e507-475d-a82a-923c9bd1ff01-registry-certificates\") pod \"f707b282-e507-475d-a82a-923c9bd1ff01\" (UID: \"f707b282-e507-475d-a82a-923c9bd1ff01\") " Oct 30 13:49:31.738192 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:31.738139 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f707b282-e507-475d-a82a-923c9bd1ff01-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f707b282-e507-475d-a82a-923c9bd1ff01-trusted-ca\") pod \"f707b282-e507-475d-a82a-923c9bd1ff01\" (UID: \"f707b282-e507-475d-a82a-923c9bd1ff01\") " Oct 30 13:49:31.738192 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:31.738167 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f707b282-e507-475d-a82a-923c9bd1ff01-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f707b282-e507-475d-a82a-923c9bd1ff01-image-registry-private-configuration\") pod \"f707b282-e507-475d-a82a-923c9bd1ff01\" (UID: \"f707b282-e507-475d-a82a-923c9bd1ff01\") " Oct 30 13:49:31.738337 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:31.738220 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f707b282-e507-475d-a82a-923c9bd1ff01-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f707b282-e507-475d-a82a-923c9bd1ff01-ca-trust-extracted\") pod \"f707b282-e507-475d-a82a-923c9bd1ff01\" (UID: \"f707b282-e507-475d-a82a-923c9bd1ff01\") " Oct 30 13:49:31.738337 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:31.738248 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f707b282-e507-475d-a82a-923c9bd1ff01-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f707b282-e507-475d-a82a-923c9bd1ff01-bound-sa-token\") pod \"f707b282-e507-475d-a82a-923c9bd1ff01\" (UID: \"f707b282-e507-475d-a82a-923c9bd1ff01\") " Oct 30 13:49:31.738337 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:31.738276 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f707b282-e507-475d-a82a-923c9bd1ff01-registry-tls\" (UniqueName: \"kubernetes.io/projected/f707b282-e507-475d-a82a-923c9bd1ff01-registry-tls\") pod \"f707b282-e507-475d-a82a-923c9bd1ff01\" (UID: \"f707b282-e507-475d-a82a-923c9bd1ff01\") " Oct 30 13:49:31.738337 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:31.738303 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f707b282-e507-475d-a82a-923c9bd1ff01-kube-api-access-vm9mr\" (UniqueName: \"kubernetes.io/projected/f707b282-e507-475d-a82a-923c9bd1ff01-kube-api-access-vm9mr\") pod \"f707b282-e507-475d-a82a-923c9bd1ff01\" (UID: \"f707b282-e507-475d-a82a-923c9bd1ff01\") " Oct 30 13:49:31.738576 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:31.738544 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f707b282-e507-475d-a82a-923c9bd1ff01-registry-certificates" pod "f707b282-e507-475d-a82a-923c9bd1ff01" (UID: "f707b282-e507-475d-a82a-923c9bd1ff01"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:49:31.738835 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:31.738733 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f707b282-e507-475d-a82a-923c9bd1ff01-trusted-ca" pod "f707b282-e507-475d-a82a-923c9bd1ff01" (UID: "f707b282-e507-475d-a82a-923c9bd1ff01"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:49:31.740820 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:31.740781 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f707b282-e507-475d-a82a-923c9bd1ff01-image-registry-private-configuration" pod "f707b282-e507-475d-a82a-923c9bd1ff01" (UID: "f707b282-e507-475d-a82a-923c9bd1ff01"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:49:31.741023 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:31.740999 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f707b282-e507-475d-a82a-923c9bd1ff01-kube-api-access-vm9mr" pod "f707b282-e507-475d-a82a-923c9bd1ff01" (UID: "f707b282-e507-475d-a82a-923c9bd1ff01"). InnerVolumeSpecName "kube-api-access-vm9mr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:49:31.741114 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:31.741034 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f707b282-e507-475d-a82a-923c9bd1ff01-installation-pull-secrets" pod "f707b282-e507-475d-a82a-923c9bd1ff01" (UID: "f707b282-e507-475d-a82a-923c9bd1ff01"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:49:31.741114 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:31.741082 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f707b282-e507-475d-a82a-923c9bd1ff01-registry-tls" pod "f707b282-e507-475d-a82a-923c9bd1ff01" (UID: "f707b282-e507-475d-a82a-923c9bd1ff01"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:49:31.741184 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:31.741122 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f707b282-e507-475d-a82a-923c9bd1ff01-bound-sa-token" pod "f707b282-e507-475d-a82a-923c9bd1ff01" (UID: "f707b282-e507-475d-a82a-923c9bd1ff01"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:49:31.745181 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:31.745157 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f707b282-e507-475d-a82a-923c9bd1ff01-ca-trust-extracted" pod "f707b282-e507-475d-a82a-923c9bd1ff01" (UID: "f707b282-e507-475d-a82a-923c9bd1ff01"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:49:31.839194 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:31.839165 4982 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f707b282-e507-475d-a82a-923c9bd1ff01-registry-certificates\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 13:49:31.839194 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:31.839189 4982 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f707b282-e507-475d-a82a-923c9bd1ff01-trusted-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 13:49:31.839347 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:31.839200 4982 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f707b282-e507-475d-a82a-923c9bd1ff01-image-registry-private-configuration\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 13:49:31.839347 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:31.839209 4982 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f707b282-e507-475d-a82a-923c9bd1ff01-ca-trust-extracted\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 13:49:31.839347 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:31.839218 4982 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f707b282-e507-475d-a82a-923c9bd1ff01-bound-sa-token\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 13:49:31.839347 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:31.839226 4982 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f707b282-e507-475d-a82a-923c9bd1ff01-registry-tls\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 13:49:31.839347 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:31.839233 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vm9mr\" (UniqueName: \"kubernetes.io/projected/f707b282-e507-475d-a82a-923c9bd1ff01-kube-api-access-vm9mr\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 13:49:31.839347 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:31.839241 4982 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f707b282-e507-475d-a82a-923c9bd1ff01-installation-pull-secrets\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 13:49:31.896376 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:31.896348 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:32.049233 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:32.049178 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:32.661775 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:32.661712 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6bc4d9765d-dp7xb" event={"ID":"f707b282-e507-475d-a82a-923c9bd1ff01","Type":"ContainerDied","Data":"8e885dc12b5153c5ae65bf07e8743fca2344b10e63e599ddfae5985778e33b1b"} Oct 30 13:49:32.661775 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:32.661760 4982 scope.go:122] "RemoveContainer" containerID="5d3d70bed58f65f77da5f578f9bf9538e3d2dff7e46bf2ef4cfbf9fc298d1ad3" Oct 30 13:49:32.662211 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:32.661723 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bc4d9765d-dp7xb" Oct 30 13:49:32.740231 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:32.740208 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6bc4d9765d-dp7xb"] Oct 30 13:49:32.753467 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:32.753440 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6bc4d9765d-dp7xb"] Oct 30 13:49:34.396212 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:34.396141 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f707b282-e507-475d-a82a-923c9bd1ff01" path="/var/lib/kubelet/pods/f707b282-e507-475d-a82a-923c9bd1ff01/volumes" Oct 30 13:49:39.839589 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:39.839554 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:40.026573 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:40.026543 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:43.426016 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:43.425980 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:43.572265 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:43.572232 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:44.395267 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:44.395227 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:44.538806 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:44.538776 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:53.333816 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:53.333755 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-87f7c8d8c-mdvsm" podUID="e16ea4af-ce12-442b-91df-092457a7e192" containerName="registry" containerID="cri-o://8d94a200c729d6c50f0f023b5beb8c1bee09c9b6218464967652d63185b89f16" gracePeriod=30 Oct 30 13:49:53.569194 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:53.569172 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-87f7c8d8c-mdvsm" Oct 30 13:49:53.688657 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:53.688575 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e16ea4af-ce12-442b-91df-092457a7e192-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e16ea4af-ce12-442b-91df-092457a7e192-image-registry-private-configuration\") pod \"e16ea4af-ce12-442b-91df-092457a7e192\" (UID: \"e16ea4af-ce12-442b-91df-092457a7e192\") " Oct 30 13:49:53.688657 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:53.688620 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e16ea4af-ce12-442b-91df-092457a7e192-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e16ea4af-ce12-442b-91df-092457a7e192-installation-pull-secrets\") pod \"e16ea4af-ce12-442b-91df-092457a7e192\" (UID: \"e16ea4af-ce12-442b-91df-092457a7e192\") " Oct 30 13:49:53.688844 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:53.688672 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e16ea4af-ce12-442b-91df-092457a7e192-kube-api-access-5v9rm\" (UniqueName: \"kubernetes.io/projected/e16ea4af-ce12-442b-91df-092457a7e192-kube-api-access-5v9rm\") pod \"e16ea4af-ce12-442b-91df-092457a7e192\" (UID: \"e16ea4af-ce12-442b-91df-092457a7e192\") " Oct 30 13:49:53.688844 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:53.688704 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e16ea4af-ce12-442b-91df-092457a7e192-registry-tls\" (UniqueName: \"kubernetes.io/projected/e16ea4af-ce12-442b-91df-092457a7e192-registry-tls\") pod \"e16ea4af-ce12-442b-91df-092457a7e192\" (UID: \"e16ea4af-ce12-442b-91df-092457a7e192\") " Oct 30 13:49:53.688844 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:53.688739 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e16ea4af-ce12-442b-91df-092457a7e192-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e16ea4af-ce12-442b-91df-092457a7e192-registry-certificates\") pod \"e16ea4af-ce12-442b-91df-092457a7e192\" (UID: \"e16ea4af-ce12-442b-91df-092457a7e192\") " Oct 30 13:49:53.688844 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:53.688801 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e16ea4af-ce12-442b-91df-092457a7e192-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e16ea4af-ce12-442b-91df-092457a7e192-ca-trust-extracted\") pod \"e16ea4af-ce12-442b-91df-092457a7e192\" (UID: \"e16ea4af-ce12-442b-91df-092457a7e192\") " Oct 30 13:49:53.688844 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:53.688835 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e16ea4af-ce12-442b-91df-092457a7e192-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e16ea4af-ce12-442b-91df-092457a7e192-trusted-ca\") pod \"e16ea4af-ce12-442b-91df-092457a7e192\" (UID: \"e16ea4af-ce12-442b-91df-092457a7e192\") " Oct 30 13:49:53.689078 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:53.688880 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e16ea4af-ce12-442b-91df-092457a7e192-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e16ea4af-ce12-442b-91df-092457a7e192-bound-sa-token\") pod \"e16ea4af-ce12-442b-91df-092457a7e192\" (UID: \"e16ea4af-ce12-442b-91df-092457a7e192\") " Oct 30 13:49:53.689349 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:53.689301 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e16ea4af-ce12-442b-91df-092457a7e192-trusted-ca" pod "e16ea4af-ce12-442b-91df-092457a7e192" (UID: "e16ea4af-ce12-442b-91df-092457a7e192"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:49:53.689961 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:53.689916 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e16ea4af-ce12-442b-91df-092457a7e192-registry-certificates" pod "e16ea4af-ce12-442b-91df-092457a7e192" (UID: "e16ea4af-ce12-442b-91df-092457a7e192"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:49:53.690940 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:53.690899 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e16ea4af-ce12-442b-91df-092457a7e192-registry-tls" pod "e16ea4af-ce12-442b-91df-092457a7e192" (UID: "e16ea4af-ce12-442b-91df-092457a7e192"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:49:53.691088 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:53.691072 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e16ea4af-ce12-442b-91df-092457a7e192-bound-sa-token" pod "e16ea4af-ce12-442b-91df-092457a7e192" (UID: "e16ea4af-ce12-442b-91df-092457a7e192"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:49:53.691279 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:53.691244 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e16ea4af-ce12-442b-91df-092457a7e192-installation-pull-secrets" pod "e16ea4af-ce12-442b-91df-092457a7e192" (UID: "e16ea4af-ce12-442b-91df-092457a7e192"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:49:53.691335 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:53.691317 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e16ea4af-ce12-442b-91df-092457a7e192-image-registry-private-configuration" pod "e16ea4af-ce12-442b-91df-092457a7e192" (UID: "e16ea4af-ce12-442b-91df-092457a7e192"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:49:53.691403 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:53.691378 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e16ea4af-ce12-442b-91df-092457a7e192-kube-api-access-5v9rm" pod "e16ea4af-ce12-442b-91df-092457a7e192" (UID: "e16ea4af-ce12-442b-91df-092457a7e192"). InnerVolumeSpecName "kube-api-access-5v9rm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:49:53.695768 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:53.695744 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e16ea4af-ce12-442b-91df-092457a7e192-ca-trust-extracted" pod "e16ea4af-ce12-442b-91df-092457a7e192" (UID: "e16ea4af-ce12-442b-91df-092457a7e192"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:49:53.747268 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:53.747244 4982 generic.go:356] "Generic (PLEG): container finished" podID="e16ea4af-ce12-442b-91df-092457a7e192" containerID="8d94a200c729d6c50f0f023b5beb8c1bee09c9b6218464967652d63185b89f16" exitCode=0 Oct 30 13:49:53.747359 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:53.747293 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-87f7c8d8c-mdvsm" Oct 30 13:49:53.747359 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:53.747321 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-87f7c8d8c-mdvsm" event={"ID":"e16ea4af-ce12-442b-91df-092457a7e192","Type":"ContainerDied","Data":"8d94a200c729d6c50f0f023b5beb8c1bee09c9b6218464967652d63185b89f16"} Oct 30 13:49:53.747463 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:53.747359 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-87f7c8d8c-mdvsm" event={"ID":"e16ea4af-ce12-442b-91df-092457a7e192","Type":"ContainerDied","Data":"45eb573d8df770fe31bcfc61dd7653472f085c3bb13d8f14ad02c5ee0f6fff53"} Oct 30 13:49:53.747463 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:53.747378 4982 scope.go:122] "RemoveContainer" containerID="8d94a200c729d6c50f0f023b5beb8c1bee09c9b6218464967652d63185b89f16" Oct 30 13:49:53.754914 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:53.754893 4982 scope.go:122] "RemoveContainer" containerID="8d94a200c729d6c50f0f023b5beb8c1bee09c9b6218464967652d63185b89f16" Oct 30 13:49:53.755187 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:49:53.755166 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d94a200c729d6c50f0f023b5beb8c1bee09c9b6218464967652d63185b89f16\": container with ID starting with 8d94a200c729d6c50f0f023b5beb8c1bee09c9b6218464967652d63185b89f16 not found: ID does not exist" containerID="8d94a200c729d6c50f0f023b5beb8c1bee09c9b6218464967652d63185b89f16" Oct 30 13:49:53.755236 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:53.755194 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d94a200c729d6c50f0f023b5beb8c1bee09c9b6218464967652d63185b89f16"} err="failed to get container status \"8d94a200c729d6c50f0f023b5beb8c1bee09c9b6218464967652d63185b89f16\": rpc error: code = NotFound desc = could not find container \"8d94a200c729d6c50f0f023b5beb8c1bee09c9b6218464967652d63185b89f16\": container with ID starting with 8d94a200c729d6c50f0f023b5beb8c1bee09c9b6218464967652d63185b89f16 not found: ID does not exist" Oct 30 13:49:53.789810 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:53.789790 4982 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e16ea4af-ce12-442b-91df-092457a7e192-image-registry-private-configuration\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 13:49:53.789884 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:53.789812 4982 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e16ea4af-ce12-442b-91df-092457a7e192-installation-pull-secrets\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 13:49:53.789884 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:53.789822 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5v9rm\" (UniqueName: \"kubernetes.io/projected/e16ea4af-ce12-442b-91df-092457a7e192-kube-api-access-5v9rm\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 13:49:53.789884 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:53.789831 4982 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e16ea4af-ce12-442b-91df-092457a7e192-registry-tls\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 13:49:53.789884 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:53.789839 4982 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e16ea4af-ce12-442b-91df-092457a7e192-registry-certificates\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 13:49:53.789884 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:53.789847 4982 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e16ea4af-ce12-442b-91df-092457a7e192-ca-trust-extracted\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 13:49:53.789884 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:53.789855 4982 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e16ea4af-ce12-442b-91df-092457a7e192-trusted-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 13:49:53.789884 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:53.789863 4982 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e16ea4af-ce12-442b-91df-092457a7e192-bound-sa-token\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 13:49:53.804364 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:53.804344 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-87f7c8d8c-mdvsm"] Oct 30 13:49:53.811873 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:53.811836 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-87f7c8d8c-mdvsm"] Oct 30 13:49:54.400112 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:54.400083 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e16ea4af-ce12-442b-91df-092457a7e192" path="/var/lib/kubelet/pods/e16ea4af-ce12-442b-91df-092457a7e192/volumes" Oct 30 13:49:54.752876 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:54.752816 4982 generic.go:356] "Generic (PLEG): container finished" podID="a4bd1bb2-25ae-423c-ba05-08eb742d340e" containerID="1c78b570941379eccb1af6dee987cc38964001897338ac0d0ac628f5ed9e5190" exitCode=0 Oct 30 13:49:54.753005 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:54.752886 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-kcgbg" event={"ID":"a4bd1bb2-25ae-423c-ba05-08eb742d340e","Type":"ContainerDied","Data":"1c78b570941379eccb1af6dee987cc38964001897338ac0d0ac628f5ed9e5190"} Oct 30 13:49:54.753227 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:54.753210 4982 scope.go:122] "RemoveContainer" containerID="1c78b570941379eccb1af6dee987cc38964001897338ac0d0ac628f5ed9e5190" Oct 30 13:49:54.836986 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:54.836956 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:55.172165 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:55.172134 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:55.335557 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:55.335526 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:49:55.759165 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:49:55.759133 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-kcgbg" event={"ID":"a4bd1bb2-25ae-423c-ba05-08eb742d340e","Type":"ContainerStarted","Data":"29360273610cd33b98cdbac34b4caad80d0a7d88f209a09bb9dd8d912a3ed2fd"} Oct 30 13:49:58.009023 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:49:58.008986 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 13:50:00.113329 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:50:00.113301 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-659879fb99-pjvcq_6d73c6c3-044d-4b88-a105-72bd5e4491cd/router/0.log" Oct 30 13:50:00.258845 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:50:00.258816 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xbkzq_6ac8569a-95ae-421e-aee0-edd189e18d11/serve-healthcheck-canary/0.log" Oct 30 13:50:10.213461 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:50:10.213402 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:10.360924 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:50:10.360892 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:11.098894 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:50:11.098832 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:12.061737 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:50:12.061687 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:15.829466 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:50:15.829434 4982 generic.go:356] "Generic (PLEG): container finished" podID="ade75be2-e3bb-4b50-bed0-c9c257739707" containerID="a6b7c298ab2f474b66c97b7a60bd466dc4ef705ba5497913a3a6f7200a9e66a4" exitCode=0 Oct 30 13:50:15.829886 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:50:15.829474 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tn69c" event={"ID":"ade75be2-e3bb-4b50-bed0-c9c257739707","Type":"ContainerDied","Data":"a6b7c298ab2f474b66c97b7a60bd466dc4ef705ba5497913a3a6f7200a9e66a4"} Oct 30 13:50:15.829886 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:50:15.829762 4982 scope.go:122] "RemoveContainer" containerID="a6b7c298ab2f474b66c97b7a60bd466dc4ef705ba5497913a3a6f7200a9e66a4" Oct 30 13:50:16.834620 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:50:16.834589 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tn69c" event={"ID":"ade75be2-e3bb-4b50-bed0-c9c257739707","Type":"ContainerStarted","Data":"3520aabebf4c34b46146244c5a5260fd608e3b30e9059845ab4b7475911c67ae"} Oct 30 13:50:28.008862 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:50:28.008826 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 13:50:39.959890 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:50:39.959855 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:50.225590 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:50:50.225553 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:50.374842 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:50:50.374814 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:50:58.008917 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:50:58.008879 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 13:51:02.986999 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:51:02.986965 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:03.131433 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:51:03.131359 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:03.278927 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:51:03.278856 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:04.950584 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:51:04.950551 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:05.111986 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:51:05.111950 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:13.428620 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:51:13.428583 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:13.567800 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:51:13.567767 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:24.186305 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:51:24.186272 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:24.327689 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:51:24.327654 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:24.701509 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:51:24.701479 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:25.825264 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:51:25.825233 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:28.009414 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:51:28.009379 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 13:51:40.104411 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:51:40.104369 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:55.742351 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:51:55.742318 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:55.910749 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:51:55.910720 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:51:58.009339 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:51:58.009296 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 13:52:07.067953 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:52:07.067917 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:07.210705 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:52:07.210665 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:28.009355 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:52:28.009317 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 13:52:40.205800 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:52:40.205725 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:52:58.009242 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:52:58.009206 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 13:53:13.019694 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:53:13.019659 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:23.741087 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:53:23.741028 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:23.887215 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:53:23.887185 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:28.009822 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:53:28.009781 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 13:53:33.777553 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:53:33.777514 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:40.323858 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:53:40.323825 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:41.154729 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:53:41.154657 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:50.420483 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:53:50.420451 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ntlqq_dc967875-1546-40bb-8376-fde09416c204/console-operator/2.log" Oct 30 13:53:50.421515 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:53:50.421491 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ntlqq_dc967875-1546-40bb-8376-fde09416c204/console-operator/2.log" Oct 30 13:53:50.425443 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:53:50.425426 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhxjf_aca47fe8-7068-4743-b6ab-283ee0874f66/ovn-acl-logging/0.log" Oct 30 13:53:50.426485 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:53:50.426466 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhxjf_aca47fe8-7068-4743-b6ab-283ee0874f66/ovn-acl-logging/0.log" Oct 30 13:53:50.653125 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:53:50.653093 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:51.435596 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:53:51.435553 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:54.051271 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:53:54.051236 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:54.208310 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:53:54.208272 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:55.890782 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:53:55.890749 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:56.084501 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:53:56.084467 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:56.439073 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:53:56.439039 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:56.591895 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:53:56.591866 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:58.010352 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:53:58.010318 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 13:53:58.307767 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:53:58.307740 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:58.871727 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:53:58.871687 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:53:59.300389 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:53:59.300316 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-pvbq6"] Oct 30 13:53:59.300853 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:53:59.300750 4982 cpu_manager.go:414] "containerMap: removing container" podUID="e16ea4af-ce12-442b-91df-092457a7e192" containerName="registry" Oct 30 13:53:59.300853 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:53:59.300768 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e16ea4af-ce12-442b-91df-092457a7e192" containerName="registry" Oct 30 13:53:59.300853 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:53:59.300787 4982 cpu_manager.go:414] "containerMap: removing container" podUID="f707b282-e507-475d-a82a-923c9bd1ff01" containerName="registry" Oct 30 13:53:59.300853 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:53:59.300792 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f707b282-e507-475d-a82a-923c9bd1ff01" containerName="registry" Oct 30 13:53:59.301018 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:53:59.300864 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="e16ea4af-ce12-442b-91df-092457a7e192" containerName="registry" Oct 30 13:53:59.301018 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:53:59.300881 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="f707b282-e507-475d-a82a-923c9bd1ff01" containerName="registry" Oct 30 13:53:59.303721 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:53:59.303705 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-pvbq6" Oct 30 13:53:59.333306 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:53:59.333273 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 13:53:59.333306 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:53:59.333288 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 13:53:59.333466 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:53:59.333289 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-njwc4\"" Oct 30 13:53:59.334186 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:53:59.334169 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-pvbq6"] Oct 30 13:53:59.349001 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:53:59.348973 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d13287fc-d0ba-4231-a5c5-7a0a128e620a-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-pvbq6\" (UID: \"d13287fc-d0ba-4231-a5c5-7a0a128e620a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-pvbq6" Oct 30 13:53:59.349109 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:53:59.349085 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck8w7\" (UniqueName: \"kubernetes.io/projected/d13287fc-d0ba-4231-a5c5-7a0a128e620a-kube-api-access-ck8w7\") pod \"cert-manager-webhook-5dbbbf75cc-pvbq6\" (UID: \"d13287fc-d0ba-4231-a5c5-7a0a128e620a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-pvbq6" Oct 30 13:53:59.449733 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:53:59.449702 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck8w7\" (UniqueName: \"kubernetes.io/projected/d13287fc-d0ba-4231-a5c5-7a0a128e620a-kube-api-access-ck8w7\") pod \"cert-manager-webhook-5dbbbf75cc-pvbq6\" (UID: \"d13287fc-d0ba-4231-a5c5-7a0a128e620a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-pvbq6" Oct 30 13:53:59.449875 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:53:59.449767 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d13287fc-d0ba-4231-a5c5-7a0a128e620a-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-pvbq6\" (UID: \"d13287fc-d0ba-4231-a5c5-7a0a128e620a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-pvbq6" Oct 30 13:53:59.475512 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:53:59.475474 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d13287fc-d0ba-4231-a5c5-7a0a128e620a-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-pvbq6\" (UID: \"d13287fc-d0ba-4231-a5c5-7a0a128e620a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-pvbq6" Oct 30 13:53:59.475667 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:53:59.475585 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck8w7\" (UniqueName: \"kubernetes.io/projected/d13287fc-d0ba-4231-a5c5-7a0a128e620a-kube-api-access-ck8w7\") pod \"cert-manager-webhook-5dbbbf75cc-pvbq6\" (UID: \"d13287fc-d0ba-4231-a5c5-7a0a128e620a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-pvbq6" Oct 30 13:53:59.628547 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:53:59.628511 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-pvbq6" Oct 30 13:53:59.928619 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:53:59.928586 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-pvbq6"] Oct 30 13:53:59.930691 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:53:59.930658 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd13287fc_d0ba_4231_a5c5_7a0a128e620a.slice/crio-27d5b9b46dcd8e4b2cbc723ec1ab7827ad96c95660c1771a2d6f93af17d093fe WatchSource:0}: Error finding container 27d5b9b46dcd8e4b2cbc723ec1ab7827ad96c95660c1771a2d6f93af17d093fe: Status 404 returned error can't find the container with id 27d5b9b46dcd8e4b2cbc723ec1ab7827ad96c95660c1771a2d6f93af17d093fe Oct 30 13:53:59.932562 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:53:59.932545 4982 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 13:54:00.661416 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:54:00.661372 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-pvbq6" event={"ID":"d13287fc-d0ba-4231-a5c5-7a0a128e620a","Type":"ContainerStarted","Data":"27d5b9b46dcd8e4b2cbc723ec1ab7827ad96c95660c1771a2d6f93af17d093fe"} Oct 30 13:54:00.842532 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:54:00.842493 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:00.990157 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:54:00.990074 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:03.109510 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:54:03.109479 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:03.248131 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:54:03.248070 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:03.675508 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:54:03.675478 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-pvbq6" event={"ID":"d13287fc-d0ba-4231-a5c5-7a0a128e620a","Type":"ContainerStarted","Data":"85b9cb55d933aa796113ea0cd89ec4a95604bef20cc06664e5d0e86cd55dfb8d"} Oct 30 13:54:03.675662 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:54:03.675539 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-pvbq6" Oct 30 13:54:03.720365 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:54:03.720319 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-pvbq6" podStartSLOduration=1.899319528 podStartE2EDuration="4.720307584s" totalImagesPullingTime="2.820988056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:53:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:54:03.719898059 +0000 UTC m=+313.464032104" watchObservedRunningTime="2026-10-30 13:54:03.720307584 +0000 UTC m=+313.464441625" Oct 30 13:54:07.443714 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:54:07.443680 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:07.597560 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:54:07.597532 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:07.745069 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:54:07.745012 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:09.680933 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:54:09.680903 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-pvbq6" Oct 30 13:54:22.454840 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:54:22.454807 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:22.604337 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:54:22.604303 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:24.073410 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:54:24.073380 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:28.010169 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:54:28.010135 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 13:54:28.484956 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:54:28.484925 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:28.669025 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:54:28.668996 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-778b67f4c5-w4d65"] Oct 30 13:54:28.672307 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:54:28.672289 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-778b67f4c5-w4d65" Oct 30 13:54:28.689288 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:54:28.689265 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:28.696143 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:54:28.696119 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-778b67f4c5-w4d65"] Oct 30 13:54:28.752421 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:54:28.752375 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/48f0aa88-b277-4ee0-ae7b-f55df681c642-metrics-cert\") pod \"lws-controller-manager-778b67f4c5-w4d65\" (UID: \"48f0aa88-b277-4ee0-ae7b-f55df681c642\") " pod="openshift-lws-operator/lws-controller-manager-778b67f4c5-w4d65" Oct 30 13:54:28.752421 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:54:28.752402 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/48f0aa88-b277-4ee0-ae7b-f55df681c642-manager-config\") pod \"lws-controller-manager-778b67f4c5-w4d65\" (UID: \"48f0aa88-b277-4ee0-ae7b-f55df681c642\") " pod="openshift-lws-operator/lws-controller-manager-778b67f4c5-w4d65" Oct 30 13:54:28.752562 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:54:28.752421 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmvp5\" (UniqueName: \"kubernetes.io/projected/48f0aa88-b277-4ee0-ae7b-f55df681c642-kube-api-access-nmvp5\") pod \"lws-controller-manager-778b67f4c5-w4d65\" (UID: \"48f0aa88-b277-4ee0-ae7b-f55df681c642\") " pod="openshift-lws-operator/lws-controller-manager-778b67f4c5-w4d65" Oct 30 13:54:28.752562 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:54:28.752521 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/48f0aa88-b277-4ee0-ae7b-f55df681c642-cert\") pod \"lws-controller-manager-778b67f4c5-w4d65\" (UID: \"48f0aa88-b277-4ee0-ae7b-f55df681c642\") " pod="openshift-lws-operator/lws-controller-manager-778b67f4c5-w4d65" Oct 30 13:54:28.778181 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:54:28.778158 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-lfdmj\"" Oct 30 13:54:28.778181 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:54:28.778164 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Oct 30 13:54:28.778181 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:54:28.778164 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Oct 30 13:54:28.778709 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:54:28.778692 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Oct 30 13:54:28.778709 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:54:28.778699 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Oct 30 13:54:28.778818 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:54:28.778716 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Oct 30 13:54:28.831437 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:54:28.831414 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:28.853580 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:54:28.853557 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/48f0aa88-b277-4ee0-ae7b-f55df681c642-cert\") pod \"lws-controller-manager-778b67f4c5-w4d65\" (UID: \"48f0aa88-b277-4ee0-ae7b-f55df681c642\") " pod="openshift-lws-operator/lws-controller-manager-778b67f4c5-w4d65" Oct 30 13:54:28.853693 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:54:28.853615 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/48f0aa88-b277-4ee0-ae7b-f55df681c642-metrics-cert\") pod \"lws-controller-manager-778b67f4c5-w4d65\" (UID: \"48f0aa88-b277-4ee0-ae7b-f55df681c642\") " pod="openshift-lws-operator/lws-controller-manager-778b67f4c5-w4d65" Oct 30 13:54:28.853693 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:54:28.853667 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/48f0aa88-b277-4ee0-ae7b-f55df681c642-manager-config\") pod \"lws-controller-manager-778b67f4c5-w4d65\" (UID: \"48f0aa88-b277-4ee0-ae7b-f55df681c642\") " pod="openshift-lws-operator/lws-controller-manager-778b67f4c5-w4d65" Oct 30 13:54:28.853809 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:54:28.853695 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmvp5\" (UniqueName: \"kubernetes.io/projected/48f0aa88-b277-4ee0-ae7b-f55df681c642-kube-api-access-nmvp5\") pod \"lws-controller-manager-778b67f4c5-w4d65\" (UID: \"48f0aa88-b277-4ee0-ae7b-f55df681c642\") " pod="openshift-lws-operator/lws-controller-manager-778b67f4c5-w4d65" Oct 30 13:54:28.854268 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:54:28.854249 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/48f0aa88-b277-4ee0-ae7b-f55df681c642-manager-config\") pod \"lws-controller-manager-778b67f4c5-w4d65\" (UID: \"48f0aa88-b277-4ee0-ae7b-f55df681c642\") " pod="openshift-lws-operator/lws-controller-manager-778b67f4c5-w4d65" Oct 30 13:54:28.855956 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:54:28.855934 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/48f0aa88-b277-4ee0-ae7b-f55df681c642-metrics-cert\") pod \"lws-controller-manager-778b67f4c5-w4d65\" (UID: \"48f0aa88-b277-4ee0-ae7b-f55df681c642\") " pod="openshift-lws-operator/lws-controller-manager-778b67f4c5-w4d65" Oct 30 13:54:28.856035 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:54:28.855978 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/48f0aa88-b277-4ee0-ae7b-f55df681c642-cert\") pod \"lws-controller-manager-778b67f4c5-w4d65\" (UID: \"48f0aa88-b277-4ee0-ae7b-f55df681c642\") " pod="openshift-lws-operator/lws-controller-manager-778b67f4c5-w4d65" Oct 30 13:54:28.878914 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:54:28.878891 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmvp5\" (UniqueName: \"kubernetes.io/projected/48f0aa88-b277-4ee0-ae7b-f55df681c642-kube-api-access-nmvp5\") pod \"lws-controller-manager-778b67f4c5-w4d65\" (UID: \"48f0aa88-b277-4ee0-ae7b-f55df681c642\") " pod="openshift-lws-operator/lws-controller-manager-778b67f4c5-w4d65" Oct 30 13:54:28.983450 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:54:28.983423 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-778b67f4c5-w4d65" Oct 30 13:54:29.279157 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:54:29.279129 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-778b67f4c5-w4d65"] Oct 30 13:54:29.280824 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:54:29.280788 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48f0aa88_b277_4ee0_ae7b_f55df681c642.slice/crio-520baee712ae5313947849bf16199c3d2efef775c102ec4197b85fb2cc1fb8fd WatchSource:0}: Error finding container 520baee712ae5313947849bf16199c3d2efef775c102ec4197b85fb2cc1fb8fd: Status 404 returned error can't find the container with id 520baee712ae5313947849bf16199c3d2efef775c102ec4197b85fb2cc1fb8fd Oct 30 13:54:29.764978 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:54:29.764946 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-778b67f4c5-w4d65" event={"ID":"48f0aa88-b277-4ee0-ae7b-f55df681c642","Type":"ContainerStarted","Data":"520baee712ae5313947849bf16199c3d2efef775c102ec4197b85fb2cc1fb8fd"} Oct 30 13:54:32.777490 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:54:32.777458 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-778b67f4c5-w4d65" event={"ID":"48f0aa88-b277-4ee0-ae7b-f55df681c642","Type":"ContainerStarted","Data":"db39b328b7ffb7114a2a862040af805963f876f87992a69819046df22f576a2b"} Oct 30 13:54:32.777879 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:54:32.777571 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-778b67f4c5-w4d65" Oct 30 13:54:32.822250 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:54:32.822210 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-778b67f4c5-w4d65" podStartSLOduration=2.336013659 podStartE2EDuration="4.822200653s" totalImagesPullingTime="2.486186994s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:54:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:54:32.821541744 +0000 UTC m=+342.565675785" watchObservedRunningTime="2026-10-30 13:54:32.822200653 +0000 UTC m=+342.566334693" Oct 30 13:54:40.417299 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:54:40.417263 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:54:43.784114 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:54:43.784085 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-778b67f4c5-w4d65" Oct 30 13:54:58.009391 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:54:58.009354 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 13:55:12.054052 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:55:12.054014 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:13.027971 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:55:13.027926 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:19.212254 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:55:19.212208 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:19.350969 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:55:19.350940 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:28.009253 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:55:28.009213 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 13:55:29.281959 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:55:29.281926 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:29.423527 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:55:29.423499 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:29.915332 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:55:29.915297 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:30.045465 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:55:30.045429 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-47k9k"] Oct 30 13:55:30.048822 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:55:30.048800 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-47k9k" Oct 30 13:55:30.075911 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:55:30.075886 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cbfe1a62-d260-44c1-9eda-ac16409bb69b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d55fbd5c9-47k9k\" (UID: \"cbfe1a62-d260-44c1-9eda-ac16409bb69b\") " pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-47k9k" Oct 30 13:55:30.076076 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:55:30.076059 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qk9q\" (UniqueName: \"kubernetes.io/projected/cbfe1a62-d260-44c1-9eda-ac16409bb69b-kube-api-access-4qk9q\") pod \"opendatahub-operator-controller-manager-d55fbd5c9-47k9k\" (UID: \"cbfe1a62-d260-44c1-9eda-ac16409bb69b\") " pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-47k9k" Oct 30 13:55:30.076254 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:55:30.076229 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cbfe1a62-d260-44c1-9eda-ac16409bb69b-metrics-cert\") pod \"opendatahub-operator-controller-manager-d55fbd5c9-47k9k\" (UID: \"cbfe1a62-d260-44c1-9eda-ac16409bb69b\") " pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-47k9k" Oct 30 13:55:30.076364 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:55:30.076270 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cbfe1a62-d260-44c1-9eda-ac16409bb69b-tmp\") pod \"opendatahub-operator-controller-manager-d55fbd5c9-47k9k\" (UID: \"cbfe1a62-d260-44c1-9eda-ac16409bb69b\") " pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-47k9k" Oct 30 13:55:30.076364 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:55:30.076293 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cbfe1a62-d260-44c1-9eda-ac16409bb69b-manifests\") pod \"opendatahub-operator-controller-manager-d55fbd5c9-47k9k\" (UID: \"cbfe1a62-d260-44c1-9eda-ac16409bb69b\") " pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-47k9k" Oct 30 13:55:30.076531 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:55:30.076456 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cbfe1a62-d260-44c1-9eda-ac16409bb69b-webhook-cert\") pod \"opendatahub-operator-controller-manager-d55fbd5c9-47k9k\" (UID: \"cbfe1a62-d260-44c1-9eda-ac16409bb69b\") " pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-47k9k" Oct 30 13:55:30.083441 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:55:30.083419 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-47k9k"] Oct 30 13:55:30.149701 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:55:30.149678 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 13:55:30.149701 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:55:30.149686 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 13:55:30.149858 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:55:30.149746 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 13:55:30.149858 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:55:30.149748 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 13:55:30.149858 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:55:30.149787 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-hnlvr\"" Oct 30 13:55:30.149858 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:55:30.149797 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\"" Oct 30 13:55:30.177253 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:55:30.177193 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cbfe1a62-d260-44c1-9eda-ac16409bb69b-tmp\") pod \"opendatahub-operator-controller-manager-d55fbd5c9-47k9k\" (UID: \"cbfe1a62-d260-44c1-9eda-ac16409bb69b\") " pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-47k9k" Oct 30 13:55:30.177253 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:55:30.177232 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cbfe1a62-d260-44c1-9eda-ac16409bb69b-manifests\") pod \"opendatahub-operator-controller-manager-d55fbd5c9-47k9k\" (UID: \"cbfe1a62-d260-44c1-9eda-ac16409bb69b\") " pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-47k9k" Oct 30 13:55:30.177253 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:55:30.177254 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cbfe1a62-d260-44c1-9eda-ac16409bb69b-webhook-cert\") pod \"opendatahub-operator-controller-manager-d55fbd5c9-47k9k\" (UID: \"cbfe1a62-d260-44c1-9eda-ac16409bb69b\") " pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-47k9k" Oct 30 13:55:30.177420 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:55:30.177276 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cbfe1a62-d260-44c1-9eda-ac16409bb69b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d55fbd5c9-47k9k\" (UID: \"cbfe1a62-d260-44c1-9eda-ac16409bb69b\") " pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-47k9k" Oct 30 13:55:30.177420 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:55:30.177306 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qk9q\" (UniqueName: \"kubernetes.io/projected/cbfe1a62-d260-44c1-9eda-ac16409bb69b-kube-api-access-4qk9q\") pod \"opendatahub-operator-controller-manager-d55fbd5c9-47k9k\" (UID: \"cbfe1a62-d260-44c1-9eda-ac16409bb69b\") " pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-47k9k" Oct 30 13:55:30.177420 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:55:30.177355 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cbfe1a62-d260-44c1-9eda-ac16409bb69b-metrics-cert\") pod \"opendatahub-operator-controller-manager-d55fbd5c9-47k9k\" (UID: \"cbfe1a62-d260-44c1-9eda-ac16409bb69b\") " pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-47k9k" Oct 30 13:55:30.177694 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:55:30.177666 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cbfe1a62-d260-44c1-9eda-ac16409bb69b-manifests\") pod \"opendatahub-operator-controller-manager-d55fbd5c9-47k9k\" (UID: \"cbfe1a62-d260-44c1-9eda-ac16409bb69b\") " pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-47k9k" Oct 30 13:55:30.179560 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:55:30.179533 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cbfe1a62-d260-44c1-9eda-ac16409bb69b-tmp\") pod \"opendatahub-operator-controller-manager-d55fbd5c9-47k9k\" (UID: \"cbfe1a62-d260-44c1-9eda-ac16409bb69b\") " pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-47k9k" Oct 30 13:55:30.179777 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:55:30.179757 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cbfe1a62-d260-44c1-9eda-ac16409bb69b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d55fbd5c9-47k9k\" (UID: \"cbfe1a62-d260-44c1-9eda-ac16409bb69b\") " pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-47k9k" Oct 30 13:55:30.179832 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:55:30.179778 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cbfe1a62-d260-44c1-9eda-ac16409bb69b-webhook-cert\") pod \"opendatahub-operator-controller-manager-d55fbd5c9-47k9k\" (UID: \"cbfe1a62-d260-44c1-9eda-ac16409bb69b\") " pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-47k9k" Oct 30 13:55:30.179832 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:55:30.179819 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cbfe1a62-d260-44c1-9eda-ac16409bb69b-metrics-cert\") pod \"opendatahub-operator-controller-manager-d55fbd5c9-47k9k\" (UID: \"cbfe1a62-d260-44c1-9eda-ac16409bb69b\") " pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-47k9k" Oct 30 13:55:30.189791 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:55:30.189761 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:30.203107 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:55:30.203082 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qk9q\" (UniqueName: \"kubernetes.io/projected/cbfe1a62-d260-44c1-9eda-ac16409bb69b-kube-api-access-4qk9q\") pod \"opendatahub-operator-controller-manager-d55fbd5c9-47k9k\" (UID: \"cbfe1a62-d260-44c1-9eda-ac16409bb69b\") " pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-47k9k" Oct 30 13:55:30.332635 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:55:30.332616 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:30.363423 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:55:30.363403 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-47k9k" Oct 30 13:55:30.477162 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:55:30.477080 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:30.728963 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:55:30.728937 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-47k9k"] Oct 30 13:55:30.730177 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:55:30.730145 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbfe1a62_d260_44c1_9eda_ac16409bb69b.slice/crio-26e9b886d611b33660728f2599e7fa2cd9b6ee48fb9a7584ba97e989214291b2 WatchSource:0}: Error finding container 26e9b886d611b33660728f2599e7fa2cd9b6ee48fb9a7584ba97e989214291b2: Status 404 returned error can't find the container with id 26e9b886d611b33660728f2599e7fa2cd9b6ee48fb9a7584ba97e989214291b2 Oct 30 13:55:30.830859 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:55:30.830821 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:30.981526 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:55:30.981436 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:30.992982 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:55:30.992949 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-47k9k" event={"ID":"cbfe1a62-d260-44c1-9eda-ac16409bb69b","Type":"ContainerStarted","Data":"26e9b886d611b33660728f2599e7fa2cd9b6ee48fb9a7584ba97e989214291b2"} Oct 30 13:55:34.007490 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:55:34.007460 4982 generic.go:356] "Generic (PLEG): container finished" podID="cbfe1a62-d260-44c1-9eda-ac16409bb69b" containerID="e1e468232ef17af7c90786d441d3af8806e36bd08cf2568edfdc3078b62abf15" exitCode=0 Oct 30 13:55:34.007788 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:55:34.007526 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-47k9k" event={"ID":"cbfe1a62-d260-44c1-9eda-ac16409bb69b","Type":"ContainerDied","Data":"e1e468232ef17af7c90786d441d3af8806e36bd08cf2568edfdc3078b62abf15"} Oct 30 13:55:35.014581 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:55:35.014538 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-47k9k" event={"ID":"cbfe1a62-d260-44c1-9eda-ac16409bb69b","Type":"ContainerStarted","Data":"c6895655b8224434431d64f73f93a92c3c0d835eef4fd6afd4daf6d7bc9192f9"} Oct 30 13:55:35.015123 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:55:35.014630 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-47k9k" Oct 30 13:55:35.118118 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:55:35.118061 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-47k9k" podStartSLOduration=2.562544216 podStartE2EDuration="6.11804354s" totalImagesPullingTime="3.125185103s" totalInitContainerRuntime="430.314221ms" isStatefulPod=false podCreationTimestamp="2026-10-30 13:55:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:55:35.103596469 +0000 UTC m=+404.847730511" watchObservedRunningTime="2026-10-30 13:55:35.11804354 +0000 UTC m=+404.862177580" Oct 30 13:55:40.530513 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:55:40.530480 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:46.021788 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:55:46.021762 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-47k9k" Oct 30 13:55:49.355800 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:55:49.355759 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:49.503695 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:55:49.503662 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:52.800062 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:55:52.800026 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:53.049865 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:55:53.049829 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:53.200431 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:55:53.200389 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:53.625146 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:55:53.625087 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:54.598083 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:55:54.598052 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:54.758181 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:55:54.758148 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:56.055248 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:55:56.055213 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:55:58.010328 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:55:58.010292 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 13:56:06.748733 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:06.748687 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:06.801439 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:06.801412 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-ct5zj"] Oct 30 13:56:06.804815 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:06.804793 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-ct5zj" Oct 30 13:56:06.837437 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:06.837417 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-9rrgt\"" Oct 30 13:56:06.846477 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:06.846455 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ad1f34b6-efe9-4981-8d30-845e2accbc06-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-ct5zj\" (UID: \"ad1f34b6-efe9-4981-8d30-845e2accbc06\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-ct5zj" Oct 30 13:56:06.846562 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:06.846514 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smkc2\" (UniqueName: \"kubernetes.io/projected/ad1f34b6-efe9-4981-8d30-845e2accbc06-kube-api-access-smkc2\") pod \"kserve-module-controller-manager-5cfb77bf8c-ct5zj\" (UID: \"ad1f34b6-efe9-4981-8d30-845e2accbc06\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-ct5zj" Oct 30 13:56:06.846618 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:06.846603 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad1f34b6-efe9-4981-8d30-845e2accbc06-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-ct5zj\" (UID: \"ad1f34b6-efe9-4981-8d30-845e2accbc06\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-ct5zj" Oct 30 13:56:06.846671 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:06.846630 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ad1f34b6-efe9-4981-8d30-845e2accbc06-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-ct5zj\" (UID: \"ad1f34b6-efe9-4981-8d30-845e2accbc06\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-ct5zj" Oct 30 13:56:06.864559 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:06.864537 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Oct 30 13:56:06.865519 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:06.865504 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-ct5zj"] Oct 30 13:56:06.946967 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:06.946946 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smkc2\" (UniqueName: \"kubernetes.io/projected/ad1f34b6-efe9-4981-8d30-845e2accbc06-kube-api-access-smkc2\") pod \"kserve-module-controller-manager-5cfb77bf8c-ct5zj\" (UID: \"ad1f34b6-efe9-4981-8d30-845e2accbc06\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-ct5zj" Oct 30 13:56:06.947050 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:06.946978 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad1f34b6-efe9-4981-8d30-845e2accbc06-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-ct5zj\" (UID: \"ad1f34b6-efe9-4981-8d30-845e2accbc06\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-ct5zj" Oct 30 13:56:06.947050 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:06.946995 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ad1f34b6-efe9-4981-8d30-845e2accbc06-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-ct5zj\" (UID: \"ad1f34b6-efe9-4981-8d30-845e2accbc06\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-ct5zj" Oct 30 13:56:06.947050 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:06.947029 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ad1f34b6-efe9-4981-8d30-845e2accbc06-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-ct5zj\" (UID: \"ad1f34b6-efe9-4981-8d30-845e2accbc06\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-ct5zj" Oct 30 13:56:06.947378 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:06.947358 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad1f34b6-efe9-4981-8d30-845e2accbc06-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-ct5zj\" (UID: \"ad1f34b6-efe9-4981-8d30-845e2accbc06\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-ct5zj" Oct 30 13:56:06.947420 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:06.947383 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ad1f34b6-efe9-4981-8d30-845e2accbc06-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-ct5zj\" (UID: \"ad1f34b6-efe9-4981-8d30-845e2accbc06\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-ct5zj" Oct 30 13:56:06.947795 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:06.947779 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ad1f34b6-efe9-4981-8d30-845e2accbc06-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-ct5zj\" (UID: \"ad1f34b6-efe9-4981-8d30-845e2accbc06\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-ct5zj" Oct 30 13:56:06.973343 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:06.973322 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smkc2\" (UniqueName: \"kubernetes.io/projected/ad1f34b6-efe9-4981-8d30-845e2accbc06-kube-api-access-smkc2\") pod \"kserve-module-controller-manager-5cfb77bf8c-ct5zj\" (UID: \"ad1f34b6-efe9-4981-8d30-845e2accbc06\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-ct5zj" Oct 30 13:56:06.973573 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:06.973557 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:07.116318 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:07.116296 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-ct5zj" Oct 30 13:56:07.433326 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:07.433300 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-ct5zj"] Oct 30 13:56:07.435143 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:56:07.435109 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad1f34b6_efe9_4981_8d30_845e2accbc06.slice/crio-18ffa1e7a9d12367b348de6093019b8dc197637cf0d2e97f2b6395c46d8ac9f2 WatchSource:0}: Error finding container 18ffa1e7a9d12367b348de6093019b8dc197637cf0d2e97f2b6395c46d8ac9f2: Status 404 returned error can't find the container with id 18ffa1e7a9d12367b348de6093019b8dc197637cf0d2e97f2b6395c46d8ac9f2 Oct 30 13:56:08.149730 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:08.149684 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-ct5zj" event={"ID":"ad1f34b6-efe9-4981-8d30-845e2accbc06","Type":"ContainerStarted","Data":"18ffa1e7a9d12367b348de6093019b8dc197637cf0d2e97f2b6395c46d8ac9f2"} Oct 30 13:56:09.838029 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:09.837994 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:10.004387 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:10.004311 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:10.119686 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:10.118713 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-vb9c4"] Oct 30 13:56:10.122320 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:10.122300 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-vb9c4" Oct 30 13:56:10.143085 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:10.143060 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-mqqsw\"" Oct 30 13:56:10.148068 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:10.148047 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 13:56:10.158251 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:10.158227 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-vb9c4"] Oct 30 13:56:10.161594 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:10.161567 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-ct5zj" event={"ID":"ad1f34b6-efe9-4981-8d30-845e2accbc06","Type":"ContainerStarted","Data":"9afb969e4ff88fb54da1aed21a6c547d0b4549a9b64783d7298c5c66c4ea4ca3"} Oct 30 13:56:10.161698 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:10.161623 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-ct5zj" Oct 30 13:56:10.171143 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:10.171124 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84f3cb27-ca2f-4a9c-8f30-53fe968ec15d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-vb9c4\" (UID: \"84f3cb27-ca2f-4a9c-8f30-53fe968ec15d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-vb9c4" Oct 30 13:56:10.171235 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:10.171155 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84f3cb27-ca2f-4a9c-8f30-53fe968ec15d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-vb9c4\" (UID: \"84f3cb27-ca2f-4a9c-8f30-53fe968ec15d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-vb9c4" Oct 30 13:56:10.171235 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:10.171183 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84f3cb27-ca2f-4a9c-8f30-53fe968ec15d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-vb9c4\" (UID: \"84f3cb27-ca2f-4a9c-8f30-53fe968ec15d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-vb9c4" Oct 30 13:56:10.171321 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:10.171301 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84f3cb27-ca2f-4a9c-8f30-53fe968ec15d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-vb9c4\" (UID: \"84f3cb27-ca2f-4a9c-8f30-53fe968ec15d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-vb9c4" Oct 30 13:56:10.171378 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:10.171331 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9kb5\" (UniqueName: \"kubernetes.io/projected/84f3cb27-ca2f-4a9c-8f30-53fe968ec15d-kube-api-access-v9kb5\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-vb9c4\" (UID: \"84f3cb27-ca2f-4a9c-8f30-53fe968ec15d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-vb9c4" Oct 30 13:56:10.171378 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:10.171351 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84f3cb27-ca2f-4a9c-8f30-53fe968ec15d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-vb9c4\" (UID: \"84f3cb27-ca2f-4a9c-8f30-53fe968ec15d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-vb9c4" Oct 30 13:56:10.171447 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:10.171430 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84f3cb27-ca2f-4a9c-8f30-53fe968ec15d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-vb9c4\" (UID: \"84f3cb27-ca2f-4a9c-8f30-53fe968ec15d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-vb9c4" Oct 30 13:56:10.171481 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:10.171464 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84f3cb27-ca2f-4a9c-8f30-53fe968ec15d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-vb9c4\" (UID: \"84f3cb27-ca2f-4a9c-8f30-53fe968ec15d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-vb9c4" Oct 30 13:56:10.171543 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:10.171527 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84f3cb27-ca2f-4a9c-8f30-53fe968ec15d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-vb9c4\" (UID: \"84f3cb27-ca2f-4a9c-8f30-53fe968ec15d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-vb9c4" Oct 30 13:56:10.222492 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:10.222438 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-ct5zj" podStartSLOduration=2.012041569 podStartE2EDuration="4.22242218s" totalImagesPullingTime="2.210380611s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:56:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:56:10.221114305 +0000 UTC m=+439.965248346" watchObservedRunningTime="2026-10-30 13:56:10.22242218 +0000 UTC m=+439.966556218" Oct 30 13:56:10.231644 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:10.231622 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:10.272935 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:10.272879 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84f3cb27-ca2f-4a9c-8f30-53fe968ec15d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-vb9c4\" (UID: \"84f3cb27-ca2f-4a9c-8f30-53fe968ec15d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-vb9c4" Oct 30 13:56:10.272935 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:10.272907 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9kb5\" (UniqueName: \"kubernetes.io/projected/84f3cb27-ca2f-4a9c-8f30-53fe968ec15d-kube-api-access-v9kb5\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-vb9c4\" (UID: \"84f3cb27-ca2f-4a9c-8f30-53fe968ec15d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-vb9c4" Oct 30 13:56:10.272935 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:10.272927 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84f3cb27-ca2f-4a9c-8f30-53fe968ec15d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-vb9c4\" (UID: \"84f3cb27-ca2f-4a9c-8f30-53fe968ec15d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-vb9c4" Oct 30 13:56:10.273105 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:10.272954 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84f3cb27-ca2f-4a9c-8f30-53fe968ec15d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-vb9c4\" (UID: \"84f3cb27-ca2f-4a9c-8f30-53fe968ec15d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-vb9c4" Oct 30 13:56:10.273105 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:10.273067 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84f3cb27-ca2f-4a9c-8f30-53fe968ec15d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-vb9c4\" (UID: \"84f3cb27-ca2f-4a9c-8f30-53fe968ec15d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-vb9c4" Oct 30 13:56:10.273200 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:10.273100 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84f3cb27-ca2f-4a9c-8f30-53fe968ec15d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-vb9c4\" (UID: \"84f3cb27-ca2f-4a9c-8f30-53fe968ec15d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-vb9c4" Oct 30 13:56:10.273200 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:10.273150 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84f3cb27-ca2f-4a9c-8f30-53fe968ec15d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-vb9c4\" (UID: \"84f3cb27-ca2f-4a9c-8f30-53fe968ec15d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-vb9c4" Oct 30 13:56:10.273200 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:10.273183 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84f3cb27-ca2f-4a9c-8f30-53fe968ec15d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-vb9c4\" (UID: \"84f3cb27-ca2f-4a9c-8f30-53fe968ec15d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-vb9c4" Oct 30 13:56:10.273346 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:10.273243 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84f3cb27-ca2f-4a9c-8f30-53fe968ec15d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-vb9c4\" (UID: \"84f3cb27-ca2f-4a9c-8f30-53fe968ec15d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-vb9c4" Oct 30 13:56:10.273554 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:10.273377 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84f3cb27-ca2f-4a9c-8f30-53fe968ec15d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-vb9c4\" (UID: \"84f3cb27-ca2f-4a9c-8f30-53fe968ec15d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-vb9c4" Oct 30 13:56:10.273554 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:10.273479 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84f3cb27-ca2f-4a9c-8f30-53fe968ec15d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-vb9c4\" (UID: \"84f3cb27-ca2f-4a9c-8f30-53fe968ec15d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-vb9c4" Oct 30 13:56:10.273779 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:10.273756 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84f3cb27-ca2f-4a9c-8f30-53fe968ec15d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-vb9c4\" (UID: \"84f3cb27-ca2f-4a9c-8f30-53fe968ec15d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-vb9c4" Oct 30 13:56:10.273867 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:10.273813 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84f3cb27-ca2f-4a9c-8f30-53fe968ec15d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-vb9c4\" (UID: \"84f3cb27-ca2f-4a9c-8f30-53fe968ec15d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-vb9c4" Oct 30 13:56:10.273867 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:10.273825 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84f3cb27-ca2f-4a9c-8f30-53fe968ec15d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-vb9c4\" (UID: \"84f3cb27-ca2f-4a9c-8f30-53fe968ec15d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-vb9c4" Oct 30 13:56:10.275722 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:10.275700 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84f3cb27-ca2f-4a9c-8f30-53fe968ec15d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-vb9c4\" (UID: \"84f3cb27-ca2f-4a9c-8f30-53fe968ec15d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-vb9c4" Oct 30 13:56:10.275878 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:10.275861 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84f3cb27-ca2f-4a9c-8f30-53fe968ec15d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-vb9c4\" (UID: \"84f3cb27-ca2f-4a9c-8f30-53fe968ec15d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-vb9c4" Oct 30 13:56:10.300809 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:10.300781 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84f3cb27-ca2f-4a9c-8f30-53fe968ec15d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-vb9c4\" (UID: \"84f3cb27-ca2f-4a9c-8f30-53fe968ec15d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-vb9c4" Oct 30 13:56:10.300971 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:10.300954 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9kb5\" (UniqueName: \"kubernetes.io/projected/84f3cb27-ca2f-4a9c-8f30-53fe968ec15d-kube-api-access-v9kb5\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-vb9c4\" (UID: \"84f3cb27-ca2f-4a9c-8f30-53fe968ec15d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-vb9c4" Oct 30 13:56:10.377044 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:10.377019 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:10.435538 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:10.435513 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-vb9c4" Oct 30 13:56:10.582022 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:10.581993 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:10.750063 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:10.750037 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-vb9c4"] Oct 30 13:56:10.752446 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:56:10.752420 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84f3cb27_ca2f_4a9c_8f30_53fe968ec15d.slice/crio-ab1cafd4cdfc2813c94fe4bc50c1c7d97843adcb8f719f796c0d633385e6ad9f WatchSource:0}: Error finding container ab1cafd4cdfc2813c94fe4bc50c1c7d97843adcb8f719f796c0d633385e6ad9f: Status 404 returned error can't find the container with id ab1cafd4cdfc2813c94fe4bc50c1c7d97843adcb8f719f796c0d633385e6ad9f Oct 30 13:56:11.166844 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:11.166805 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-vb9c4" event={"ID":"84f3cb27-ca2f-4a9c-8f30-53fe968ec15d","Type":"ContainerStarted","Data":"ab1cafd4cdfc2813c94fe4bc50c1c7d97843adcb8f719f796c0d633385e6ad9f"} Oct 30 13:56:13.206543 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:13.206510 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 13:56:13.206843 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:13.206578 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 13:56:13.206843 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:13.206608 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 13:56:14.182135 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:14.182099 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-vb9c4" event={"ID":"84f3cb27-ca2f-4a9c-8f30-53fe968ec15d","Type":"ContainerStarted","Data":"1c55a91d5f668efb283ae322aeb35f78cbde2c827818589809c2647c64375cd5"} Oct 30 13:56:14.254588 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:14.254538 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-vb9c4" podStartSLOduration=1.803476117 podStartE2EDuration="4.25452697s" totalImagesPullingTime="2.451050853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:56:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:56:14.25393673 +0000 UTC m=+443.998070769" watchObservedRunningTime="2026-10-30 13:56:14.25452697 +0000 UTC m=+443.998661007" Oct 30 13:56:14.436671 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:14.436578 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-vb9c4" Oct 30 13:56:14.441262 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:14.441237 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-vb9c4" Oct 30 13:56:15.186666 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:15.186614 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-vb9c4" Oct 30 13:56:15.187498 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:15.187477 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-vb9c4" Oct 30 13:56:28.009937 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:56:28.009897 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 13:56:30.204902 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:30.204870 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:31.180136 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:31.180100 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-ct5zj" Oct 30 13:56:40.691605 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:40.691525 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:54.264112 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:54.264080 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:54.461584 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:54.461551 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:54.576559 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:54.576527 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-bxsb8"] Oct 30 13:56:54.590595 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:54.590567 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-bxsb8" Oct 30 13:56:54.608663 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:54.608610 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-bxsb8"] Oct 30 13:56:54.616729 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:54.616698 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1be45a6e-572d-4d76-a68b-08464bac8146-cert\") pod \"kserve-controller-manager-776cc66d47-bxsb8\" (UID: \"1be45a6e-572d-4d76-a68b-08464bac8146\") " pod="opendatahub/kserve-controller-manager-776cc66d47-bxsb8" Oct 30 13:56:54.616872 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:54.616733 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfgbc\" (UniqueName: \"kubernetes.io/projected/1be45a6e-572d-4d76-a68b-08464bac8146-kube-api-access-kfgbc\") pod \"kserve-controller-manager-776cc66d47-bxsb8\" (UID: \"1be45a6e-572d-4d76-a68b-08464bac8146\") " pod="opendatahub/kserve-controller-manager-776cc66d47-bxsb8" Oct 30 13:56:54.620976 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:54.620930 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Oct 30 13:56:54.621341 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:54.621315 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:54.621554 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:54.621532 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-vlpkx\"" Oct 30 13:56:54.621877 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:54.621858 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-xhkbz"] Oct 30 13:56:54.625420 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:54.625401 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xhkbz" Oct 30 13:56:54.649713 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:54.649683 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-xhkbz"] Oct 30 13:56:54.675671 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:54.675629 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-64r4n"] Oct 30 13:56:54.679018 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:54.679002 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-64r4n" Oct 30 13:56:54.683349 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:54.683328 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-mxq74\"" Oct 30 13:56:54.683471 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:54.683382 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Oct 30 13:56:54.712793 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:54.712771 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-64r4n"] Oct 30 13:56:54.717145 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:54.717121 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tlc7\" (UniqueName: \"kubernetes.io/projected/f6c2daab-dae2-4d3f-8eb2-e47f65ccbbfa-kube-api-access-6tlc7\") pod \"llmisvc-controller-manager-869d5dcc5d-xhkbz\" (UID: \"f6c2daab-dae2-4d3f-8eb2-e47f65ccbbfa\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xhkbz" Oct 30 13:56:54.717335 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:54.717314 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpmqp\" (UniqueName: \"kubernetes.io/projected/4a67cbd2-b867-4cf9-9cfd-c35ae950a761-kube-api-access-fpmqp\") pod \"model-serving-api-75bd89fdfc-64r4n\" (UID: \"4a67cbd2-b867-4cf9-9cfd-c35ae950a761\") " pod="opendatahub/model-serving-api-75bd89fdfc-64r4n" Oct 30 13:56:54.717433 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:54.717372 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a67cbd2-b867-4cf9-9cfd-c35ae950a761-tls-certs\") pod \"model-serving-api-75bd89fdfc-64r4n\" (UID: \"4a67cbd2-b867-4cf9-9cfd-c35ae950a761\") " pod="opendatahub/model-serving-api-75bd89fdfc-64r4n" Oct 30 13:56:54.717499 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:54.717478 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f6c2daab-dae2-4d3f-8eb2-e47f65ccbbfa-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-xhkbz\" (UID: \"f6c2daab-dae2-4d3f-8eb2-e47f65ccbbfa\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xhkbz" Oct 30 13:56:54.717543 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:54.717520 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1be45a6e-572d-4d76-a68b-08464bac8146-cert\") pod \"kserve-controller-manager-776cc66d47-bxsb8\" (UID: \"1be45a6e-572d-4d76-a68b-08464bac8146\") " pod="opendatahub/kserve-controller-manager-776cc66d47-bxsb8" Oct 30 13:56:54.717592 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:54.717544 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfgbc\" (UniqueName: \"kubernetes.io/projected/1be45a6e-572d-4d76-a68b-08464bac8146-kube-api-access-kfgbc\") pod \"kserve-controller-manager-776cc66d47-bxsb8\" (UID: \"1be45a6e-572d-4d76-a68b-08464bac8146\") " pod="opendatahub/kserve-controller-manager-776cc66d47-bxsb8" Oct 30 13:56:54.719958 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:54.719936 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1be45a6e-572d-4d76-a68b-08464bac8146-cert\") pod \"kserve-controller-manager-776cc66d47-bxsb8\" (UID: \"1be45a6e-572d-4d76-a68b-08464bac8146\") " pod="opendatahub/kserve-controller-manager-776cc66d47-bxsb8" Oct 30 13:56:54.728799 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:54.728777 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Oct 30 13:56:54.729436 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:54.729416 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-j2lqb\"" Oct 30 13:56:54.746018 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:54.745994 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfgbc\" (UniqueName: \"kubernetes.io/projected/1be45a6e-572d-4d76-a68b-08464bac8146-kube-api-access-kfgbc\") pod \"kserve-controller-manager-776cc66d47-bxsb8\" (UID: \"1be45a6e-572d-4d76-a68b-08464bac8146\") " pod="opendatahub/kserve-controller-manager-776cc66d47-bxsb8" Oct 30 13:56:54.773743 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:54.773720 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:54.818463 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:54.818438 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a67cbd2-b867-4cf9-9cfd-c35ae950a761-tls-certs\") pod \"model-serving-api-75bd89fdfc-64r4n\" (UID: \"4a67cbd2-b867-4cf9-9cfd-c35ae950a761\") " pod="opendatahub/model-serving-api-75bd89fdfc-64r4n" Oct 30 13:56:54.818552 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:54.818476 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f6c2daab-dae2-4d3f-8eb2-e47f65ccbbfa-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-xhkbz\" (UID: \"f6c2daab-dae2-4d3f-8eb2-e47f65ccbbfa\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xhkbz" Oct 30 13:56:54.818552 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:54.818499 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tlc7\" (UniqueName: \"kubernetes.io/projected/f6c2daab-dae2-4d3f-8eb2-e47f65ccbbfa-kube-api-access-6tlc7\") pod \"llmisvc-controller-manager-869d5dcc5d-xhkbz\" (UID: \"f6c2daab-dae2-4d3f-8eb2-e47f65ccbbfa\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xhkbz" Oct 30 13:56:54.818552 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:54.818531 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpmqp\" (UniqueName: \"kubernetes.io/projected/4a67cbd2-b867-4cf9-9cfd-c35ae950a761-kube-api-access-fpmqp\") pod \"model-serving-api-75bd89fdfc-64r4n\" (UID: \"4a67cbd2-b867-4cf9-9cfd-c35ae950a761\") " pod="opendatahub/model-serving-api-75bd89fdfc-64r4n" Oct 30 13:56:54.820954 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:54.820931 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f6c2daab-dae2-4d3f-8eb2-e47f65ccbbfa-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-xhkbz\" (UID: \"f6c2daab-dae2-4d3f-8eb2-e47f65ccbbfa\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xhkbz" Oct 30 13:56:54.821048 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:54.820931 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a67cbd2-b867-4cf9-9cfd-c35ae950a761-tls-certs\") pod \"model-serving-api-75bd89fdfc-64r4n\" (UID: \"4a67cbd2-b867-4cf9-9cfd-c35ae950a761\") " pod="opendatahub/model-serving-api-75bd89fdfc-64r4n" Oct 30 13:56:54.845887 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:54.845834 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tlc7\" (UniqueName: \"kubernetes.io/projected/f6c2daab-dae2-4d3f-8eb2-e47f65ccbbfa-kube-api-access-6tlc7\") pod \"llmisvc-controller-manager-869d5dcc5d-xhkbz\" (UID: \"f6c2daab-dae2-4d3f-8eb2-e47f65ccbbfa\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xhkbz" Oct 30 13:56:54.845887 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:54.845871 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpmqp\" (UniqueName: \"kubernetes.io/projected/4a67cbd2-b867-4cf9-9cfd-c35ae950a761-kube-api-access-fpmqp\") pod \"model-serving-api-75bd89fdfc-64r4n\" (UID: \"4a67cbd2-b867-4cf9-9cfd-c35ae950a761\") " pod="opendatahub/model-serving-api-75bd89fdfc-64r4n" Oct 30 13:56:54.902357 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:54.902332 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-bxsb8" Oct 30 13:56:54.937709 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:54.937677 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xhkbz" Oct 30 13:56:54.991002 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:54.990969 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-64r4n" Oct 30 13:56:55.154294 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:55.154158 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:55.204017 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:55.203996 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-bxsb8"] Oct 30 13:56:55.205877 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:56:55.205854 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1be45a6e_572d_4d76_a68b_08464bac8146.slice/crio-81f7457df53e175ae41a09ab2901c3518e49a289da4d948fbd06cc333485e79c WatchSource:0}: Error finding container 81f7457df53e175ae41a09ab2901c3518e49a289da4d948fbd06cc333485e79c: Status 404 returned error can't find the container with id 81f7457df53e175ae41a09ab2901c3518e49a289da4d948fbd06cc333485e79c Oct 30 13:56:55.245336 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:55.245311 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-xhkbz"] Oct 30 13:56:55.246508 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:56:55.246489 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf6c2daab_dae2_4d3f_8eb2_e47f65ccbbfa.slice/crio-968e28e32ab07eebd2b54f3b3a5f169d5d942713c3e0838549d78ecd01b050f9 WatchSource:0}: Error finding container 968e28e32ab07eebd2b54f3b3a5f169d5d942713c3e0838549d78ecd01b050f9: Status 404 returned error can't find the container with id 968e28e32ab07eebd2b54f3b3a5f169d5d942713c3e0838549d78ecd01b050f9 Oct 30 13:56:55.298520 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:55.298499 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-64r4n"] Oct 30 13:56:55.299706 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:56:55.299684 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a67cbd2_b867_4cf9_9cfd_c35ae950a761.slice/crio-e0566d61e87aee478ca431c7a8792269a1e0ebee7dea0686756ca1e57cea708f WatchSource:0}: Error finding container e0566d61e87aee478ca431c7a8792269a1e0ebee7dea0686756ca1e57cea708f: Status 404 returned error can't find the container with id e0566d61e87aee478ca431c7a8792269a1e0ebee7dea0686756ca1e57cea708f Oct 30 13:56:55.351286 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:55.351260 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-bxsb8" event={"ID":"1be45a6e-572d-4d76-a68b-08464bac8146","Type":"ContainerStarted","Data":"81f7457df53e175ae41a09ab2901c3518e49a289da4d948fbd06cc333485e79c"} Oct 30 13:56:55.352446 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:55.352424 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-64r4n" event={"ID":"4a67cbd2-b867-4cf9-9cfd-c35ae950a761","Type":"ContainerStarted","Data":"e0566d61e87aee478ca431c7a8792269a1e0ebee7dea0686756ca1e57cea708f"} Oct 30 13:56:55.353510 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:55.353480 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xhkbz" event={"ID":"f6c2daab-dae2-4d3f-8eb2-e47f65ccbbfa","Type":"ContainerStarted","Data":"968e28e32ab07eebd2b54f3b3a5f169d5d942713c3e0838549d78ecd01b050f9"} Oct 30 13:56:57.209387 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:57.209232 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:57.426332 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:57.426302 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:57.572386 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:57.572061 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:57.723099 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:57.723031 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:56:58.010490 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:56:58.010406 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 13:56:58.372530 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:58.372495 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-bxsb8" event={"ID":"1be45a6e-572d-4d76-a68b-08464bac8146","Type":"ContainerStarted","Data":"a10f0b800b26e8ae0e7c5ded229a70ebcfb6fae2fcce3caca7ffaee519ea7185"} Oct 30 13:56:58.372958 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:58.372601 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-bxsb8" Oct 30 13:56:58.374168 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:58.374141 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-64r4n" event={"ID":"4a67cbd2-b867-4cf9-9cfd-c35ae950a761","Type":"ContainerStarted","Data":"219f22af394d69efc1f7ac284d1aaea667f16aaa02cfc28f5a319ed9abb882f0"} Oct 30 13:56:58.374272 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:58.374191 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-64r4n" Oct 30 13:56:58.375438 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:58.375412 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xhkbz" event={"ID":"f6c2daab-dae2-4d3f-8eb2-e47f65ccbbfa","Type":"ContainerStarted","Data":"fca32da24b0fe55e94b8803c1ec71edc6abb7646353c115a46f9a9d4a18055e8"} Oct 30 13:56:58.375540 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:58.375514 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xhkbz" Oct 30 13:56:58.418335 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:58.418282 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-bxsb8" podStartSLOduration=2.086637418 podStartE2EDuration="4.418266617s" totalImagesPullingTime="2.331629199s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:56:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:56:58.417142573 +0000 UTC m=+488.161276627" watchObservedRunningTime="2026-10-30 13:56:58.418266617 +0000 UTC m=+488.162400658" Oct 30 13:56:58.463534 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:58.463490 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-64r4n" podStartSLOduration=2.418301797 podStartE2EDuration="4.463479448s" totalImagesPullingTime="2.045177651s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:56:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:56:58.463424747 +0000 UTC m=+488.207558779" watchObservedRunningTime="2026-10-30 13:56:58.463479448 +0000 UTC m=+488.207613468" Oct 30 13:56:58.511609 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:56:58.511576 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xhkbz" podStartSLOduration=2.289428349 podStartE2EDuration="4.511563487s" totalImagesPullingTime="2.222135138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:56:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:56:58.510431173 +0000 UTC m=+488.254565214" watchObservedRunningTime="2026-10-30 13:56:58.511563487 +0000 UTC m=+488.255697580" Oct 30 13:57:09.384262 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:57:09.384233 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-64r4n" Oct 30 13:57:28.010070 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:57:28.010038 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 13:57:29.382306 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:57:29.382272 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xhkbz" Oct 30 13:57:29.386005 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:57:29.385980 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-bxsb8" Oct 30 13:57:40.784425 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:57:40.784386 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:42.671014 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:57:42.670979 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:42.826940 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:57:42.826914 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:57:58.010411 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:57:58.010374 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 13:58:28.008776 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:58:28.008735 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 13:58:31.407769 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:58:31.407738 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:32.205521 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:58:32.205487 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:32.993230 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:58:32.993201 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:36.153943 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:58:36.153901 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:36.300093 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:58:36.300058 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:36.597172 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:58:36.597139 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:37.689696 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:58:37.689663 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:37.856751 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:58:37.856716 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:39.224427 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:58:39.224389 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:39.419487 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:58:39.419454 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:40.898411 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:58:40.898378 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:41.475098 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:58:41.475058 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:42.876748 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:58:42.876718 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:43.956336 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:58:43.956301 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-5m4s9"] Oct 30 13:58:43.959709 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:58:43.959686 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5m4s9" Oct 30 13:58:43.980756 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:58:43.980726 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-5m4s9"] Oct 30 13:58:43.992055 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:58:43.992029 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:43.992344 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:58:43.992327 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 13:58:43.992445 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:58:43.992380 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 13:58:43.992445 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:58:43.992391 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Oct 30 13:58:43.992734 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:58:43.992720 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Oct 30 13:58:43.992795 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:58:43.992751 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-x9mcf\"" Oct 30 13:58:44.049459 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:58:44.049437 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c4c5\" (UniqueName: \"kubernetes.io/projected/303ce1a6-cb22-49ce-b424-be1f64e5f950-kube-api-access-7c4c5\") pod \"kuadrant-console-plugin-66c4fb575d-5m4s9\" (UID: \"303ce1a6-cb22-49ce-b424-be1f64e5f950\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5m4s9" Oct 30 13:58:44.049560 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:58:44.049474 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/303ce1a6-cb22-49ce-b424-be1f64e5f950-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-5m4s9\" (UID: \"303ce1a6-cb22-49ce-b424-be1f64e5f950\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5m4s9" Oct 30 13:58:44.049686 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:58:44.049667 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/303ce1a6-cb22-49ce-b424-be1f64e5f950-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-5m4s9\" (UID: \"303ce1a6-cb22-49ce-b424-be1f64e5f950\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5m4s9" Oct 30 13:58:44.148095 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:58:44.148068 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:44.150150 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:58:44.150130 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c4c5\" (UniqueName: \"kubernetes.io/projected/303ce1a6-cb22-49ce-b424-be1f64e5f950-kube-api-access-7c4c5\") pod \"kuadrant-console-plugin-66c4fb575d-5m4s9\" (UID: \"303ce1a6-cb22-49ce-b424-be1f64e5f950\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5m4s9" Oct 30 13:58:44.150218 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:58:44.150164 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/303ce1a6-cb22-49ce-b424-be1f64e5f950-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-5m4s9\" (UID: \"303ce1a6-cb22-49ce-b424-be1f64e5f950\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5m4s9" Oct 30 13:58:44.150218 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:58:44.150212 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/303ce1a6-cb22-49ce-b424-be1f64e5f950-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-5m4s9\" (UID: \"303ce1a6-cb22-49ce-b424-be1f64e5f950\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5m4s9" Oct 30 13:58:44.150774 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:58:44.150758 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/303ce1a6-cb22-49ce-b424-be1f64e5f950-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-5m4s9\" (UID: \"303ce1a6-cb22-49ce-b424-be1f64e5f950\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5m4s9" Oct 30 13:58:44.152473 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:58:44.152454 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/303ce1a6-cb22-49ce-b424-be1f64e5f950-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-5m4s9\" (UID: \"303ce1a6-cb22-49ce-b424-be1f64e5f950\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5m4s9" Oct 30 13:58:44.176685 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:58:44.176656 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c4c5\" (UniqueName: \"kubernetes.io/projected/303ce1a6-cb22-49ce-b424-be1f64e5f950-kube-api-access-7c4c5\") pod \"kuadrant-console-plugin-66c4fb575d-5m4s9\" (UID: \"303ce1a6-cb22-49ce-b424-be1f64e5f950\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5m4s9" Oct 30 13:58:44.270728 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:58:44.270680 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5m4s9" Oct 30 13:58:44.564499 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:58:44.564475 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-5m4s9"] Oct 30 13:58:44.566187 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:58:44.566158 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod303ce1a6_cb22_49ce_b424_be1f64e5f950.slice/crio-0912c75cbdaa69d8a42cb6bde1ed4cd6fae7b88f6ce3385f2602142cedb0379d WatchSource:0}: Error finding container 0912c75cbdaa69d8a42cb6bde1ed4cd6fae7b88f6ce3385f2602142cedb0379d: Status 404 returned error can't find the container with id 0912c75cbdaa69d8a42cb6bde1ed4cd6fae7b88f6ce3385f2602142cedb0379d Oct 30 13:58:44.818032 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:58:44.817962 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5m4s9" event={"ID":"303ce1a6-cb22-49ce-b424-be1f64e5f950","Type":"ContainerStarted","Data":"0912c75cbdaa69d8a42cb6bde1ed4cd6fae7b88f6ce3385f2602142cedb0379d"} Oct 30 13:58:46.616541 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:58:46.616491 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:46.757775 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:58:46.757739 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:47.014463 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:58:47.014381 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:47.163110 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:58:47.163075 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:47.413173 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:58:47.413138 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:47.558525 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:58:47.558490 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:49.842793 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:58:49.842758 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5m4s9" event={"ID":"303ce1a6-cb22-49ce-b424-be1f64e5f950","Type":"ContainerStarted","Data":"fefdf7610d98e9a7106e90ac32dd17d71ba7feb8093df09fc2afd520ab2c58ce"} Oct 30 13:58:49.885845 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:58:49.885799 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5m4s9" podStartSLOduration=2.286338371 podStartE2EDuration="6.885787173s" totalImagesPullingTime="4.599448802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:58:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:58:49.885168288 +0000 UTC m=+599.629302325" watchObservedRunningTime="2026-10-30 13:58:49.885787173 +0000 UTC m=+599.629921253" Oct 30 13:58:50.448141 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:58:50.448109 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ntlqq_dc967875-1546-40bb-8376-fde09416c204/console-operator/2.log" Oct 30 13:58:50.448342 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:58:50.448312 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ntlqq_dc967875-1546-40bb-8376-fde09416c204/console-operator/2.log" Oct 30 13:58:50.453077 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:58:50.453056 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhxjf_aca47fe8-7068-4743-b6ab-283ee0874f66/ovn-acl-logging/0.log" Oct 30 13:58:50.453496 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:58:50.453478 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhxjf_aca47fe8-7068-4743-b6ab-283ee0874f66/ovn-acl-logging/0.log" Oct 30 13:58:54.623201 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:58:54.623167 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-n5twb"] Oct 30 13:58:54.627805 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:58:54.627782 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-n5twb" Oct 30 13:58:54.660407 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:58:54.660384 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-zdx8r\"" Oct 30 13:58:54.662325 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:58:54.662303 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-n5twb"] Oct 30 13:58:54.737308 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:58:54.737280 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/73dbb672-c893-4ef2-bba1-3b9f92333c82-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-956956bc4-n5twb\" (UID: \"73dbb672-c893-4ef2-bba1-3b9f92333c82\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-n5twb" Oct 30 13:58:54.737431 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:58:54.737329 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfdrz\" (UniqueName: \"kubernetes.io/projected/73dbb672-c893-4ef2-bba1-3b9f92333c82-kube-api-access-jfdrz\") pod \"kuadrant-operator-controller-manager-956956bc4-n5twb\" (UID: \"73dbb672-c893-4ef2-bba1-3b9f92333c82\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-n5twb" Oct 30 13:58:54.837917 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:58:54.837889 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/73dbb672-c893-4ef2-bba1-3b9f92333c82-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-956956bc4-n5twb\" (UID: \"73dbb672-c893-4ef2-bba1-3b9f92333c82\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-n5twb" Oct 30 13:58:54.838021 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:58:54.837939 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfdrz\" (UniqueName: \"kubernetes.io/projected/73dbb672-c893-4ef2-bba1-3b9f92333c82-kube-api-access-jfdrz\") pod \"kuadrant-operator-controller-manager-956956bc4-n5twb\" (UID: \"73dbb672-c893-4ef2-bba1-3b9f92333c82\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-n5twb" Oct 30 13:58:54.838245 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:58:54.838223 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/73dbb672-c893-4ef2-bba1-3b9f92333c82-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-956956bc4-n5twb\" (UID: \"73dbb672-c893-4ef2-bba1-3b9f92333c82\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-n5twb" Oct 30 13:58:54.875318 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:58:54.875264 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfdrz\" (UniqueName: \"kubernetes.io/projected/73dbb672-c893-4ef2-bba1-3b9f92333c82-kube-api-access-jfdrz\") pod \"kuadrant-operator-controller-manager-956956bc4-n5twb\" (UID: \"73dbb672-c893-4ef2-bba1-3b9f92333c82\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-n5twb" Oct 30 13:58:54.943759 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:58:54.943736 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-n5twb" Oct 30 13:58:55.284487 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:58:55.284459 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-n5twb"] Oct 30 13:58:55.285379 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:58:55.285357 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73dbb672_c893_4ef2_bba1_3b9f92333c82.slice/crio-3f84af5bcf61c6e2ebbb35a08acb0737782b11af990f577720e29f517a5c4d65 WatchSource:0}: Error finding container 3f84af5bcf61c6e2ebbb35a08acb0737782b11af990f577720e29f517a5c4d65: Status 404 returned error can't find the container with id 3f84af5bcf61c6e2ebbb35a08acb0737782b11af990f577720e29f517a5c4d65 Oct 30 13:58:55.544737 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:58:55.544636 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:55.753920 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:58:55.753871 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:55.868244 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:58:55.868201 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-n5twb" event={"ID":"73dbb672-c893-4ef2-bba1-3b9f92333c82","Type":"ContainerStarted","Data":"3f84af5bcf61c6e2ebbb35a08acb0737782b11af990f577720e29f517a5c4d65"} Oct 30 13:58:57.890495 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:58:57.890458 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:58:58.010190 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:58:58.010158 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 13:58:58.060848 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:58:58.060817 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:02.629308 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:59:02.629281 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:02.776466 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:59:02.776387 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:02.903530 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:59:02.903494 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-n5twb" event={"ID":"73dbb672-c893-4ef2-bba1-3b9f92333c82","Type":"ContainerStarted","Data":"7bfbe3871c9018eadb26c9425055f0636ce6c996875d073c281ed4b8bbbcfb90"} Oct 30 13:59:02.903705 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:59:02.903667 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-n5twb" Oct 30 13:59:02.978383 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:59:02.978338 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-n5twb" podStartSLOduration=1.755715564 podStartE2EDuration="8.978326614s" totalImagesPullingTime="7.22261105s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:58:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:59:02.964993269 +0000 UTC m=+612.709127311" watchObservedRunningTime="2026-10-30 13:59:02.978326614 +0000 UTC m=+612.722460652" Oct 30 13:59:03.060305 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:59:03.060271 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:03.195805 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:59:03.195778 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:07.844959 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:59:07.844900 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:07.989170 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:59:07.989119 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:08.244654 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:59:08.244572 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:08.394562 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:59:08.394534 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:08.544450 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:59:08.544392 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:13.909421 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:59:13.909383 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-n5twb" Oct 30 13:59:14.959119 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:59:14.959087 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:15.111842 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:59:15.111807 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:16.451219 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:59:16.451176 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-n5twb"] Oct 30 13:59:16.451781 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:59:16.451361 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-n5twb" podUID="73dbb672-c893-4ef2-bba1-3b9f92333c82" containerName="manager" containerID="cri-o://7bfbe3871c9018eadb26c9425055f0636ce6c996875d073c281ed4b8bbbcfb90" gracePeriod=2 Oct 30 13:59:16.458888 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:59:16.458860 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-n5twb"] Oct 30 13:59:16.466261 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:59:16.466230 4982 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-956956bc4-n5twb\" is forbidden: User \"system:node:ip-10-0-140-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-105.ec2.internal' and this object" podUID="73dbb672-c893-4ef2-bba1-3b9f92333c82" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-n5twb" Oct 30 13:59:16.708995 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:59:16.708932 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-n5twb" Oct 30 13:59:16.723686 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:59:16.723656 4982 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-956956bc4-n5twb\" is forbidden: User \"system:node:ip-10-0-140-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-105.ec2.internal' and this object" podUID="73dbb672-c893-4ef2-bba1-3b9f92333c82" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-n5twb" Oct 30 13:59:16.823044 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:59:16.823019 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73dbb672-c893-4ef2-bba1-3b9f92333c82-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/73dbb672-c893-4ef2-bba1-3b9f92333c82-extensions-socket-volume\") pod \"73dbb672-c893-4ef2-bba1-3b9f92333c82\" (UID: \"73dbb672-c893-4ef2-bba1-3b9f92333c82\") " Oct 30 13:59:16.823134 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:59:16.823104 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73dbb672-c893-4ef2-bba1-3b9f92333c82-kube-api-access-jfdrz\" (UniqueName: \"kubernetes.io/projected/73dbb672-c893-4ef2-bba1-3b9f92333c82-kube-api-access-jfdrz\") pod \"73dbb672-c893-4ef2-bba1-3b9f92333c82\" (UID: \"73dbb672-c893-4ef2-bba1-3b9f92333c82\") " Oct 30 13:59:16.823475 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:59:16.823445 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73dbb672-c893-4ef2-bba1-3b9f92333c82-extensions-socket-volume" pod "73dbb672-c893-4ef2-bba1-3b9f92333c82" (UID: "73dbb672-c893-4ef2-bba1-3b9f92333c82"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:59:16.825155 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:59:16.825132 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73dbb672-c893-4ef2-bba1-3b9f92333c82-kube-api-access-jfdrz" pod "73dbb672-c893-4ef2-bba1-3b9f92333c82" (UID: "73dbb672-c893-4ef2-bba1-3b9f92333c82"). InnerVolumeSpecName "kube-api-access-jfdrz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:59:16.881029 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:59:16.881003 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:16.924153 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:59:16.924130 4982 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/73dbb672-c893-4ef2-bba1-3b9f92333c82-extensions-socket-volume\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 13:59:16.924153 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:59:16.924151 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfdrz\" (UniqueName: \"kubernetes.io/projected/73dbb672-c893-4ef2-bba1-3b9f92333c82-kube-api-access-jfdrz\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 13:59:16.962183 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:59:16.962126 4982 generic.go:356] "Generic (PLEG): container finished" podID="73dbb672-c893-4ef2-bba1-3b9f92333c82" containerID="7bfbe3871c9018eadb26c9425055f0636ce6c996875d073c281ed4b8bbbcfb90" exitCode=0 Oct 30 13:59:16.962183 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:59:16.962169 4982 scope.go:122] "RemoveContainer" containerID="7bfbe3871c9018eadb26c9425055f0636ce6c996875d073c281ed4b8bbbcfb90" Oct 30 13:59:16.962308 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:59:16.962168 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-n5twb" Oct 30 13:59:16.970590 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:59:16.970572 4982 scope.go:122] "RemoveContainer" containerID="7bfbe3871c9018eadb26c9425055f0636ce6c996875d073c281ed4b8bbbcfb90" Oct 30 13:59:16.970864 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:59:16.970844 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bfbe3871c9018eadb26c9425055f0636ce6c996875d073c281ed4b8bbbcfb90\": container with ID starting with 7bfbe3871c9018eadb26c9425055f0636ce6c996875d073c281ed4b8bbbcfb90 not found: ID does not exist" containerID="7bfbe3871c9018eadb26c9425055f0636ce6c996875d073c281ed4b8bbbcfb90" Oct 30 13:59:16.970918 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:59:16.970873 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bfbe3871c9018eadb26c9425055f0636ce6c996875d073c281ed4b8bbbcfb90"} err="failed to get container status \"7bfbe3871c9018eadb26c9425055f0636ce6c996875d073c281ed4b8bbbcfb90\": rpc error: code = NotFound desc = could not find container \"7bfbe3871c9018eadb26c9425055f0636ce6c996875d073c281ed4b8bbbcfb90\": container with ID starting with 7bfbe3871c9018eadb26c9425055f0636ce6c996875d073c281ed4b8bbbcfb90 not found: ID does not exist" Oct 30 13:59:16.976813 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:59:16.976792 4982 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-956956bc4-n5twb\" is forbidden: User \"system:node:ip-10-0-140-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-105.ec2.internal' and this object" podUID="73dbb672-c893-4ef2-bba1-3b9f92333c82" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-n5twb" Oct 30 13:59:16.991551 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:59:16.991529 4982 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-956956bc4-n5twb\" is forbidden: User \"system:node:ip-10-0-140-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-105.ec2.internal' and this object" podUID="73dbb672-c893-4ef2-bba1-3b9f92333c82" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-n5twb" Oct 30 13:59:17.033402 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:59:17.033378 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:18.396953 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:59:18.396919 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73dbb672-c893-4ef2-bba1-3b9f92333c82" path="/var/lib/kubelet/pods/73dbb672-c893-4ef2-bba1-3b9f92333c82/volumes" Oct 30 13:59:19.344332 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:59:19.344299 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:19.491490 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:59:19.491462 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:24.050417 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:59:24.050383 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:24.198076 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:59:24.198048 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:26.546274 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:59:26.546244 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:27.696608 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:59:27.696574 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:28.010790 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:59:28.010711 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 13:59:29.175682 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:59:29.175633 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-p5rrv"] Oct 30 13:59:29.176174 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:59:29.176153 4982 cpu_manager.go:414] "containerMap: removing container" podUID="73dbb672-c893-4ef2-bba1-3b9f92333c82" containerName="manager" Oct 30 13:59:29.176262 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:59:29.176177 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73dbb672-c893-4ef2-bba1-3b9f92333c82" containerName="manager" Oct 30 13:59:29.176326 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:59:29.176281 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="73dbb672-c893-4ef2-bba1-3b9f92333c82" containerName="manager" Oct 30 13:59:29.191776 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:59:29.191753 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-p5rrv" Oct 30 13:59:29.194489 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:59:29.194458 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-p5rrv"] Oct 30 13:59:29.207712 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:59:29.207686 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a3210e1a-fd2b-4aea-9e2a-acecc54a7144-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-p5rrv\" (UID: \"a3210e1a-fd2b-4aea-9e2a-acecc54a7144\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-p5rrv" Oct 30 13:59:29.207835 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:59:29.207732 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrltl\" (UniqueName: \"kubernetes.io/projected/a3210e1a-fd2b-4aea-9e2a-acecc54a7144-kube-api-access-zrltl\") pod \"kuadrant-operator-controller-manager-57bb9bfff-p5rrv\" (UID: \"a3210e1a-fd2b-4aea-9e2a-acecc54a7144\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-p5rrv" Oct 30 13:59:29.209459 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:59:29.209438 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-zdx8r\"" Oct 30 13:59:29.308880 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:59:29.308855 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a3210e1a-fd2b-4aea-9e2a-acecc54a7144-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-p5rrv\" (UID: \"a3210e1a-fd2b-4aea-9e2a-acecc54a7144\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-p5rrv" Oct 30 13:59:29.308971 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:59:29.308886 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrltl\" (UniqueName: \"kubernetes.io/projected/a3210e1a-fd2b-4aea-9e2a-acecc54a7144-kube-api-access-zrltl\") pod \"kuadrant-operator-controller-manager-57bb9bfff-p5rrv\" (UID: \"a3210e1a-fd2b-4aea-9e2a-acecc54a7144\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-p5rrv" Oct 30 13:59:29.309183 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:59:29.309163 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a3210e1a-fd2b-4aea-9e2a-acecc54a7144-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-p5rrv\" (UID: \"a3210e1a-fd2b-4aea-9e2a-acecc54a7144\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-p5rrv" Oct 30 13:59:29.334931 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:59:29.334907 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrltl\" (UniqueName: \"kubernetes.io/projected/a3210e1a-fd2b-4aea-9e2a-acecc54a7144-kube-api-access-zrltl\") pod \"kuadrant-operator-controller-manager-57bb9bfff-p5rrv\" (UID: \"a3210e1a-fd2b-4aea-9e2a-acecc54a7144\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-p5rrv" Oct 30 13:59:29.504450 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:59:29.504403 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-p5rrv" Oct 30 13:59:29.830552 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:59:29.830523 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-p5rrv"] Oct 30 13:59:29.831355 ip-10-0-140-105 kubenswrapper[4982]: W1030 13:59:29.831328 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3210e1a_fd2b_4aea_9e2a_acecc54a7144.slice/crio-4c399ca1c32d19b4a249b976af944e419ea267c5e93b679d144e7a6f27320e49 WatchSource:0}: Error finding container 4c399ca1c32d19b4a249b976af944e419ea267c5e93b679d144e7a6f27320e49: Status 404 returned error can't find the container with id 4c399ca1c32d19b4a249b976af944e419ea267c5e93b679d144e7a6f27320e49 Oct 30 13:59:30.019557 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:59:30.019521 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-p5rrv" event={"ID":"a3210e1a-fd2b-4aea-9e2a-acecc54a7144","Type":"ContainerStarted","Data":"475a35cfd1e1a9906890478c8110763ceb6ba89035ebacba244561747235c3ec"} Oct 30 13:59:30.019760 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:59:30.019612 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-p5rrv" event={"ID":"a3210e1a-fd2b-4aea-9e2a-acecc54a7144","Type":"ContainerStarted","Data":"4c399ca1c32d19b4a249b976af944e419ea267c5e93b679d144e7a6f27320e49"} Oct 30 13:59:30.019831 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:59:30.019760 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-p5rrv" Oct 30 13:59:30.069109 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:59:30.069053 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-p5rrv" podStartSLOduration=1.069041408 podStartE2EDuration="1.069041408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:59:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:59:30.068158077 +0000 UTC m=+639.812292116" watchObservedRunningTime="2026-10-30 13:59:30.069041408 +0000 UTC m=+639.813175427" Oct 30 13:59:41.017059 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:59:41.017026 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:41.025565 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:59:41.025541 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-p5rrv" Oct 30 13:59:46.218389 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:59:46.218356 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:46.459998 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:59:46.459965 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:54.139945 ip-10-0-140-105 kubenswrapper[4982]: I1030 13:59:54.139909 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 13:59:58.009582 ip-10-0-140-105 kubenswrapper[4982]: E1030 13:59:58.009544 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:00:01.930701 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:00:01.930660 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:02.075667 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:00:02.075613 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:02.219953 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:00:02.219880 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:02.934396 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:00:02.934362 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:03.187326 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:00:03.187230 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:10.608824 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:00:10.608793 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:10.756201 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:00:10.756172 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:28.009776 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:00:28.009742 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:00:33.659736 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:00:33.659693 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:33.814389 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:00:33.814361 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:34.529911 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:00:34.529875 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:40.185179 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:00:40.185142 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:41.133160 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:00:41.133119 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:45.603042 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:00:45.603009 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:46.605124 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:00:46.605096 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:46.754324 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:00:46.754293 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:46.909413 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:00:46.909347 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:56.578063 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:00:56.578029 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:56.727192 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:00:56.727157 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:56.877435 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:00:56.877373 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:00:58.010284 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:00:58.010240 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:01:00.443590 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:01:00.443557 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:01:00.612516 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:01:00.612494 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:01:04.129326 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:01:04.129289 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:01:04.296718 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:01:04.296690 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:01:28.009371 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:01:28.009334 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:01:33.900693 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:01:33.900594 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:01:34.046197 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:01:34.046167 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:01:35.230567 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:01:35.230536 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:01:35.434255 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:01:35.434225 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:01:35.586222 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:01:35.586192 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:01:35.734416 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:01:35.734394 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:01:35.769090 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:01:35.769064 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-hplkd"] Oct 30 14:01:35.772541 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:01:35.772525 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-hplkd" Oct 30 14:01:35.789748 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:01:35.789724 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:01:35.802343 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:01:35.802322 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:01:35.802687 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:01:35.802580 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-54whp\"" Oct 30 14:01:35.802800 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:01:35.802772 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 14:01:35.802948 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:01:35.802929 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 14:01:35.804375 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:01:35.804355 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-hplkd"] Oct 30 14:01:35.827251 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:01:35.827230 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4jjx\" (UniqueName: \"kubernetes.io/projected/5fe51702-54a7-4f97-8b08-ffdbdd6fc93a-kube-api-access-p4jjx\") pod \"maas-api-5c48877c75-hplkd\" (UID: \"5fe51702-54a7-4f97-8b08-ffdbdd6fc93a\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-hplkd" Oct 30 14:01:35.827347 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:01:35.827265 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fe51702-54a7-4f97-8b08-ffdbdd6fc93a-openshift-service-ca\") pod \"maas-api-5c48877c75-hplkd\" (UID: \"5fe51702-54a7-4f97-8b08-ffdbdd6fc93a\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-hplkd" Oct 30 14:01:35.827347 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:01:35.827297 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fe51702-54a7-4f97-8b08-ffdbdd6fc93a-metrics-certs\") pod \"maas-api-5c48877c75-hplkd\" (UID: \"5fe51702-54a7-4f97-8b08-ffdbdd6fc93a\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-hplkd" Oct 30 14:01:35.827459 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:01:35.827432 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fe51702-54a7-4f97-8b08-ffdbdd6fc93a-maas-api-tls\") pod \"maas-api-5c48877c75-hplkd\" (UID: \"5fe51702-54a7-4f97-8b08-ffdbdd6fc93a\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-hplkd" Oct 30 14:01:35.928000 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:01:35.927948 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4jjx\" (UniqueName: \"kubernetes.io/projected/5fe51702-54a7-4f97-8b08-ffdbdd6fc93a-kube-api-access-p4jjx\") pod \"maas-api-5c48877c75-hplkd\" (UID: \"5fe51702-54a7-4f97-8b08-ffdbdd6fc93a\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-hplkd" Oct 30 14:01:35.928000 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:01:35.927984 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fe51702-54a7-4f97-8b08-ffdbdd6fc93a-openshift-service-ca\") pod \"maas-api-5c48877c75-hplkd\" (UID: \"5fe51702-54a7-4f97-8b08-ffdbdd6fc93a\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-hplkd" Oct 30 14:01:35.928158 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:01:35.928017 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fe51702-54a7-4f97-8b08-ffdbdd6fc93a-metrics-certs\") pod \"maas-api-5c48877c75-hplkd\" (UID: \"5fe51702-54a7-4f97-8b08-ffdbdd6fc93a\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-hplkd" Oct 30 14:01:35.928158 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:01:35.928034 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fe51702-54a7-4f97-8b08-ffdbdd6fc93a-maas-api-tls\") pod \"maas-api-5c48877c75-hplkd\" (UID: \"5fe51702-54a7-4f97-8b08-ffdbdd6fc93a\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-hplkd" Oct 30 14:01:35.928718 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:01:35.928698 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fe51702-54a7-4f97-8b08-ffdbdd6fc93a-openshift-service-ca\") pod \"maas-api-5c48877c75-hplkd\" (UID: \"5fe51702-54a7-4f97-8b08-ffdbdd6fc93a\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-hplkd" Oct 30 14:01:35.930352 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:01:35.930324 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fe51702-54a7-4f97-8b08-ffdbdd6fc93a-maas-api-tls\") pod \"maas-api-5c48877c75-hplkd\" (UID: \"5fe51702-54a7-4f97-8b08-ffdbdd6fc93a\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-hplkd" Oct 30 14:01:35.930440 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:01:35.930424 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fe51702-54a7-4f97-8b08-ffdbdd6fc93a-metrics-certs\") pod \"maas-api-5c48877c75-hplkd\" (UID: \"5fe51702-54a7-4f97-8b08-ffdbdd6fc93a\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-hplkd" Oct 30 14:01:35.964530 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:01:35.964502 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4jjx\" (UniqueName: \"kubernetes.io/projected/5fe51702-54a7-4f97-8b08-ffdbdd6fc93a-kube-api-access-p4jjx\") pod \"maas-api-5c48877c75-hplkd\" (UID: \"5fe51702-54a7-4f97-8b08-ffdbdd6fc93a\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-hplkd" Oct 30 14:01:36.084589 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:01:36.084553 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-hplkd" Oct 30 14:01:36.382681 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:01:36.382654 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-hplkd"] Oct 30 14:01:36.384289 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:01:36.384263 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fe51702_54a7_4f97_8b08_ffdbdd6fc93a.slice/crio-c4deee74f0ff479f705413944ffa71af7a5e49a0cc12645a22e8cda317f73e87 WatchSource:0}: Error finding container c4deee74f0ff479f705413944ffa71af7a5e49a0cc12645a22e8cda317f73e87: Status 404 returned error can't find the container with id c4deee74f0ff479f705413944ffa71af7a5e49a0cc12645a22e8cda317f73e87 Oct 30 14:01:36.385598 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:01:36.385579 4982 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 14:01:36.546900 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:01:36.546868 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-hplkd" event={"ID":"5fe51702-54a7-4f97-8b08-ffdbdd6fc93a","Type":"ContainerStarted","Data":"c4deee74f0ff479f705413944ffa71af7a5e49a0cc12645a22e8cda317f73e87"} Oct 30 14:01:39.564197 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:01:39.564163 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-hplkd" event={"ID":"5fe51702-54a7-4f97-8b08-ffdbdd6fc93a","Type":"ContainerStarted","Data":"53c29989b094344bdcf8a4e7e6d33b3aada96b1bedf8509e40c8215e4f2cdb4e"} Oct 30 14:01:39.564557 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:01:39.564279 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-hplkd" Oct 30 14:01:39.610348 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:01:39.610300 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c48877c75-hplkd" podStartSLOduration=2.237044065 podStartE2EDuration="4.610287797s" totalImagesPullingTime="2.373243732s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:01:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:01:39.609625212 +0000 UTC m=+769.353759252" watchObservedRunningTime="2026-10-30 14:01:39.610287797 +0000 UTC m=+769.354421837" Oct 30 14:01:41.257911 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:01:41.257878 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:01:45.573428 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:01:45.573396 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-hplkd" Oct 30 14:01:46.751245 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:01:46.751209 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-j5x4l"] Oct 30 14:01:46.754983 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:01:46.754959 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-j5x4l" Oct 30 14:01:46.776171 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:01:46.776147 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-j5x4l"] Oct 30 14:01:46.917852 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:01:46.917824 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gxcf\" (UniqueName: \"kubernetes.io/projected/3c2a2ce1-4b10-45db-846b-574da819ac25-kube-api-access-7gxcf\") pod \"llmisvc-controller-manager-6466c554d8-j5x4l\" (UID: \"3c2a2ce1-4b10-45db-846b-574da819ac25\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-j5x4l" Oct 30 14:01:46.917978 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:01:46.917878 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3c2a2ce1-4b10-45db-846b-574da819ac25-cert\") pod \"llmisvc-controller-manager-6466c554d8-j5x4l\" (UID: \"3c2a2ce1-4b10-45db-846b-574da819ac25\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-j5x4l" Oct 30 14:01:47.018804 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:01:47.018736 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gxcf\" (UniqueName: \"kubernetes.io/projected/3c2a2ce1-4b10-45db-846b-574da819ac25-kube-api-access-7gxcf\") pod \"llmisvc-controller-manager-6466c554d8-j5x4l\" (UID: \"3c2a2ce1-4b10-45db-846b-574da819ac25\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-j5x4l" Oct 30 14:01:47.018923 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:01:47.018801 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3c2a2ce1-4b10-45db-846b-574da819ac25-cert\") pod \"llmisvc-controller-manager-6466c554d8-j5x4l\" (UID: \"3c2a2ce1-4b10-45db-846b-574da819ac25\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-j5x4l" Oct 30 14:01:47.021076 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:01:47.021055 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3c2a2ce1-4b10-45db-846b-574da819ac25-cert\") pod \"llmisvc-controller-manager-6466c554d8-j5x4l\" (UID: \"3c2a2ce1-4b10-45db-846b-574da819ac25\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-j5x4l" Oct 30 14:01:47.044881 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:01:47.044856 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gxcf\" (UniqueName: \"kubernetes.io/projected/3c2a2ce1-4b10-45db-846b-574da819ac25-kube-api-access-7gxcf\") pod \"llmisvc-controller-manager-6466c554d8-j5x4l\" (UID: \"3c2a2ce1-4b10-45db-846b-574da819ac25\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-j5x4l" Oct 30 14:01:47.066785 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:01:47.066761 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-j5x4l" Oct 30 14:01:47.362902 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:01:47.362870 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-j5x4l"] Oct 30 14:01:47.363145 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:01:47.363116 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c2a2ce1_4b10_45db_846b_574da819ac25.slice/crio-d71a51798d49fab8d579ef3504199775fa27425510f71cd1681638299bac66e6 WatchSource:0}: Error finding container d71a51798d49fab8d579ef3504199775fa27425510f71cd1681638299bac66e6: Status 404 returned error can't find the container with id d71a51798d49fab8d579ef3504199775fa27425510f71cd1681638299bac66e6 Oct 30 14:01:47.596347 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:01:47.596306 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-j5x4l" event={"ID":"3c2a2ce1-4b10-45db-846b-574da819ac25","Type":"ContainerStarted","Data":"d71a51798d49fab8d579ef3504199775fa27425510f71cd1681638299bac66e6"} Oct 30 14:01:48.601342 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:01:48.601310 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-j5x4l" event={"ID":"3c2a2ce1-4b10-45db-846b-574da819ac25","Type":"ContainerStarted","Data":"1435dcbe32732802faccee57663412cbf65fd5708ef14ea1372ea7ec426e5ed8"} Oct 30 14:01:48.601712 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:01:48.601430 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-j5x4l" Oct 30 14:01:48.649939 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:01:48.649824 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-j5x4l" podStartSLOduration=2.330886053 podStartE2EDuration="2.649810419s" totalImagesPullingTime="318.924366ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:01:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:01:48.648825463 +0000 UTC m=+778.392959504" watchObservedRunningTime="2026-10-30 14:01:48.649810419 +0000 UTC m=+778.393944460" Oct 30 14:01:58.009106 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:01:58.009073 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:02:19.606507 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:02:19.606477 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-j5x4l" Oct 30 14:02:19.682048 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:02:19.682014 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-xhkbz"] Oct 30 14:02:19.682314 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:02:19.682259 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xhkbz" podUID="f6c2daab-dae2-4d3f-8eb2-e47f65ccbbfa" containerName="manager" containerID="cri-o://fca32da24b0fe55e94b8803c1ec71edc6abb7646353c115a46f9a9d4a18055e8" gracePeriod=30 Oct 30 14:02:19.932232 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:02:19.932206 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xhkbz" Oct 30 14:02:20.063792 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:02:20.063763 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6c2daab-dae2-4d3f-8eb2-e47f65ccbbfa-cert\" (UniqueName: \"kubernetes.io/secret/f6c2daab-dae2-4d3f-8eb2-e47f65ccbbfa-cert\") pod \"f6c2daab-dae2-4d3f-8eb2-e47f65ccbbfa\" (UID: \"f6c2daab-dae2-4d3f-8eb2-e47f65ccbbfa\") " Oct 30 14:02:20.063943 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:02:20.063858 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6c2daab-dae2-4d3f-8eb2-e47f65ccbbfa-kube-api-access-6tlc7\" (UniqueName: \"kubernetes.io/projected/f6c2daab-dae2-4d3f-8eb2-e47f65ccbbfa-kube-api-access-6tlc7\") pod \"f6c2daab-dae2-4d3f-8eb2-e47f65ccbbfa\" (UID: \"f6c2daab-dae2-4d3f-8eb2-e47f65ccbbfa\") " Oct 30 14:02:20.066035 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:02:20.066003 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6c2daab-dae2-4d3f-8eb2-e47f65ccbbfa-cert" pod "f6c2daab-dae2-4d3f-8eb2-e47f65ccbbfa" (UID: "f6c2daab-dae2-4d3f-8eb2-e47f65ccbbfa"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:02:20.066148 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:02:20.066068 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6c2daab-dae2-4d3f-8eb2-e47f65ccbbfa-kube-api-access-6tlc7" pod "f6c2daab-dae2-4d3f-8eb2-e47f65ccbbfa" (UID: "f6c2daab-dae2-4d3f-8eb2-e47f65ccbbfa"). InnerVolumeSpecName "kube-api-access-6tlc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:02:20.164859 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:02:20.164798 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tlc7\" (UniqueName: \"kubernetes.io/projected/f6c2daab-dae2-4d3f-8eb2-e47f65ccbbfa-kube-api-access-6tlc7\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:02:20.164859 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:02:20.164822 4982 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f6c2daab-dae2-4d3f-8eb2-e47f65ccbbfa-cert\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:02:20.629907 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:02:20.629873 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:02:20.745339 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:02:20.745305 4982 generic.go:356] "Generic (PLEG): container finished" podID="f6c2daab-dae2-4d3f-8eb2-e47f65ccbbfa" containerID="fca32da24b0fe55e94b8803c1ec71edc6abb7646353c115a46f9a9d4a18055e8" exitCode=0 Oct 30 14:02:20.745471 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:02:20.745348 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xhkbz" Oct 30 14:02:20.745471 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:02:20.745375 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xhkbz" event={"ID":"f6c2daab-dae2-4d3f-8eb2-e47f65ccbbfa","Type":"ContainerDied","Data":"fca32da24b0fe55e94b8803c1ec71edc6abb7646353c115a46f9a9d4a18055e8"} Oct 30 14:02:20.745471 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:02:20.745430 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xhkbz" event={"ID":"f6c2daab-dae2-4d3f-8eb2-e47f65ccbbfa","Type":"ContainerDied","Data":"968e28e32ab07eebd2b54f3b3a5f169d5d942713c3e0838549d78ecd01b050f9"} Oct 30 14:02:20.745471 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:02:20.745453 4982 scope.go:122] "RemoveContainer" containerID="fca32da24b0fe55e94b8803c1ec71edc6abb7646353c115a46f9a9d4a18055e8" Oct 30 14:02:20.753557 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:02:20.753538 4982 scope.go:122] "RemoveContainer" containerID="fca32da24b0fe55e94b8803c1ec71edc6abb7646353c115a46f9a9d4a18055e8" Oct 30 14:02:20.753829 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:02:20.753803 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fca32da24b0fe55e94b8803c1ec71edc6abb7646353c115a46f9a9d4a18055e8\": container with ID starting with fca32da24b0fe55e94b8803c1ec71edc6abb7646353c115a46f9a9d4a18055e8 not found: ID does not exist" containerID="fca32da24b0fe55e94b8803c1ec71edc6abb7646353c115a46f9a9d4a18055e8" Oct 30 14:02:20.753897 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:02:20.753842 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fca32da24b0fe55e94b8803c1ec71edc6abb7646353c115a46f9a9d4a18055e8"} err="failed to get container status \"fca32da24b0fe55e94b8803c1ec71edc6abb7646353c115a46f9a9d4a18055e8\": rpc error: code = NotFound desc = could not find container \"fca32da24b0fe55e94b8803c1ec71edc6abb7646353c115a46f9a9d4a18055e8\": container with ID starting with fca32da24b0fe55e94b8803c1ec71edc6abb7646353c115a46f9a9d4a18055e8 not found: ID does not exist" Oct 30 14:02:20.776625 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:02:20.776591 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:02:20.802152 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:02:20.802131 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-xhkbz"] Oct 30 14:02:20.815542 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:02:20.815519 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-xhkbz"] Oct 30 14:02:22.396551 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:02:22.396518 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6c2daab-dae2-4d3f-8eb2-e47f65ccbbfa" path="/var/lib/kubelet/pods/f6c2daab-dae2-4d3f-8eb2-e47f65ccbbfa/volumes" Oct 30 14:02:28.009134 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:02:28.009102 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:02:37.197741 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:02:37.197674 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:02:40.232400 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:02:40.232335 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:02:41.400322 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:02:41.400266 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:02:41.887245 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:02:41.887206 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:02:43.554880 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:02:43.554814 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:02:45.135300 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:02:45.135268 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:02:46.575010 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:02:46.574963 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:02:48.443870 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:02:48.443835 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:02:58.009762 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:02:58.009726 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:03:02.248031 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:02.247997 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5842c"] Oct 30 14:03:02.248384 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:02.248343 4982 cpu_manager.go:414] "containerMap: removing container" podUID="f6c2daab-dae2-4d3f-8eb2-e47f65ccbbfa" containerName="manager" Oct 30 14:03:02.248384 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:02.248354 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6c2daab-dae2-4d3f-8eb2-e47f65ccbbfa" containerName="manager" Oct 30 14:03:02.248452 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:02.248418 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6c2daab-dae2-4d3f-8eb2-e47f65ccbbfa" containerName="manager" Oct 30 14:03:02.252673 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:02.252654 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5842c" Oct 30 14:03:02.273884 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:02.273854 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-5drtm\"" Oct 30 14:03:02.287781 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:02.287757 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5842c"] Oct 30 14:03:02.398462 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:02.398434 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f56cfd10-4ad6-4c14-b614-ae3b48b87dbe-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-5842c\" (UID: \"f56cfd10-4ad6-4c14-b614-ae3b48b87dbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5842c" Oct 30 14:03:02.398624 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:02.398476 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f56cfd10-4ad6-4c14-b614-ae3b48b87dbe-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-5842c\" (UID: \"f56cfd10-4ad6-4c14-b614-ae3b48b87dbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5842c" Oct 30 14:03:02.398624 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:02.398558 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ssg9\" (UniqueName: \"kubernetes.io/projected/f56cfd10-4ad6-4c14-b614-ae3b48b87dbe-kube-api-access-8ssg9\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-5842c\" (UID: \"f56cfd10-4ad6-4c14-b614-ae3b48b87dbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5842c" Oct 30 14:03:02.398624 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:02.398615 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f56cfd10-4ad6-4c14-b614-ae3b48b87dbe-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-5842c\" (UID: \"f56cfd10-4ad6-4c14-b614-ae3b48b87dbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5842c" Oct 30 14:03:02.398764 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:02.398703 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f56cfd10-4ad6-4c14-b614-ae3b48b87dbe-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-5842c\" (UID: \"f56cfd10-4ad6-4c14-b614-ae3b48b87dbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5842c" Oct 30 14:03:02.398764 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:02.398733 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f56cfd10-4ad6-4c14-b614-ae3b48b87dbe-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-5842c\" (UID: \"f56cfd10-4ad6-4c14-b614-ae3b48b87dbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5842c" Oct 30 14:03:02.398842 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:02.398765 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f56cfd10-4ad6-4c14-b614-ae3b48b87dbe-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-5842c\" (UID: \"f56cfd10-4ad6-4c14-b614-ae3b48b87dbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5842c" Oct 30 14:03:02.398880 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:02.398840 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f56cfd10-4ad6-4c14-b614-ae3b48b87dbe-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-5842c\" (UID: \"f56cfd10-4ad6-4c14-b614-ae3b48b87dbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5842c" Oct 30 14:03:02.398880 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:02.398877 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f56cfd10-4ad6-4c14-b614-ae3b48b87dbe-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-5842c\" (UID: \"f56cfd10-4ad6-4c14-b614-ae3b48b87dbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5842c" Oct 30 14:03:02.500051 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:02.499978 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f56cfd10-4ad6-4c14-b614-ae3b48b87dbe-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-5842c\" (UID: \"f56cfd10-4ad6-4c14-b614-ae3b48b87dbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5842c" Oct 30 14:03:02.500051 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:02.500012 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f56cfd10-4ad6-4c14-b614-ae3b48b87dbe-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-5842c\" (UID: \"f56cfd10-4ad6-4c14-b614-ae3b48b87dbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5842c" Oct 30 14:03:02.500051 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:02.500028 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f56cfd10-4ad6-4c14-b614-ae3b48b87dbe-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-5842c\" (UID: \"f56cfd10-4ad6-4c14-b614-ae3b48b87dbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5842c" Oct 30 14:03:02.500333 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:02.500142 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f56cfd10-4ad6-4c14-b614-ae3b48b87dbe-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-5842c\" (UID: \"f56cfd10-4ad6-4c14-b614-ae3b48b87dbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5842c" Oct 30 14:03:02.500333 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:02.500206 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f56cfd10-4ad6-4c14-b614-ae3b48b87dbe-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-5842c\" (UID: \"f56cfd10-4ad6-4c14-b614-ae3b48b87dbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5842c" Oct 30 14:03:02.500333 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:02.500240 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f56cfd10-4ad6-4c14-b614-ae3b48b87dbe-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-5842c\" (UID: \"f56cfd10-4ad6-4c14-b614-ae3b48b87dbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5842c" Oct 30 14:03:02.500491 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:02.500464 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f56cfd10-4ad6-4c14-b614-ae3b48b87dbe-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-5842c\" (UID: \"f56cfd10-4ad6-4c14-b614-ae3b48b87dbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5842c" Oct 30 14:03:02.500547 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:02.500506 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ssg9\" (UniqueName: \"kubernetes.io/projected/f56cfd10-4ad6-4c14-b614-ae3b48b87dbe-kube-api-access-8ssg9\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-5842c\" (UID: \"f56cfd10-4ad6-4c14-b614-ae3b48b87dbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5842c" Oct 30 14:03:02.500547 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:02.500467 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f56cfd10-4ad6-4c14-b614-ae3b48b87dbe-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-5842c\" (UID: \"f56cfd10-4ad6-4c14-b614-ae3b48b87dbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5842c" Oct 30 14:03:02.500547 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:02.500533 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f56cfd10-4ad6-4c14-b614-ae3b48b87dbe-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-5842c\" (UID: \"f56cfd10-4ad6-4c14-b614-ae3b48b87dbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5842c" Oct 30 14:03:02.500727 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:02.500689 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f56cfd10-4ad6-4c14-b614-ae3b48b87dbe-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-5842c\" (UID: \"f56cfd10-4ad6-4c14-b614-ae3b48b87dbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5842c" Oct 30 14:03:02.500959 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:02.500936 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f56cfd10-4ad6-4c14-b614-ae3b48b87dbe-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-5842c\" (UID: \"f56cfd10-4ad6-4c14-b614-ae3b48b87dbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5842c" Oct 30 14:03:02.501135 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:02.501115 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f56cfd10-4ad6-4c14-b614-ae3b48b87dbe-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-5842c\" (UID: \"f56cfd10-4ad6-4c14-b614-ae3b48b87dbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5842c" Oct 30 14:03:02.501197 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:02.501132 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f56cfd10-4ad6-4c14-b614-ae3b48b87dbe-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-5842c\" (UID: \"f56cfd10-4ad6-4c14-b614-ae3b48b87dbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5842c" Oct 30 14:03:02.502414 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:02.502395 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f56cfd10-4ad6-4c14-b614-ae3b48b87dbe-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-5842c\" (UID: \"f56cfd10-4ad6-4c14-b614-ae3b48b87dbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5842c" Oct 30 14:03:02.502718 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:02.502700 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f56cfd10-4ad6-4c14-b614-ae3b48b87dbe-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-5842c\" (UID: \"f56cfd10-4ad6-4c14-b614-ae3b48b87dbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5842c" Oct 30 14:03:02.528882 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:02.528854 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f56cfd10-4ad6-4c14-b614-ae3b48b87dbe-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-5842c\" (UID: \"f56cfd10-4ad6-4c14-b614-ae3b48b87dbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5842c" Oct 30 14:03:02.529161 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:02.529137 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ssg9\" (UniqueName: \"kubernetes.io/projected/f56cfd10-4ad6-4c14-b614-ae3b48b87dbe-kube-api-access-8ssg9\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-5842c\" (UID: \"f56cfd10-4ad6-4c14-b614-ae3b48b87dbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5842c" Oct 30 14:03:02.585888 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:02.585861 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5842c" Oct 30 14:03:02.901246 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:02.901217 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5842c"] Oct 30 14:03:02.902516 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:03:02.902490 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf56cfd10_4ad6_4c14_b614_ae3b48b87dbe.slice/crio-3e0cb5a32365a330219ffa4727e432e72e114a1113514db6597d4fe554306eb8 WatchSource:0}: Error finding container 3e0cb5a32365a330219ffa4727e432e72e114a1113514db6597d4fe554306eb8: Status 404 returned error can't find the container with id 3e0cb5a32365a330219ffa4727e432e72e114a1113514db6597d4fe554306eb8 Oct 30 14:03:02.904540 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:02.904511 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:03:02.904598 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:02.904579 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:03:02.904653 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:02.904608 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:03:02.940029 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:02.940004 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5842c" event={"ID":"f56cfd10-4ad6-4c14-b614-ae3b48b87dbe","Type":"ContainerStarted","Data":"3e0cb5a32365a330219ffa4727e432e72e114a1113514db6597d4fe554306eb8"} Oct 30 14:03:03.668465 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:03.668427 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:03:03.946407 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:03.946330 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5842c" event={"ID":"f56cfd10-4ad6-4c14-b614-ae3b48b87dbe","Type":"ContainerStarted","Data":"dffe6a02a7a9b8f26bd1e813d4a04b50f1970cb1dbe6cc1cdb7998e5cfc012d4"} Oct 30 14:03:04.032566 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:04.032515 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5842c" podStartSLOduration=2.032503276 podStartE2EDuration="2.032503276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:03:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:03:04.018461451 +0000 UTC m=+853.762595491" watchObservedRunningTime="2026-10-30 14:03:04.032503276 +0000 UTC m=+853.776637317" Oct 30 14:03:04.593663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:04.591534 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5842c" Oct 30 14:03:04.593987 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:04.593956 4982 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-5842c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Oct 30 14:03:04.594188 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:04.594156 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5842c" podUID="f56cfd10-4ad6-4c14-b614-ae3b48b87dbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Oct 30 14:03:05.459298 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:05.459267 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:03:05.604678 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:05.602988 4982 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-5842c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Oct 30 14:03:05.604678 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:05.603063 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5842c" podUID="f56cfd10-4ad6-4c14-b614-ae3b48b87dbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Oct 30 14:03:05.738440 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:05.738365 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:03:06.601010 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:06.600971 4982 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-5842c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Oct 30 14:03:06.601574 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:06.601541 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5842c" podUID="f56cfd10-4ad6-4c14-b614-ae3b48b87dbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Oct 30 14:03:07.592021 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:07.591980 4982 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-5842c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Oct 30 14:03:07.592380 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:07.592346 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5842c" podUID="f56cfd10-4ad6-4c14-b614-ae3b48b87dbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Oct 30 14:03:08.586392 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:08.586358 4982 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-5842c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Oct 30 14:03:08.586795 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:08.586407 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5842c" podUID="f56cfd10-4ad6-4c14-b614-ae3b48b87dbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Oct 30 14:03:09.586766 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:09.586728 4982 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-5842c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Oct 30 14:03:09.587131 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:09.586791 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5842c" podUID="f56cfd10-4ad6-4c14-b614-ae3b48b87dbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Oct 30 14:03:10.284847 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:10.284810 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:03:10.441920 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:10.441882 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:03:11.588031 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:11.587991 4982 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-5842c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:03:11.588445 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:11.588057 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5842c" podUID="f56cfd10-4ad6-4c14-b614-ae3b48b87dbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:03:12.629833 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:12.629789 4982 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-5842c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:03:12.630238 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:12.629853 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5842c" podUID="f56cfd10-4ad6-4c14-b614-ae3b48b87dbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:03:12.945778 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:12.945698 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5842c" Oct 30 14:03:12.946112 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:12.946074 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5842c" Oct 30 14:03:12.947442 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:12.947423 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5842c" Oct 30 14:03:17.289590 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:17.289556 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mqc7s"] Oct 30 14:03:17.293668 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:17.293630 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mqc7s" Oct 30 14:03:17.328280 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:17.328254 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mqc7s"] Oct 30 14:03:17.333745 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:17.333720 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f784c488-9f50-4924-a54b-811e5be8661f-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mqc7s\" (UID: \"f784c488-9f50-4924-a54b-811e5be8661f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mqc7s" Oct 30 14:03:17.333825 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:17.333758 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4f77\" (UniqueName: \"kubernetes.io/projected/f784c488-9f50-4924-a54b-811e5be8661f-kube-api-access-c4f77\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mqc7s\" (UID: \"f784c488-9f50-4924-a54b-811e5be8661f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mqc7s" Oct 30 14:03:17.333899 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:17.333876 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f784c488-9f50-4924-a54b-811e5be8661f-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mqc7s\" (UID: \"f784c488-9f50-4924-a54b-811e5be8661f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mqc7s" Oct 30 14:03:17.333936 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:17.333916 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f784c488-9f50-4924-a54b-811e5be8661f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mqc7s\" (UID: \"f784c488-9f50-4924-a54b-811e5be8661f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mqc7s" Oct 30 14:03:17.333986 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:17.333971 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f784c488-9f50-4924-a54b-811e5be8661f-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mqc7s\" (UID: \"f784c488-9f50-4924-a54b-811e5be8661f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mqc7s" Oct 30 14:03:17.334020 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:17.333993 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f784c488-9f50-4924-a54b-811e5be8661f-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mqc7s\" (UID: \"f784c488-9f50-4924-a54b-811e5be8661f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mqc7s" Oct 30 14:03:17.334051 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:17.334045 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f784c488-9f50-4924-a54b-811e5be8661f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mqc7s\" (UID: \"f784c488-9f50-4924-a54b-811e5be8661f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mqc7s" Oct 30 14:03:17.334085 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:17.334062 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f784c488-9f50-4924-a54b-811e5be8661f-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mqc7s\" (UID: \"f784c488-9f50-4924-a54b-811e5be8661f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mqc7s" Oct 30 14:03:17.334114 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:17.334088 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f784c488-9f50-4924-a54b-811e5be8661f-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mqc7s\" (UID: \"f784c488-9f50-4924-a54b-811e5be8661f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mqc7s" Oct 30 14:03:17.434830 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:17.434801 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f784c488-9f50-4924-a54b-811e5be8661f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mqc7s\" (UID: \"f784c488-9f50-4924-a54b-811e5be8661f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mqc7s" Oct 30 14:03:17.435004 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:17.434845 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f784c488-9f50-4924-a54b-811e5be8661f-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mqc7s\" (UID: \"f784c488-9f50-4924-a54b-811e5be8661f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mqc7s" Oct 30 14:03:17.435004 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:17.434866 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f784c488-9f50-4924-a54b-811e5be8661f-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mqc7s\" (UID: \"f784c488-9f50-4924-a54b-811e5be8661f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mqc7s" Oct 30 14:03:17.435004 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:17.434905 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f784c488-9f50-4924-a54b-811e5be8661f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mqc7s\" (UID: \"f784c488-9f50-4924-a54b-811e5be8661f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mqc7s" Oct 30 14:03:17.435004 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:17.434932 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f784c488-9f50-4924-a54b-811e5be8661f-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mqc7s\" (UID: \"f784c488-9f50-4924-a54b-811e5be8661f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mqc7s" Oct 30 14:03:17.435004 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:17.434959 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f784c488-9f50-4924-a54b-811e5be8661f-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mqc7s\" (UID: \"f784c488-9f50-4924-a54b-811e5be8661f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mqc7s" Oct 30 14:03:17.435311 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:17.435080 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f784c488-9f50-4924-a54b-811e5be8661f-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mqc7s\" (UID: \"f784c488-9f50-4924-a54b-811e5be8661f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mqc7s" Oct 30 14:03:17.435311 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:17.435108 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4f77\" (UniqueName: \"kubernetes.io/projected/f784c488-9f50-4924-a54b-811e5be8661f-kube-api-access-c4f77\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mqc7s\" (UID: \"f784c488-9f50-4924-a54b-811e5be8661f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mqc7s" Oct 30 14:03:17.435311 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:17.435133 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f784c488-9f50-4924-a54b-811e5be8661f-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mqc7s\" (UID: \"f784c488-9f50-4924-a54b-811e5be8661f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mqc7s" Oct 30 14:03:17.435311 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:17.435265 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f784c488-9f50-4924-a54b-811e5be8661f-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mqc7s\" (UID: \"f784c488-9f50-4924-a54b-811e5be8661f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mqc7s" Oct 30 14:03:17.435749 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:17.435719 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f784c488-9f50-4924-a54b-811e5be8661f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mqc7s\" (UID: \"f784c488-9f50-4924-a54b-811e5be8661f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mqc7s" Oct 30 14:03:17.435877 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:17.435848 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f784c488-9f50-4924-a54b-811e5be8661f-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mqc7s\" (UID: \"f784c488-9f50-4924-a54b-811e5be8661f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mqc7s" Oct 30 14:03:17.435877 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:17.435872 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f784c488-9f50-4924-a54b-811e5be8661f-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mqc7s\" (UID: \"f784c488-9f50-4924-a54b-811e5be8661f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mqc7s" Oct 30 14:03:17.436050 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:17.435906 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f784c488-9f50-4924-a54b-811e5be8661f-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mqc7s\" (UID: \"f784c488-9f50-4924-a54b-811e5be8661f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mqc7s" Oct 30 14:03:17.437485 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:17.437464 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f784c488-9f50-4924-a54b-811e5be8661f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mqc7s\" (UID: \"f784c488-9f50-4924-a54b-811e5be8661f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mqc7s" Oct 30 14:03:17.437590 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:17.437573 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f784c488-9f50-4924-a54b-811e5be8661f-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mqc7s\" (UID: \"f784c488-9f50-4924-a54b-811e5be8661f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mqc7s" Oct 30 14:03:17.460912 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:17.460887 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f784c488-9f50-4924-a54b-811e5be8661f-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mqc7s\" (UID: \"f784c488-9f50-4924-a54b-811e5be8661f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mqc7s" Oct 30 14:03:17.460997 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:17.460947 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4f77\" (UniqueName: \"kubernetes.io/projected/f784c488-9f50-4924-a54b-811e5be8661f-kube-api-access-c4f77\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mqc7s\" (UID: \"f784c488-9f50-4924-a54b-811e5be8661f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mqc7s" Oct 30 14:03:17.605920 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:17.605898 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mqc7s" Oct 30 14:03:17.919182 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:17.919154 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mqc7s"] Oct 30 14:03:17.921108 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:03:17.921084 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf784c488_9f50_4924_a54b_811e5be8661f.slice/crio-e3ebe690e8529b7024d430a50b3b2b5fb66a010a67161e6678631f78264e9b02 WatchSource:0}: Error finding container e3ebe690e8529b7024d430a50b3b2b5fb66a010a67161e6678631f78264e9b02: Status 404 returned error can't find the container with id e3ebe690e8529b7024d430a50b3b2b5fb66a010a67161e6678631f78264e9b02 Oct 30 14:03:17.923124 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:17.923095 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:03:17.923193 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:17.923153 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:03:17.923193 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:17.923180 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:03:18.017237 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:18.017205 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mqc7s" event={"ID":"f784c488-9f50-4924-a54b-811e5be8661f","Type":"ContainerStarted","Data":"0a1761441bf64aba063de96a8471b4c44e57bd0a4368287d3c662bc5631a97eb"} Oct 30 14:03:18.017381 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:18.017257 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mqc7s" event={"ID":"f784c488-9f50-4924-a54b-811e5be8661f","Type":"ContainerStarted","Data":"e3ebe690e8529b7024d430a50b3b2b5fb66a010a67161e6678631f78264e9b02"} Oct 30 14:03:18.066514 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:18.066307 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mqc7s" podStartSLOduration=1.066294734 podStartE2EDuration="1.066294734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:03:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:03:18.065665284 +0000 UTC m=+867.809799326" watchObservedRunningTime="2026-10-30 14:03:18.066294734 +0000 UTC m=+867.810428841" Oct 30 14:03:18.606895 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:18.606848 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mqc7s" Oct 30 14:03:18.608563 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:18.608529 4982 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-mqc7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Oct 30 14:03:18.608725 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:18.608590 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mqc7s" podUID="f784c488-9f50-4924-a54b-811e5be8661f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Oct 30 14:03:19.607352 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:19.607316 4982 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-mqc7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Oct 30 14:03:19.607930 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:19.607888 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mqc7s" podUID="f784c488-9f50-4924-a54b-811e5be8661f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Oct 30 14:03:20.609910 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:20.609869 4982 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-mqc7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Oct 30 14:03:20.610406 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:20.609930 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mqc7s" podUID="f784c488-9f50-4924-a54b-811e5be8661f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Oct 30 14:03:21.607143 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:21.607094 4982 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-mqc7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Oct 30 14:03:21.607353 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:21.607159 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mqc7s" podUID="f784c488-9f50-4924-a54b-811e5be8661f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Oct 30 14:03:22.607049 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:22.607014 4982 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-mqc7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Oct 30 14:03:22.607508 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:22.607065 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mqc7s" podUID="f784c488-9f50-4924-a54b-811e5be8661f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Oct 30 14:03:23.606741 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:23.606696 4982 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-mqc7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Oct 30 14:03:23.606946 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:23.606751 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mqc7s" podUID="f784c488-9f50-4924-a54b-811e5be8661f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Oct 30 14:03:24.606681 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:24.606624 4982 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-mqc7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Oct 30 14:03:24.607095 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:24.606700 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mqc7s" podUID="f784c488-9f50-4924-a54b-811e5be8661f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Oct 30 14:03:25.801186 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:25.801148 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mqc7s" Oct 30 14:03:25.801611 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:25.801479 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mqc7s" Oct 30 14:03:25.802442 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:25.802420 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mqc7s" Oct 30 14:03:28.010208 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:03:28.010112 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:03:35.888271 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:35.888238 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:03:36.419210 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:36.419173 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:03:38.056351 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:38.056313 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:03:38.199453 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:38.199420 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:03:39.573953 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:39.573922 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:03:39.729577 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:39.729546 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:03:41.492346 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:41.492306 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:03:50.476465 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:50.476437 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ntlqq_dc967875-1546-40bb-8376-fde09416c204/console-operator/2.log" Oct 30 14:03:50.476883 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:50.476573 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ntlqq_dc967875-1546-40bb-8376-fde09416c204/console-operator/2.log" Oct 30 14:03:50.484667 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:50.484626 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhxjf_aca47fe8-7068-4743-b6ab-283ee0874f66/ovn-acl-logging/0.log" Oct 30 14:03:50.484814 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:50.484652 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhxjf_aca47fe8-7068-4743-b6ab-283ee0874f66/ovn-acl-logging/0.log" Oct 30 14:03:57.132910 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:03:57.132879 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:03:58.009923 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:03:58.009888 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:04:23.050612 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:23.050567 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:23.189184 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:23.189148 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:23.575027 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:23.574969 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:23.733785 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:23.733745 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:23.889600 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:23.889493 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:24.042044 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:24.041988 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:24.205132 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:24.205034 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:24.354248 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:24.354211 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:24.506795 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:24.506694 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:24.667182 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:24.667145 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:27.091635 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:27.091600 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:27.250096 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:27.250061 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:27.815283 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:27.815216 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:28.013209 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:04:28.013169 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:04:29.050304 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:29.050264 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:29.462837 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:29.462799 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:29.631814 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:29.631774 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:31.140406 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:31.140376 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:31.282218 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:31.282174 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:31.968223 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:31.968191 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:32.119905 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:32.119870 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:32.747215 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:32.747177 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-00926c-b8457bc55-7f4kk"] Oct 30 14:04:32.750799 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:32.750776 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-00926c-b8457bc55-7f4kk" Oct 30 14:04:32.770111 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:32.770074 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-00926c-serving-cert\"" Oct 30 14:04:32.785130 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:32.785092 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-00926c-b8457bc55-7f4kk"] Oct 30 14:04:32.898494 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:32.898456 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h6gw\" (UniqueName: \"kubernetes.io/projected/f547db66-885a-4c00-8451-119811fffcac-kube-api-access-7h6gw\") pod \"maas-api-e2e-worker-w3-00926c-b8457bc55-7f4kk\" (UID: \"f547db66-885a-4c00-8451-119811fffcac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-00926c-b8457bc55-7f4kk" Oct 30 14:04:32.898689 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:32.898520 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f547db66-885a-4c00-8451-119811fffcac-maas-api-tls\") pod \"maas-api-e2e-worker-w3-00926c-b8457bc55-7f4kk\" (UID: \"f547db66-885a-4c00-8451-119811fffcac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-00926c-b8457bc55-7f4kk" Oct 30 14:04:32.898689 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:32.898603 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f547db66-885a-4c00-8451-119811fffcac-metrics-certs\") pod \"maas-api-e2e-worker-w3-00926c-b8457bc55-7f4kk\" (UID: \"f547db66-885a-4c00-8451-119811fffcac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-00926c-b8457bc55-7f4kk" Oct 30 14:04:32.898689 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:32.898633 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f547db66-885a-4c00-8451-119811fffcac-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-00926c-b8457bc55-7f4kk\" (UID: \"f547db66-885a-4c00-8451-119811fffcac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-00926c-b8457bc55-7f4kk" Oct 30 14:04:32.937557 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:32.937526 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:33.000048 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:32.999932 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f547db66-885a-4c00-8451-119811fffcac-metrics-certs\") pod \"maas-api-e2e-worker-w3-00926c-b8457bc55-7f4kk\" (UID: \"f547db66-885a-4c00-8451-119811fffcac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-00926c-b8457bc55-7f4kk" Oct 30 14:04:33.000048 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:32.999987 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f547db66-885a-4c00-8451-119811fffcac-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-00926c-b8457bc55-7f4kk\" (UID: \"f547db66-885a-4c00-8451-119811fffcac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-00926c-b8457bc55-7f4kk" Oct 30 14:04:33.000297 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:33.000076 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h6gw\" (UniqueName: \"kubernetes.io/projected/f547db66-885a-4c00-8451-119811fffcac-kube-api-access-7h6gw\") pod \"maas-api-e2e-worker-w3-00926c-b8457bc55-7f4kk\" (UID: \"f547db66-885a-4c00-8451-119811fffcac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-00926c-b8457bc55-7f4kk" Oct 30 14:04:33.000297 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:33.000161 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f547db66-885a-4c00-8451-119811fffcac-maas-api-tls\") pod \"maas-api-e2e-worker-w3-00926c-b8457bc55-7f4kk\" (UID: \"f547db66-885a-4c00-8451-119811fffcac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-00926c-b8457bc55-7f4kk" Oct 30 14:04:33.001201 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:33.001168 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f547db66-885a-4c00-8451-119811fffcac-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-00926c-b8457bc55-7f4kk\" (UID: \"f547db66-885a-4c00-8451-119811fffcac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-00926c-b8457bc55-7f4kk" Oct 30 14:04:33.003381 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:33.003345 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f547db66-885a-4c00-8451-119811fffcac-maas-api-tls\") pod \"maas-api-e2e-worker-w3-00926c-b8457bc55-7f4kk\" (UID: \"f547db66-885a-4c00-8451-119811fffcac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-00926c-b8457bc55-7f4kk" Oct 30 14:04:33.003976 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:33.003952 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f547db66-885a-4c00-8451-119811fffcac-metrics-certs\") pod \"maas-api-e2e-worker-w3-00926c-b8457bc55-7f4kk\" (UID: \"f547db66-885a-4c00-8451-119811fffcac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-00926c-b8457bc55-7f4kk" Oct 30 14:04:33.031434 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:33.031400 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h6gw\" (UniqueName: \"kubernetes.io/projected/f547db66-885a-4c00-8451-119811fffcac-kube-api-access-7h6gw\") pod \"maas-api-e2e-worker-w3-00926c-b8457bc55-7f4kk\" (UID: \"f547db66-885a-4c00-8451-119811fffcac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-00926c-b8457bc55-7f4kk" Oct 30 14:04:33.064119 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:33.064087 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-00926c-b8457bc55-7f4kk" Oct 30 14:04:33.090598 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:33.090569 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:33.376113 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:33.376085 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-00926c-b8457bc55-7f4kk"] Oct 30 14:04:33.377850 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:04:33.377822 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf547db66_885a_4c00_8451_119811fffcac.slice/crio-911575d8d3795f6fa443515dff2b871b19522271049f59e5cbb64d344a31a218 WatchSource:0}: Error finding container 911575d8d3795f6fa443515dff2b871b19522271049f59e5cbb64d344a31a218: Status 404 returned error can't find the container with id 911575d8d3795f6fa443515dff2b871b19522271049f59e5cbb64d344a31a218 Oct 30 14:04:34.369010 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:34.368971 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-00926c-b8457bc55-7f4kk" event={"ID":"f547db66-885a-4c00-8451-119811fffcac","Type":"ContainerStarted","Data":"128eff73857a3f12988f0eacf24635b09fa70e160d4045dbe6c36eae4d276f09"} Oct 30 14:04:34.369386 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:34.369025 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-00926c-b8457bc55-7f4kk" event={"ID":"f547db66-885a-4c00-8451-119811fffcac","Type":"ContainerStarted","Data":"911575d8d3795f6fa443515dff2b871b19522271049f59e5cbb64d344a31a218"} Oct 30 14:04:34.369386 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:34.369064 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-00926c-b8457bc55-7f4kk" Oct 30 14:04:34.415902 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:34.415859 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-00926c-b8457bc55-7f4kk" podStartSLOduration=2.008724182 podStartE2EDuration="2.415847268s" totalImagesPullingTime="407.123086ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:04:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:04:34.415064131 +0000 UTC m=+944.159198174" watchObservedRunningTime="2026-10-30 14:04:34.415847268 +0000 UTC m=+944.159981310" Oct 30 14:04:35.137802 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:35.137754 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:35.278691 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:35.278658 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:35.438607 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:35.438524 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:36.860205 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:36.860158 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:37.015865 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:37.015835 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:37.075100 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:37.075050 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-39529e-64b696694c-69mqr"] Oct 30 14:04:37.079469 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:37.079398 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-39529e-64b696694c-69mqr" Oct 30 14:04:37.116799 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:37.116714 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-39529e-64b696694c-69mqr"] Oct 30 14:04:37.124479 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:37.124450 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-39529e-serving-cert\"" Oct 30 14:04:37.218013 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:37.217974 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:37.240417 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:37.240386 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76be0827-b086-49a8-873d-5141c13bfc6e-metrics-certs\") pod \"maas-api-e2e-worker-w6-39529e-64b696694c-69mqr\" (UID: \"76be0827-b086-49a8-873d-5141c13bfc6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-39529e-64b696694c-69mqr" Oct 30 14:04:37.240417 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:37.240422 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp849\" (UniqueName: \"kubernetes.io/projected/76be0827-b086-49a8-873d-5141c13bfc6e-kube-api-access-hp849\") pod \"maas-api-e2e-worker-w6-39529e-64b696694c-69mqr\" (UID: \"76be0827-b086-49a8-873d-5141c13bfc6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-39529e-64b696694c-69mqr" Oct 30 14:04:37.240592 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:37.240442 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76be0827-b086-49a8-873d-5141c13bfc6e-maas-api-tls\") pod \"maas-api-e2e-worker-w6-39529e-64b696694c-69mqr\" (UID: \"76be0827-b086-49a8-873d-5141c13bfc6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-39529e-64b696694c-69mqr" Oct 30 14:04:37.240592 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:37.240563 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76be0827-b086-49a8-873d-5141c13bfc6e-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-39529e-64b696694c-69mqr\" (UID: \"76be0827-b086-49a8-873d-5141c13bfc6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-39529e-64b696694c-69mqr" Oct 30 14:04:37.341004 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:37.340966 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76be0827-b086-49a8-873d-5141c13bfc6e-metrics-certs\") pod \"maas-api-e2e-worker-w6-39529e-64b696694c-69mqr\" (UID: \"76be0827-b086-49a8-873d-5141c13bfc6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-39529e-64b696694c-69mqr" Oct 30 14:04:37.341197 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:37.341008 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp849\" (UniqueName: \"kubernetes.io/projected/76be0827-b086-49a8-873d-5141c13bfc6e-kube-api-access-hp849\") pod \"maas-api-e2e-worker-w6-39529e-64b696694c-69mqr\" (UID: \"76be0827-b086-49a8-873d-5141c13bfc6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-39529e-64b696694c-69mqr" Oct 30 14:04:37.341197 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:37.341068 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76be0827-b086-49a8-873d-5141c13bfc6e-maas-api-tls\") pod \"maas-api-e2e-worker-w6-39529e-64b696694c-69mqr\" (UID: \"76be0827-b086-49a8-873d-5141c13bfc6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-39529e-64b696694c-69mqr" Oct 30 14:04:37.341197 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:37.341123 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76be0827-b086-49a8-873d-5141c13bfc6e-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-39529e-64b696694c-69mqr\" (UID: \"76be0827-b086-49a8-873d-5141c13bfc6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-39529e-64b696694c-69mqr" Oct 30 14:04:37.341917 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:37.341896 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76be0827-b086-49a8-873d-5141c13bfc6e-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-39529e-64b696694c-69mqr\" (UID: \"76be0827-b086-49a8-873d-5141c13bfc6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-39529e-64b696694c-69mqr" Oct 30 14:04:37.343808 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:37.343786 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76be0827-b086-49a8-873d-5141c13bfc6e-metrics-certs\") pod \"maas-api-e2e-worker-w6-39529e-64b696694c-69mqr\" (UID: \"76be0827-b086-49a8-873d-5141c13bfc6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-39529e-64b696694c-69mqr" Oct 30 14:04:37.343905 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:37.343877 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76be0827-b086-49a8-873d-5141c13bfc6e-maas-api-tls\") pod \"maas-api-e2e-worker-w6-39529e-64b696694c-69mqr\" (UID: \"76be0827-b086-49a8-873d-5141c13bfc6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-39529e-64b696694c-69mqr" Oct 30 14:04:37.353275 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:37.353244 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:37.368254 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:37.368199 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp849\" (UniqueName: \"kubernetes.io/projected/76be0827-b086-49a8-873d-5141c13bfc6e-kube-api-access-hp849\") pod \"maas-api-e2e-worker-w6-39529e-64b696694c-69mqr\" (UID: \"76be0827-b086-49a8-873d-5141c13bfc6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-39529e-64b696694c-69mqr" Oct 30 14:04:37.399023 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:37.398998 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-39529e-64b696694c-69mqr" Oct 30 14:04:37.703323 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:37.703289 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-39529e-64b696694c-69mqr"] Oct 30 14:04:37.703463 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:04:37.703381 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76be0827_b086_49a8_873d_5141c13bfc6e.slice/crio-434a9c80cfb58b064a8fd9008a544383ea8ed96f103a2be3b8a4e41519bfeb42 WatchSource:0}: Error finding container 434a9c80cfb58b064a8fd9008a544383ea8ed96f103a2be3b8a4e41519bfeb42: Status 404 returned error can't find the container with id 434a9c80cfb58b064a8fd9008a544383ea8ed96f103a2be3b8a4e41519bfeb42 Oct 30 14:04:38.397101 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:38.397034 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-39529e-64b696694c-69mqr" event={"ID":"76be0827-b086-49a8-873d-5141c13bfc6e","Type":"ContainerStarted","Data":"b98448b173f66469b134dc8731f11edcb2b32d1e651e7a93a84835499ff96c0e"} Oct 30 14:04:38.397101 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:38.397087 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-39529e-64b696694c-69mqr" event={"ID":"76be0827-b086-49a8-873d-5141c13bfc6e","Type":"ContainerStarted","Data":"434a9c80cfb58b064a8fd9008a544383ea8ed96f103a2be3b8a4e41519bfeb42"} Oct 30 14:04:38.397468 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:38.397112 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-39529e-64b696694c-69mqr" Oct 30 14:04:38.436903 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:38.436851 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-39529e-64b696694c-69mqr" podStartSLOduration=1.055547533 podStartE2EDuration="1.436837194s" totalImagesPullingTime="381.289661ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:04:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:04:38.436245628 +0000 UTC m=+948.180379669" watchObservedRunningTime="2026-10-30 14:04:38.436837194 +0000 UTC m=+948.180971235" Oct 30 14:04:38.964004 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:38.963969 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-2df0ab-58454c8987-cjnqk"] Oct 30 14:04:38.967520 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:38.967501 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2df0ab-58454c8987-cjnqk" Oct 30 14:04:38.999055 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:38.999034 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-2df0ab-serving-cert\"" Oct 30 14:04:39.001368 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:39.001324 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-2df0ab-58454c8987-cjnqk"] Oct 30 14:04:39.056524 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:39.056501 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t789t\" (UniqueName: \"kubernetes.io/projected/757d5732-c143-453c-9149-5744017ea635-kube-api-access-t789t\") pod \"maas-api-e2e-worker-w0-2df0ab-58454c8987-cjnqk\" (UID: \"757d5732-c143-453c-9149-5744017ea635\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2df0ab-58454c8987-cjnqk" Oct 30 14:04:39.056612 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:39.056550 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/757d5732-c143-453c-9149-5744017ea635-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-2df0ab-58454c8987-cjnqk\" (UID: \"757d5732-c143-453c-9149-5744017ea635\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2df0ab-58454c8987-cjnqk" Oct 30 14:04:39.056695 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:39.056674 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/757d5732-c143-453c-9149-5744017ea635-maas-api-tls\") pod \"maas-api-e2e-worker-w0-2df0ab-58454c8987-cjnqk\" (UID: \"757d5732-c143-453c-9149-5744017ea635\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2df0ab-58454c8987-cjnqk" Oct 30 14:04:39.056734 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:39.056726 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/757d5732-c143-453c-9149-5744017ea635-metrics-certs\") pod \"maas-api-e2e-worker-w0-2df0ab-58454c8987-cjnqk\" (UID: \"757d5732-c143-453c-9149-5744017ea635\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2df0ab-58454c8987-cjnqk" Oct 30 14:04:39.095392 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:39.095365 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:39.157300 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:39.157270 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t789t\" (UniqueName: \"kubernetes.io/projected/757d5732-c143-453c-9149-5744017ea635-kube-api-access-t789t\") pod \"maas-api-e2e-worker-w0-2df0ab-58454c8987-cjnqk\" (UID: \"757d5732-c143-453c-9149-5744017ea635\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2df0ab-58454c8987-cjnqk" Oct 30 14:04:39.157428 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:39.157339 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/757d5732-c143-453c-9149-5744017ea635-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-2df0ab-58454c8987-cjnqk\" (UID: \"757d5732-c143-453c-9149-5744017ea635\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2df0ab-58454c8987-cjnqk" Oct 30 14:04:39.157428 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:39.157390 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/757d5732-c143-453c-9149-5744017ea635-maas-api-tls\") pod \"maas-api-e2e-worker-w0-2df0ab-58454c8987-cjnqk\" (UID: \"757d5732-c143-453c-9149-5744017ea635\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2df0ab-58454c8987-cjnqk" Oct 30 14:04:39.157543 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:39.157439 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/757d5732-c143-453c-9149-5744017ea635-metrics-certs\") pod \"maas-api-e2e-worker-w0-2df0ab-58454c8987-cjnqk\" (UID: \"757d5732-c143-453c-9149-5744017ea635\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2df0ab-58454c8987-cjnqk" Oct 30 14:04:39.157601 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:04:39.157569 4982 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-2df0ab-serving-cert: secret "maas-api-e2e-worker-w0-2df0ab-serving-cert" not found Oct 30 14:04:39.157667 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:04:39.157628 4982 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/757d5732-c143-453c-9149-5744017ea635-maas-api-tls podName:757d5732-c143-453c-9149-5744017ea635 nodeName:}" failed. No retries permitted until 2026-10-30 14:04:39.657613189 +0000 UTC m=+949.401747212 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/757d5732-c143-453c-9149-5744017ea635-maas-api-tls") pod "maas-api-e2e-worker-w0-2df0ab-58454c8987-cjnqk" (UID: "757d5732-c143-453c-9149-5744017ea635") : secret "maas-api-e2e-worker-w0-2df0ab-serving-cert" not found Oct 30 14:04:39.158008 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:39.157989 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/757d5732-c143-453c-9149-5744017ea635-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-2df0ab-58454c8987-cjnqk\" (UID: \"757d5732-c143-453c-9149-5744017ea635\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2df0ab-58454c8987-cjnqk" Oct 30 14:04:39.159976 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:39.159953 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/757d5732-c143-453c-9149-5744017ea635-metrics-certs\") pod \"maas-api-e2e-worker-w0-2df0ab-58454c8987-cjnqk\" (UID: \"757d5732-c143-453c-9149-5744017ea635\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2df0ab-58454c8987-cjnqk" Oct 30 14:04:39.188956 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:39.188930 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t789t\" (UniqueName: \"kubernetes.io/projected/757d5732-c143-453c-9149-5744017ea635-kube-api-access-t789t\") pod \"maas-api-e2e-worker-w0-2df0ab-58454c8987-cjnqk\" (UID: \"757d5732-c143-453c-9149-5744017ea635\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2df0ab-58454c8987-cjnqk" Oct 30 14:04:39.338816 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:39.338783 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:39.484955 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:39.484920 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:39.662742 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:39.662636 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/757d5732-c143-453c-9149-5744017ea635-maas-api-tls\") pod \"maas-api-e2e-worker-w0-2df0ab-58454c8987-cjnqk\" (UID: \"757d5732-c143-453c-9149-5744017ea635\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2df0ab-58454c8987-cjnqk" Oct 30 14:04:39.665356 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:39.665322 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/757d5732-c143-453c-9149-5744017ea635-maas-api-tls\") pod \"maas-api-e2e-worker-w0-2df0ab-58454c8987-cjnqk\" (UID: \"757d5732-c143-453c-9149-5744017ea635\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2df0ab-58454c8987-cjnqk" Oct 30 14:04:39.880631 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:39.880589 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2df0ab-58454c8987-cjnqk" Oct 30 14:04:40.192309 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:40.192275 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-2df0ab-58454c8987-cjnqk"] Oct 30 14:04:40.193269 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:04:40.193243 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod757d5732_c143_453c_9149_5744017ea635.slice/crio-61ead91e528a60d02ba4ba82344925d8de92b728bdde5c74c6b7828d1a9d14f6 WatchSource:0}: Error finding container 61ead91e528a60d02ba4ba82344925d8de92b728bdde5c74c6b7828d1a9d14f6: Status 404 returned error can't find the container with id 61ead91e528a60d02ba4ba82344925d8de92b728bdde5c74c6b7828d1a9d14f6 Oct 30 14:04:40.379815 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:40.379790 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-00926c-b8457bc55-7f4kk" Oct 30 14:04:40.405772 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:40.405691 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2df0ab-58454c8987-cjnqk" event={"ID":"757d5732-c143-453c-9149-5744017ea635","Type":"ContainerStarted","Data":"61ead91e528a60d02ba4ba82344925d8de92b728bdde5c74c6b7828d1a9d14f6"} Oct 30 14:04:41.059578 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:41.059543 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:41.412855 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:41.412765 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2df0ab-58454c8987-cjnqk" event={"ID":"757d5732-c143-453c-9149-5744017ea635","Type":"ContainerStarted","Data":"650906439f75a7905bac8c85cda95633476909dd2e6361fa52494a36efd02472"} Oct 30 14:04:41.413019 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:41.412909 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2df0ab-58454c8987-cjnqk" Oct 30 14:04:41.462723 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:41.462667 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2df0ab-58454c8987-cjnqk" podStartSLOduration=3.171670299 podStartE2EDuration="3.462630254s" totalImagesPullingTime="290.959955ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:04:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:04:41.46069491 +0000 UTC m=+951.204828953" watchObservedRunningTime="2026-10-30 14:04:41.462630254 +0000 UTC m=+951.206764297" Oct 30 14:04:41.604818 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:41.604788 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:43.092622 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:43.092585 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:43.235418 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:43.235383 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:43.386274 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:43.386203 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:44.401558 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:44.401529 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-39529e-64b696694c-69mqr" Oct 30 14:04:45.587462 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:45.587428 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:45.725977 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:45.725937 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:46.024864 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:46.024784 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:46.173342 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:46.173303 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:47.423610 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:47.423571 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2df0ab-58454c8987-cjnqk" Oct 30 14:04:47.856958 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:47.856922 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:48.010451 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:48.010408 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:48.501300 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:48.501264 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:50.583190 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:50.583151 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:52.093066 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:52.093033 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:52.714928 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:52.714897 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:52.869067 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:52.869035 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:53.015067 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:53.014981 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:54.045612 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:54.045576 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v"] Oct 30 14:04:54.050620 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:54.050600 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v" Oct 30 14:04:54.074353 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:54.074312 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v"] Oct 30 14:04:54.099698 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:54.099669 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlp77\" (UniqueName: \"kubernetes.io/projected/68c5e59c-004d-4751-bed1-f7943e0548fd-kube-api-access-hlp77\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v\" (UID: \"68c5e59c-004d-4751-bed1-f7943e0548fd\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v" Oct 30 14:04:54.099806 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:54.099783 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68c5e59c-004d-4751-bed1-f7943e0548fd-istio-envoy\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v\" (UID: \"68c5e59c-004d-4751-bed1-f7943e0548fd\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v" Oct 30 14:04:54.099852 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:54.099818 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68c5e59c-004d-4751-bed1-f7943e0548fd-istiod-ca-cert\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v\" (UID: \"68c5e59c-004d-4751-bed1-f7943e0548fd\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v" Oct 30 14:04:54.099852 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:54.099844 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68c5e59c-004d-4751-bed1-f7943e0548fd-credential-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v\" (UID: \"68c5e59c-004d-4751-bed1-f7943e0548fd\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v" Oct 30 14:04:54.099984 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:54.099969 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68c5e59c-004d-4751-bed1-f7943e0548fd-istio-token\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v\" (UID: \"68c5e59c-004d-4751-bed1-f7943e0548fd\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v" Oct 30 14:04:54.100027 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:54.100016 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68c5e59c-004d-4751-bed1-f7943e0548fd-workload-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v\" (UID: \"68c5e59c-004d-4751-bed1-f7943e0548fd\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v" Oct 30 14:04:54.100061 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:54.100034 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68c5e59c-004d-4751-bed1-f7943e0548fd-istio-data\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v\" (UID: \"68c5e59c-004d-4751-bed1-f7943e0548fd\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v" Oct 30 14:04:54.100108 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:54.100063 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68c5e59c-004d-4751-bed1-f7943e0548fd-workload-certs\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v\" (UID: \"68c5e59c-004d-4751-bed1-f7943e0548fd\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v" Oct 30 14:04:54.100108 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:54.100091 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68c5e59c-004d-4751-bed1-f7943e0548fd-istio-podinfo\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v\" (UID: \"68c5e59c-004d-4751-bed1-f7943e0548fd\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v" Oct 30 14:04:54.155196 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:54.155171 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-39529e-openshift-default-dockercfg-l4zq2\"" Oct 30 14:04:54.200826 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:54.200794 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68c5e59c-004d-4751-bed1-f7943e0548fd-credential-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v\" (UID: \"68c5e59c-004d-4751-bed1-f7943e0548fd\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v" Oct 30 14:04:54.200935 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:54.200883 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68c5e59c-004d-4751-bed1-f7943e0548fd-istio-token\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v\" (UID: \"68c5e59c-004d-4751-bed1-f7943e0548fd\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v" Oct 30 14:04:54.201000 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:54.200932 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68c5e59c-004d-4751-bed1-f7943e0548fd-workload-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v\" (UID: \"68c5e59c-004d-4751-bed1-f7943e0548fd\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v" Oct 30 14:04:54.201000 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:54.200956 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68c5e59c-004d-4751-bed1-f7943e0548fd-istio-data\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v\" (UID: \"68c5e59c-004d-4751-bed1-f7943e0548fd\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v" Oct 30 14:04:54.201000 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:54.200987 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68c5e59c-004d-4751-bed1-f7943e0548fd-workload-certs\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v\" (UID: \"68c5e59c-004d-4751-bed1-f7943e0548fd\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v" Oct 30 14:04:54.201141 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:54.201011 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68c5e59c-004d-4751-bed1-f7943e0548fd-istio-podinfo\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v\" (UID: \"68c5e59c-004d-4751-bed1-f7943e0548fd\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v" Oct 30 14:04:54.201141 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:54.201042 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlp77\" (UniqueName: \"kubernetes.io/projected/68c5e59c-004d-4751-bed1-f7943e0548fd-kube-api-access-hlp77\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v\" (UID: \"68c5e59c-004d-4751-bed1-f7943e0548fd\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v" Oct 30 14:04:54.201141 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:54.201105 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68c5e59c-004d-4751-bed1-f7943e0548fd-istio-envoy\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v\" (UID: \"68c5e59c-004d-4751-bed1-f7943e0548fd\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v" Oct 30 14:04:54.201141 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:54.201135 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68c5e59c-004d-4751-bed1-f7943e0548fd-istiod-ca-cert\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v\" (UID: \"68c5e59c-004d-4751-bed1-f7943e0548fd\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v" Oct 30 14:04:54.201356 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:54.201246 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68c5e59c-004d-4751-bed1-f7943e0548fd-credential-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v\" (UID: \"68c5e59c-004d-4751-bed1-f7943e0548fd\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v" Oct 30 14:04:54.201415 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:54.201354 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68c5e59c-004d-4751-bed1-f7943e0548fd-workload-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v\" (UID: \"68c5e59c-004d-4751-bed1-f7943e0548fd\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v" Oct 30 14:04:54.201798 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:54.201769 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68c5e59c-004d-4751-bed1-f7943e0548fd-istio-data\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v\" (UID: \"68c5e59c-004d-4751-bed1-f7943e0548fd\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v" Oct 30 14:04:54.201901 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:54.201837 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68c5e59c-004d-4751-bed1-f7943e0548fd-workload-certs\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v\" (UID: \"68c5e59c-004d-4751-bed1-f7943e0548fd\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v" Oct 30 14:04:54.201901 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:54.201873 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68c5e59c-004d-4751-bed1-f7943e0548fd-istiod-ca-cert\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v\" (UID: \"68c5e59c-004d-4751-bed1-f7943e0548fd\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v" Oct 30 14:04:54.203813 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:54.203786 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68c5e59c-004d-4751-bed1-f7943e0548fd-istio-envoy\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v\" (UID: \"68c5e59c-004d-4751-bed1-f7943e0548fd\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v" Oct 30 14:04:54.204234 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:54.204212 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68c5e59c-004d-4751-bed1-f7943e0548fd-istio-podinfo\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v\" (UID: \"68c5e59c-004d-4751-bed1-f7943e0548fd\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v" Oct 30 14:04:54.226859 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:54.226828 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68c5e59c-004d-4751-bed1-f7943e0548fd-istio-token\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v\" (UID: \"68c5e59c-004d-4751-bed1-f7943e0548fd\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v" Oct 30 14:04:54.226965 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:54.226937 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlp77\" (UniqueName: \"kubernetes.io/projected/68c5e59c-004d-4751-bed1-f7943e0548fd-kube-api-access-hlp77\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v\" (UID: \"68c5e59c-004d-4751-bed1-f7943e0548fd\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v" Oct 30 14:04:54.363444 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:54.363419 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v" Oct 30 14:04:54.760389 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:54.760357 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v"] Oct 30 14:04:54.761843 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:04:54.761814 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68c5e59c_004d_4751_bed1_f7943e0548fd.slice/crio-6abce4cc1e54e59aaafc3953569dd3aa6354667791b4acf8535e572dca3a45aa WatchSource:0}: Error finding container 6abce4cc1e54e59aaafc3953569dd3aa6354667791b4acf8535e572dca3a45aa: Status 404 returned error can't find the container with id 6abce4cc1e54e59aaafc3953569dd3aa6354667791b4acf8535e572dca3a45aa Oct 30 14:04:54.764339 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:54.764306 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:04:54.764416 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:54.764366 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:04:54.764416 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:54.764393 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:04:54.903058 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:54.902990 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:55.302662 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:55.290464 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb"] Oct 30 14:04:55.308659 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:55.307976 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" Oct 30 14:04:55.321588 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:55.321556 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb"] Oct 30 14:04:55.332047 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:55.332017 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-00926c\"/\"simulator-w3-00926c-kserve-self-signed-certs\"" Oct 30 14:04:55.332191 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:55.332147 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-00926c\"/\"default-dockercfg-lbdrq\"" Oct 30 14:04:55.337776 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:55.337757 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-00926c\"/\"openshift-service-ca.crt\"" Oct 30 14:04:55.337776 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:55.337769 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-00926c\"/\"kube-root-ca.crt\"" Oct 30 14:04:55.384663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:55.379732 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:55.416591 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:55.416549 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d65dee9b-4f8b-4f09-b00b-0793d732d3ab-model-cache\") pod \"simulator-w3-00926c-kserve-d6597cf86-4f8zb\" (UID: \"d65dee9b-4f8b-4f09-b00b-0793d732d3ab\") " pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" Oct 30 14:04:55.416808 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:55.416601 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d65dee9b-4f8b-4f09-b00b-0793d732d3ab-home\") pod \"simulator-w3-00926c-kserve-d6597cf86-4f8zb\" (UID: \"d65dee9b-4f8b-4f09-b00b-0793d732d3ab\") " pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" Oct 30 14:04:55.416808 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:55.416657 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d65dee9b-4f8b-4f09-b00b-0793d732d3ab-tls-certs\") pod \"simulator-w3-00926c-kserve-d6597cf86-4f8zb\" (UID: \"d65dee9b-4f8b-4f09-b00b-0793d732d3ab\") " pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" Oct 30 14:04:55.416808 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:55.416759 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d65dee9b-4f8b-4f09-b00b-0793d732d3ab-tmp-dir\") pod \"simulator-w3-00926c-kserve-d6597cf86-4f8zb\" (UID: \"d65dee9b-4f8b-4f09-b00b-0793d732d3ab\") " pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" Oct 30 14:04:55.416808 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:55.416792 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jth7w\" (UniqueName: \"kubernetes.io/projected/d65dee9b-4f8b-4f09-b00b-0793d732d3ab-kube-api-access-jth7w\") pod \"simulator-w3-00926c-kserve-d6597cf86-4f8zb\" (UID: \"d65dee9b-4f8b-4f09-b00b-0793d732d3ab\") " pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" Oct 30 14:04:55.416941 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:55.416850 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d65dee9b-4f8b-4f09-b00b-0793d732d3ab-dshm\") pod \"simulator-w3-00926c-kserve-d6597cf86-4f8zb\" (UID: \"d65dee9b-4f8b-4f09-b00b-0793d732d3ab\") " pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" Oct 30 14:04:55.506543 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:55.506506 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v" event={"ID":"68c5e59c-004d-4751-bed1-f7943e0548fd","Type":"ContainerStarted","Data":"57dee7f0140bb9eb6c64184aa7957d4c5fa8b3dd914b28e44e360302195b4983"} Oct 30 14:04:55.506893 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:55.506867 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v" event={"ID":"68c5e59c-004d-4751-bed1-f7943e0548fd","Type":"ContainerStarted","Data":"6abce4cc1e54e59aaafc3953569dd3aa6354667791b4acf8535e572dca3a45aa"} Oct 30 14:04:55.517635 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:55.517611 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d65dee9b-4f8b-4f09-b00b-0793d732d3ab-tmp-dir\") pod \"simulator-w3-00926c-kserve-d6597cf86-4f8zb\" (UID: \"d65dee9b-4f8b-4f09-b00b-0793d732d3ab\") " pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" Oct 30 14:04:55.517852 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:55.517828 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jth7w\" (UniqueName: \"kubernetes.io/projected/d65dee9b-4f8b-4f09-b00b-0793d732d3ab-kube-api-access-jth7w\") pod \"simulator-w3-00926c-kserve-d6597cf86-4f8zb\" (UID: \"d65dee9b-4f8b-4f09-b00b-0793d732d3ab\") " pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" Oct 30 14:04:55.517938 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:55.517863 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d65dee9b-4f8b-4f09-b00b-0793d732d3ab-dshm\") pod \"simulator-w3-00926c-kserve-d6597cf86-4f8zb\" (UID: \"d65dee9b-4f8b-4f09-b00b-0793d732d3ab\") " pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" Oct 30 14:04:55.518001 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:55.517938 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d65dee9b-4f8b-4f09-b00b-0793d732d3ab-model-cache\") pod \"simulator-w3-00926c-kserve-d6597cf86-4f8zb\" (UID: \"d65dee9b-4f8b-4f09-b00b-0793d732d3ab\") " pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" Oct 30 14:04:55.518001 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:55.517987 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d65dee9b-4f8b-4f09-b00b-0793d732d3ab-home\") pod \"simulator-w3-00926c-kserve-d6597cf86-4f8zb\" (UID: \"d65dee9b-4f8b-4f09-b00b-0793d732d3ab\") " pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" Oct 30 14:04:55.518098 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:55.518029 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d65dee9b-4f8b-4f09-b00b-0793d732d3ab-tls-certs\") pod \"simulator-w3-00926c-kserve-d6597cf86-4f8zb\" (UID: \"d65dee9b-4f8b-4f09-b00b-0793d732d3ab\") " pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" Oct 30 14:04:55.518404 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:55.518376 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d65dee9b-4f8b-4f09-b00b-0793d732d3ab-tmp-dir\") pod \"simulator-w3-00926c-kserve-d6597cf86-4f8zb\" (UID: \"d65dee9b-4f8b-4f09-b00b-0793d732d3ab\") " pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" Oct 30 14:04:55.518507 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:55.518395 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d65dee9b-4f8b-4f09-b00b-0793d732d3ab-model-cache\") pod \"simulator-w3-00926c-kserve-d6597cf86-4f8zb\" (UID: \"d65dee9b-4f8b-4f09-b00b-0793d732d3ab\") " pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" Oct 30 14:04:55.518591 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:55.518570 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d65dee9b-4f8b-4f09-b00b-0793d732d3ab-home\") pod \"simulator-w3-00926c-kserve-d6597cf86-4f8zb\" (UID: \"d65dee9b-4f8b-4f09-b00b-0793d732d3ab\") " pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" Oct 30 14:04:55.520295 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:55.520274 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d65dee9b-4f8b-4f09-b00b-0793d732d3ab-dshm\") pod \"simulator-w3-00926c-kserve-d6597cf86-4f8zb\" (UID: \"d65dee9b-4f8b-4f09-b00b-0793d732d3ab\") " pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" Oct 30 14:04:55.520567 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:55.520548 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d65dee9b-4f8b-4f09-b00b-0793d732d3ab-tls-certs\") pod \"simulator-w3-00926c-kserve-d6597cf86-4f8zb\" (UID: \"d65dee9b-4f8b-4f09-b00b-0793d732d3ab\") " pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" Oct 30 14:04:55.549659 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:55.546864 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jth7w\" (UniqueName: \"kubernetes.io/projected/d65dee9b-4f8b-4f09-b00b-0793d732d3ab-kube-api-access-jth7w\") pod \"simulator-w3-00926c-kserve-d6597cf86-4f8zb\" (UID: \"d65dee9b-4f8b-4f09-b00b-0793d732d3ab\") " pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" Oct 30 14:04:55.559371 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:55.559325 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v" podStartSLOduration=1.559311879 podStartE2EDuration="1.559311879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:04:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:04:55.558122254 +0000 UTC m=+965.302256298" watchObservedRunningTime="2026-10-30 14:04:55.559311879 +0000 UTC m=+965.303445920" Oct 30 14:04:55.631662 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:55.631616 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" Oct 30 14:04:55.957745 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:55.957708 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb"] Oct 30 14:04:55.958539 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:04:55.958513 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd65dee9b_4f8b_4f09_b00b_0793d732d3ab.slice/crio-21a1e0312170883e575298bbd71b8e4a591113545ff9db4b39907d97fd62b61f WatchSource:0}: Error finding container 21a1e0312170883e575298bbd71b8e4a591113545ff9db4b39907d97fd62b61f: Status 404 returned error can't find the container with id 21a1e0312170883e575298bbd71b8e4a591113545ff9db4b39907d97fd62b61f Oct 30 14:04:56.363926 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:56.363852 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v" Oct 30 14:04:56.365530 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:56.365501 4982 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Oct 30 14:04:56.365697 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:56.365553 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v" podUID="68c5e59c-004d-4751-bed1-f7943e0548fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Oct 30 14:04:56.523921 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:56.523164 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" event={"ID":"d65dee9b-4f8b-4f09-b00b-0793d732d3ab","Type":"ContainerStarted","Data":"21a1e0312170883e575298bbd71b8e4a591113545ff9db4b39907d97fd62b61f"} Oct 30 14:04:57.364108 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:57.364065 4982 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Oct 30 14:04:57.364514 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:57.364133 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v" podUID="68c5e59c-004d-4751-bed1-f7943e0548fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Oct 30 14:04:58.034558 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:04:58.034477 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:04:58.112108 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:58.112068 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:58.258287 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:58.258245 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:58.364325 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:58.364290 4982 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Oct 30 14:04:58.364803 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:58.364349 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v" podUID="68c5e59c-004d-4751-bed1-f7943e0548fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Oct 30 14:04:58.537453 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:58.537417 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" event={"ID":"d65dee9b-4f8b-4f09-b00b-0793d732d3ab","Type":"ContainerStarted","Data":"2eb837ccd1af3f8712b30d47022687fe2c17e8013ee26e231bf5eb16986b6432"} Oct 30 14:04:58.631960 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:58.631884 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" Oct 30 14:04:58.839964 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:58.839752 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:59.012715 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:59.012121 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:59.161772 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:59.161740 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:04:59.364825 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:59.364788 4982 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Oct 30 14:04:59.365313 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:04:59.364855 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v" podUID="68c5e59c-004d-4751-bed1-f7943e0548fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Oct 30 14:05:00.363804 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:00.363768 4982 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Oct 30 14:05:00.363993 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:00.363828 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v" podUID="68c5e59c-004d-4751-bed1-f7943e0548fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Oct 30 14:05:01.367995 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:01.367959 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v" Oct 30 14:05:01.368412 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:01.368254 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v" Oct 30 14:05:01.368973 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:01.368955 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v" Oct 30 14:05:01.430955 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:01.430899 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" podStartSLOduration=4.730210062 podStartE2EDuration="6.430883519s" totalImagesPullingTime="1.700673457s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:04:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:04:58.59573042 +0000 UTC m=+968.339864463" watchObservedRunningTime="2026-10-30 14:05:01.430883519 +0000 UTC m=+971.175017561" Oct 30 14:05:01.490909 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:01.490870 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:03.425913 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:03.425876 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:03.577784 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:03.577748 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:03.775708 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:03.775578 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:04.131071 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:04.131038 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:04.280491 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:04.280452 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:04.434761 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:04.434723 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:05.494234 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:05.494192 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:05.632302 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:05.632264 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" Oct 30 14:05:05.645473 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:05.645441 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" Oct 30 14:05:06.585865 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:06.585832 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" Oct 30 14:05:07.414625 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:07.414583 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9"] Oct 30 14:05:07.421488 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:07.421460 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" Oct 30 14:05:07.439884 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:07.439848 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:07.440128 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:07.439887 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9"] Oct 30 14:05:07.451185 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:07.451148 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-da78c4\"/\"openshift-service-ca.crt\"" Oct 30 14:05:07.451185 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:07.451174 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-da78c4\"/\"kube-root-ca.crt\"" Oct 30 14:05:07.451409 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:07.451215 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-da78c4\"/\"simulator-w2-da78c4-kserve-self-signed-certs\"" Oct 30 14:05:07.451409 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:07.451271 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-da78c4\"/\"default-dockercfg-hskdp\"" Oct 30 14:05:07.543772 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:07.543727 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc3a3a28-d723-459a-93d0-ff115eac479a-dshm\") pod \"simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9\" (UID: \"cc3a3a28-d723-459a-93d0-ff115eac479a\") " pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" Oct 30 14:05:07.543954 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:07.543928 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxcj5\" (UniqueName: \"kubernetes.io/projected/cc3a3a28-d723-459a-93d0-ff115eac479a-kube-api-access-sxcj5\") pod \"simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9\" (UID: \"cc3a3a28-d723-459a-93d0-ff115eac479a\") " pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" Oct 30 14:05:07.544016 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:07.544004 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc3a3a28-d723-459a-93d0-ff115eac479a-tmp-dir\") pod \"simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9\" (UID: \"cc3a3a28-d723-459a-93d0-ff115eac479a\") " pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" Oct 30 14:05:07.544092 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:07.544077 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc3a3a28-d723-459a-93d0-ff115eac479a-model-cache\") pod \"simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9\" (UID: \"cc3a3a28-d723-459a-93d0-ff115eac479a\") " pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" Oct 30 14:05:07.544135 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:07.544108 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc3a3a28-d723-459a-93d0-ff115eac479a-tls-certs\") pod \"simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9\" (UID: \"cc3a3a28-d723-459a-93d0-ff115eac479a\") " pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" Oct 30 14:05:07.544191 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:07.544143 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc3a3a28-d723-459a-93d0-ff115eac479a-home\") pod \"simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9\" (UID: \"cc3a3a28-d723-459a-93d0-ff115eac479a\") " pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" Oct 30 14:05:07.575149 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:07.575086 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:07.645453 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:07.645392 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxcj5\" (UniqueName: \"kubernetes.io/projected/cc3a3a28-d723-459a-93d0-ff115eac479a-kube-api-access-sxcj5\") pod \"simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9\" (UID: \"cc3a3a28-d723-459a-93d0-ff115eac479a\") " pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" Oct 30 14:05:07.645961 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:07.645490 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc3a3a28-d723-459a-93d0-ff115eac479a-tmp-dir\") pod \"simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9\" (UID: \"cc3a3a28-d723-459a-93d0-ff115eac479a\") " pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" Oct 30 14:05:07.645961 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:07.645540 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc3a3a28-d723-459a-93d0-ff115eac479a-model-cache\") pod \"simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9\" (UID: \"cc3a3a28-d723-459a-93d0-ff115eac479a\") " pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" Oct 30 14:05:07.645961 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:07.645567 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc3a3a28-d723-459a-93d0-ff115eac479a-tls-certs\") pod \"simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9\" (UID: \"cc3a3a28-d723-459a-93d0-ff115eac479a\") " pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" Oct 30 14:05:07.645961 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:07.645599 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc3a3a28-d723-459a-93d0-ff115eac479a-home\") pod \"simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9\" (UID: \"cc3a3a28-d723-459a-93d0-ff115eac479a\") " pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" Oct 30 14:05:07.645961 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:07.645688 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc3a3a28-d723-459a-93d0-ff115eac479a-dshm\") pod \"simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9\" (UID: \"cc3a3a28-d723-459a-93d0-ff115eac479a\") " pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" Oct 30 14:05:07.646237 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:07.646092 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc3a3a28-d723-459a-93d0-ff115eac479a-tmp-dir\") pod \"simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9\" (UID: \"cc3a3a28-d723-459a-93d0-ff115eac479a\") " pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" Oct 30 14:05:07.646237 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:07.646157 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc3a3a28-d723-459a-93d0-ff115eac479a-home\") pod \"simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9\" (UID: \"cc3a3a28-d723-459a-93d0-ff115eac479a\") " pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" Oct 30 14:05:07.646370 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:07.646344 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc3a3a28-d723-459a-93d0-ff115eac479a-model-cache\") pod \"simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9\" (UID: \"cc3a3a28-d723-459a-93d0-ff115eac479a\") " pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" Oct 30 14:05:07.649079 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:07.649053 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc3a3a28-d723-459a-93d0-ff115eac479a-dshm\") pod \"simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9\" (UID: \"cc3a3a28-d723-459a-93d0-ff115eac479a\") " pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" Oct 30 14:05:07.649303 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:07.649282 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc3a3a28-d723-459a-93d0-ff115eac479a-tls-certs\") pod \"simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9\" (UID: \"cc3a3a28-d723-459a-93d0-ff115eac479a\") " pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" Oct 30 14:05:07.671047 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:07.670981 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxcj5\" (UniqueName: \"kubernetes.io/projected/cc3a3a28-d723-459a-93d0-ff115eac479a-kube-api-access-sxcj5\") pod \"simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9\" (UID: \"cc3a3a28-d723-459a-93d0-ff115eac479a\") " pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" Oct 30 14:05:07.735039 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:07.735006 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" Oct 30 14:05:08.035881 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:08.035849 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9"] Oct 30 14:05:08.036605 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:05:08.036582 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcc3a3a28_d723_459a_93d0_ff115eac479a.slice/crio-26bb3474e07bc23b7dfd3208b530d26739d9d9d8f95de226cfe3d9b51b56bdef WatchSource:0}: Error finding container 26bb3474e07bc23b7dfd3208b530d26739d9d9d8f95de226cfe3d9b51b56bdef: Status 404 returned error can't find the container with id 26bb3474e07bc23b7dfd3208b530d26739d9d9d8f95de226cfe3d9b51b56bdef Oct 30 14:05:08.255340 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:08.255300 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:08.585140 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:08.585102 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" event={"ID":"cc3a3a28-d723-459a-93d0-ff115eac479a","Type":"ContainerStarted","Data":"7a58d3b4cf4311a3f1456c31660222e83e69f0f3555a552a5ae5d549a85e8a5b"} Oct 30 14:05:08.585338 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:08.585148 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" event={"ID":"cc3a3a28-d723-459a-93d0-ff115eac479a","Type":"ContainerStarted","Data":"26bb3474e07bc23b7dfd3208b530d26739d9d9d8f95de226cfe3d9b51b56bdef"} Oct 30 14:05:08.631257 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:08.631206 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" podStartSLOduration=1.631191518 podStartE2EDuration="1.631191518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:05:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:05:08.629206265 +0000 UTC m=+978.373340307" watchObservedRunningTime="2026-10-30 14:05:08.631191518 +0000 UTC m=+978.375325560" Oct 30 14:05:08.705789 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:08.705754 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:08.735747 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:08.735712 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" Oct 30 14:05:08.849692 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:08.849569 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:08.995815 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:08.995783 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:09.150095 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:09.150020 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:09.314155 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:09.314111 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:09.471270 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:09.471174 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:09.636225 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:09.636189 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:10.990382 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:10.990352 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:11.347751 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:11.347716 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:13.203581 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:13.203552 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:15.024529 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:15.024494 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:16.965733 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:16.965695 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:17.736093 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:17.736051 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" Oct 30 14:05:17.749219 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:17.749183 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" Oct 30 14:05:18.262438 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:18.262403 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7"] Oct 30 14:05:18.267007 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:18.266984 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" Oct 30 14:05:18.287879 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:18.287846 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7"] Oct 30 14:05:18.299160 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:18.299121 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-39529e\"/\"openshift-service-ca.crt\"" Oct 30 14:05:18.299160 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:18.299137 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-39529e\"/\"default-dockercfg-xxqx5\"" Oct 30 14:05:18.299160 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:18.299138 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-39529e\"/\"kube-root-ca.crt\"" Oct 30 14:05:18.299401 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:18.299229 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-39529e\"/\"simulator-w6-39529e-kserve-self-signed-certs\"" Oct 30 14:05:18.347181 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:18.347138 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab822fe6-4349-4051-b2fd-c6e5ceb50d08-tmp-dir\") pod \"simulator-w6-39529e-kserve-77b9765c58-jntm7\" (UID: \"ab822fe6-4349-4051-b2fd-c6e5ceb50d08\") " pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" Oct 30 14:05:18.347366 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:18.347199 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4sf7\" (UniqueName: \"kubernetes.io/projected/ab822fe6-4349-4051-b2fd-c6e5ceb50d08-kube-api-access-w4sf7\") pod \"simulator-w6-39529e-kserve-77b9765c58-jntm7\" (UID: \"ab822fe6-4349-4051-b2fd-c6e5ceb50d08\") " pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" Oct 30 14:05:18.347366 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:18.347335 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab822fe6-4349-4051-b2fd-c6e5ceb50d08-home\") pod \"simulator-w6-39529e-kserve-77b9765c58-jntm7\" (UID: \"ab822fe6-4349-4051-b2fd-c6e5ceb50d08\") " pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" Oct 30 14:05:18.347443 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:18.347427 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab822fe6-4349-4051-b2fd-c6e5ceb50d08-dshm\") pod \"simulator-w6-39529e-kserve-77b9765c58-jntm7\" (UID: \"ab822fe6-4349-4051-b2fd-c6e5ceb50d08\") " pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" Oct 30 14:05:18.347498 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:18.347482 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab822fe6-4349-4051-b2fd-c6e5ceb50d08-tls-certs\") pod \"simulator-w6-39529e-kserve-77b9765c58-jntm7\" (UID: \"ab822fe6-4349-4051-b2fd-c6e5ceb50d08\") " pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" Oct 30 14:05:18.347552 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:18.347541 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab822fe6-4349-4051-b2fd-c6e5ceb50d08-model-cache\") pod \"simulator-w6-39529e-kserve-77b9765c58-jntm7\" (UID: \"ab822fe6-4349-4051-b2fd-c6e5ceb50d08\") " pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" Oct 30 14:05:18.351285 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:18.351255 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:18.448393 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:18.448349 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab822fe6-4349-4051-b2fd-c6e5ceb50d08-tmp-dir\") pod \"simulator-w6-39529e-kserve-77b9765c58-jntm7\" (UID: \"ab822fe6-4349-4051-b2fd-c6e5ceb50d08\") " pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" Oct 30 14:05:18.448588 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:18.448424 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4sf7\" (UniqueName: \"kubernetes.io/projected/ab822fe6-4349-4051-b2fd-c6e5ceb50d08-kube-api-access-w4sf7\") pod \"simulator-w6-39529e-kserve-77b9765c58-jntm7\" (UID: \"ab822fe6-4349-4051-b2fd-c6e5ceb50d08\") " pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" Oct 30 14:05:18.448588 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:18.448453 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab822fe6-4349-4051-b2fd-c6e5ceb50d08-home\") pod \"simulator-w6-39529e-kserve-77b9765c58-jntm7\" (UID: \"ab822fe6-4349-4051-b2fd-c6e5ceb50d08\") " pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" Oct 30 14:05:18.448588 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:18.448507 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab822fe6-4349-4051-b2fd-c6e5ceb50d08-dshm\") pod \"simulator-w6-39529e-kserve-77b9765c58-jntm7\" (UID: \"ab822fe6-4349-4051-b2fd-c6e5ceb50d08\") " pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" Oct 30 14:05:18.448588 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:18.448561 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab822fe6-4349-4051-b2fd-c6e5ceb50d08-tls-certs\") pod \"simulator-w6-39529e-kserve-77b9765c58-jntm7\" (UID: \"ab822fe6-4349-4051-b2fd-c6e5ceb50d08\") " pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" Oct 30 14:05:18.448838 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:18.448618 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab822fe6-4349-4051-b2fd-c6e5ceb50d08-model-cache\") pod \"simulator-w6-39529e-kserve-77b9765c58-jntm7\" (UID: \"ab822fe6-4349-4051-b2fd-c6e5ceb50d08\") " pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" Oct 30 14:05:18.448838 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:18.448784 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab822fe6-4349-4051-b2fd-c6e5ceb50d08-tmp-dir\") pod \"simulator-w6-39529e-kserve-77b9765c58-jntm7\" (UID: \"ab822fe6-4349-4051-b2fd-c6e5ceb50d08\") " pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" Oct 30 14:05:18.449056 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:18.449025 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab822fe6-4349-4051-b2fd-c6e5ceb50d08-model-cache\") pod \"simulator-w6-39529e-kserve-77b9765c58-jntm7\" (UID: \"ab822fe6-4349-4051-b2fd-c6e5ceb50d08\") " pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" Oct 30 14:05:18.449196 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:18.449115 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab822fe6-4349-4051-b2fd-c6e5ceb50d08-home\") pod \"simulator-w6-39529e-kserve-77b9765c58-jntm7\" (UID: \"ab822fe6-4349-4051-b2fd-c6e5ceb50d08\") " pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" Oct 30 14:05:18.450878 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:18.450852 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab822fe6-4349-4051-b2fd-c6e5ceb50d08-dshm\") pod \"simulator-w6-39529e-kserve-77b9765c58-jntm7\" (UID: \"ab822fe6-4349-4051-b2fd-c6e5ceb50d08\") " pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" Oct 30 14:05:18.451168 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:18.451151 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab822fe6-4349-4051-b2fd-c6e5ceb50d08-tls-certs\") pod \"simulator-w6-39529e-kserve-77b9765c58-jntm7\" (UID: \"ab822fe6-4349-4051-b2fd-c6e5ceb50d08\") " pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" Oct 30 14:05:18.474143 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:18.474114 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4sf7\" (UniqueName: \"kubernetes.io/projected/ab822fe6-4349-4051-b2fd-c6e5ceb50d08-kube-api-access-w4sf7\") pod \"simulator-w6-39529e-kserve-77b9765c58-jntm7\" (UID: \"ab822fe6-4349-4051-b2fd-c6e5ceb50d08\") " pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" Oct 30 14:05:18.580748 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:18.580714 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" Oct 30 14:05:18.664037 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:18.664002 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" Oct 30 14:05:18.883803 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:18.883580 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:18.907069 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:18.907038 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7"] Oct 30 14:05:18.908301 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:05:18.908276 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podab822fe6_4349_4051_b2fd_c6e5ceb50d08.slice/crio-7bef8037bddf6276f8cb8759524bba6744408690479b27c8d76a23e1588f51f6 WatchSource:0}: Error finding container 7bef8037bddf6276f8cb8759524bba6744408690479b27c8d76a23e1588f51f6: Status 404 returned error can't find the container with id 7bef8037bddf6276f8cb8759524bba6744408690479b27c8d76a23e1588f51f6 Oct 30 14:05:19.651142 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:19.651102 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" event={"ID":"ab822fe6-4349-4051-b2fd-c6e5ceb50d08","Type":"ContainerStarted","Data":"5176658323e9e61440288595d8b112cebf5264072222b1a68a58ade511c01c6d"} Oct 30 14:05:19.651585 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:19.651149 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" event={"ID":"ab822fe6-4349-4051-b2fd-c6e5ceb50d08","Type":"ContainerStarted","Data":"7bef8037bddf6276f8cb8759524bba6744408690479b27c8d76a23e1588f51f6"} Oct 30 14:05:19.696849 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:19.696802 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" podStartSLOduration=1.696789479 podStartE2EDuration="1.696789479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:05:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:05:19.6961386 +0000 UTC m=+989.440272645" watchObservedRunningTime="2026-10-30 14:05:19.696789479 +0000 UTC m=+989.440923520" Oct 30 14:05:20.580995 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:20.580962 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" Oct 30 14:05:20.672818 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:20.672782 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:21.458562 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:21.458525 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:22.084869 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:22.084827 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:22.234616 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:22.234579 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:22.379021 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:22.378931 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:22.737270 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:22.737179 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd"] Oct 30 14:05:22.741246 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:22.741216 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" Oct 30 14:05:22.764173 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:22.763662 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd"] Oct 30 14:05:22.774846 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:22.774822 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-a7bb0c\"/\"default-dockercfg-jhpvm\"" Oct 30 14:05:22.774988 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:22.774922 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-a7bb0c\"/\"simulator-w5-a7bb0c-kserve-self-signed-certs\"" Oct 30 14:05:22.781825 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:22.781801 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-a7bb0c\"/\"kube-root-ca.crt\"" Oct 30 14:05:22.781950 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:22.781899 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-a7bb0c\"/\"openshift-service-ca.crt\"" Oct 30 14:05:22.842286 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:22.842258 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:22.890577 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:22.890551 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15e7b7f9-2570-410a-a398-4c07c2e465ea-tmp-dir\") pod \"simulator-w5-a7bb0c-kserve-6b48f65659-p68vd\" (UID: \"15e7b7f9-2570-410a-a398-4c07c2e465ea\") " pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" Oct 30 14:05:22.890748 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:22.890598 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15e7b7f9-2570-410a-a398-4c07c2e465ea-home\") pod \"simulator-w5-a7bb0c-kserve-6b48f65659-p68vd\" (UID: \"15e7b7f9-2570-410a-a398-4c07c2e465ea\") " pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" Oct 30 14:05:22.890748 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:22.890618 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15e7b7f9-2570-410a-a398-4c07c2e465ea-model-cache\") pod \"simulator-w5-a7bb0c-kserve-6b48f65659-p68vd\" (UID: \"15e7b7f9-2570-410a-a398-4c07c2e465ea\") " pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" Oct 30 14:05:22.890869 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:22.890776 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl8b2\" (UniqueName: \"kubernetes.io/projected/15e7b7f9-2570-410a-a398-4c07c2e465ea-kube-api-access-nl8b2\") pod \"simulator-w5-a7bb0c-kserve-6b48f65659-p68vd\" (UID: \"15e7b7f9-2570-410a-a398-4c07c2e465ea\") " pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" Oct 30 14:05:22.890869 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:22.890835 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15e7b7f9-2570-410a-a398-4c07c2e465ea-tls-certs\") pod \"simulator-w5-a7bb0c-kserve-6b48f65659-p68vd\" (UID: \"15e7b7f9-2570-410a-a398-4c07c2e465ea\") " pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" Oct 30 14:05:22.890971 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:22.890867 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15e7b7f9-2570-410a-a398-4c07c2e465ea-dshm\") pod \"simulator-w5-a7bb0c-kserve-6b48f65659-p68vd\" (UID: \"15e7b7f9-2570-410a-a398-4c07c2e465ea\") " pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" Oct 30 14:05:22.994983 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:22.992385 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15e7b7f9-2570-410a-a398-4c07c2e465ea-home\") pod \"simulator-w5-a7bb0c-kserve-6b48f65659-p68vd\" (UID: \"15e7b7f9-2570-410a-a398-4c07c2e465ea\") " pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" Oct 30 14:05:22.994983 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:22.992438 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15e7b7f9-2570-410a-a398-4c07c2e465ea-model-cache\") pod \"simulator-w5-a7bb0c-kserve-6b48f65659-p68vd\" (UID: \"15e7b7f9-2570-410a-a398-4c07c2e465ea\") " pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" Oct 30 14:05:22.994983 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:22.992534 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl8b2\" (UniqueName: \"kubernetes.io/projected/15e7b7f9-2570-410a-a398-4c07c2e465ea-kube-api-access-nl8b2\") pod \"simulator-w5-a7bb0c-kserve-6b48f65659-p68vd\" (UID: \"15e7b7f9-2570-410a-a398-4c07c2e465ea\") " pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" Oct 30 14:05:22.994983 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:22.992610 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15e7b7f9-2570-410a-a398-4c07c2e465ea-tls-certs\") pod \"simulator-w5-a7bb0c-kserve-6b48f65659-p68vd\" (UID: \"15e7b7f9-2570-410a-a398-4c07c2e465ea\") " pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" Oct 30 14:05:22.994983 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:22.992635 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15e7b7f9-2570-410a-a398-4c07c2e465ea-dshm\") pod \"simulator-w5-a7bb0c-kserve-6b48f65659-p68vd\" (UID: \"15e7b7f9-2570-410a-a398-4c07c2e465ea\") " pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" Oct 30 14:05:22.994983 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:22.992769 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15e7b7f9-2570-410a-a398-4c07c2e465ea-tmp-dir\") pod \"simulator-w5-a7bb0c-kserve-6b48f65659-p68vd\" (UID: \"15e7b7f9-2570-410a-a398-4c07c2e465ea\") " pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" Oct 30 14:05:22.994983 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:22.993242 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15e7b7f9-2570-410a-a398-4c07c2e465ea-tmp-dir\") pod \"simulator-w5-a7bb0c-kserve-6b48f65659-p68vd\" (UID: \"15e7b7f9-2570-410a-a398-4c07c2e465ea\") " pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" Oct 30 14:05:22.994983 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:22.993543 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15e7b7f9-2570-410a-a398-4c07c2e465ea-home\") pod \"simulator-w5-a7bb0c-kserve-6b48f65659-p68vd\" (UID: \"15e7b7f9-2570-410a-a398-4c07c2e465ea\") " pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" Oct 30 14:05:22.994983 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:22.993840 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15e7b7f9-2570-410a-a398-4c07c2e465ea-model-cache\") pod \"simulator-w5-a7bb0c-kserve-6b48f65659-p68vd\" (UID: \"15e7b7f9-2570-410a-a398-4c07c2e465ea\") " pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" Oct 30 14:05:22.997429 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:22.997401 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15e7b7f9-2570-410a-a398-4c07c2e465ea-tls-certs\") pod \"simulator-w5-a7bb0c-kserve-6b48f65659-p68vd\" (UID: \"15e7b7f9-2570-410a-a398-4c07c2e465ea\") " pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" Oct 30 14:05:23.000740 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:23.000287 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15e7b7f9-2570-410a-a398-4c07c2e465ea-dshm\") pod \"simulator-w5-a7bb0c-kserve-6b48f65659-p68vd\" (UID: \"15e7b7f9-2570-410a-a398-4c07c2e465ea\") " pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" Oct 30 14:05:23.020237 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:23.020206 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl8b2\" (UniqueName: \"kubernetes.io/projected/15e7b7f9-2570-410a-a398-4c07c2e465ea-kube-api-access-nl8b2\") pod \"simulator-w5-a7bb0c-kserve-6b48f65659-p68vd\" (UID: \"15e7b7f9-2570-410a-a398-4c07c2e465ea\") " pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" Oct 30 14:05:23.061333 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:23.061307 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" Oct 30 14:05:23.064678 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:23.064547 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:23.382673 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:23.367660 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd"] Oct 30 14:05:23.382673 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:05:23.373755 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod15e7b7f9_2570_410a_a398_4c07c2e465ea.slice/crio-9f60ca8e065d8af1e8dc1a4e9d2d4cfe2f1f1df4ba78c16e1539bc47c9a461fa WatchSource:0}: Error finding container 9f60ca8e065d8af1e8dc1a4e9d2d4cfe2f1f1df4ba78c16e1539bc47c9a461fa: Status 404 returned error can't find the container with id 9f60ca8e065d8af1e8dc1a4e9d2d4cfe2f1f1df4ba78c16e1539bc47c9a461fa Oct 30 14:05:23.672757 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:23.672669 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" event={"ID":"15e7b7f9-2570-410a-a398-4c07c2e465ea","Type":"ContainerStarted","Data":"170fae7c862522701e03c398e0da08f9e9ab01e2937569bbb2b821605438a838"} Oct 30 14:05:23.672757 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:23.672716 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" event={"ID":"15e7b7f9-2570-410a-a398-4c07c2e465ea","Type":"ContainerStarted","Data":"9f60ca8e065d8af1e8dc1a4e9d2d4cfe2f1f1df4ba78c16e1539bc47c9a461fa"} Oct 30 14:05:23.719720 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:23.719671 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" podStartSLOduration=1.7196508339999999 podStartE2EDuration="1.719650834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:05:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:05:23.718606492 +0000 UTC m=+993.462740535" watchObservedRunningTime="2026-10-30 14:05:23.719650834 +0000 UTC m=+993.463784880" Oct 30 14:05:24.062055 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:24.062021 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" Oct 30 14:05:24.759031 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:24.758987 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:24.962084 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:24.962056 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:26.695909 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:26.695869 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:26.967995 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:26.967915 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:28.011133 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:05:28.011094 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:05:28.581608 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:28.581575 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" Oct 30 14:05:28.594030 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:28.594001 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" Oct 30 14:05:28.709416 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:28.709386 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" Oct 30 14:05:28.847857 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:28.847789 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:30.754226 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:30.754191 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:32.732110 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:32.732078 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:33.062399 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:33.062367 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" Oct 30 14:05:33.075349 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:33.075327 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" Oct 30 14:05:33.737963 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:33.737934 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" Oct 30 14:05:34.515259 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:34.515224 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:36.471817 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:36.471779 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:37.480352 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:37.480318 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8"] Oct 30 14:05:37.485131 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:37.485110 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8" Oct 30 14:05:37.505143 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:37.505115 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8"] Oct 30 14:05:37.516528 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:37.516503 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-ca229c\"/\"kube-root-ca.crt\"" Oct 30 14:05:37.516665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:37.516539 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-ca229c\"/\"openshift-service-ca.crt\"" Oct 30 14:05:37.516665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:37.516660 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-ca229c\"/\"default-dockercfg-499ms\"" Oct 30 14:05:37.516798 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:37.516720 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-ca229c\"/\"simulator-w1-ca229c-kserve-self-signed-certs\"" Oct 30 14:05:37.574289 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:37.574264 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:37.642647 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:37.642604 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3093abe-9b1e-492a-8837-4bd42ea39d2d-model-cache\") pod \"simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8\" (UID: \"f3093abe-9b1e-492a-8837-4bd42ea39d2d\") " pod="e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8" Oct 30 14:05:37.642783 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:37.642669 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3093abe-9b1e-492a-8837-4bd42ea39d2d-tls-certs\") pod \"simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8\" (UID: \"f3093abe-9b1e-492a-8837-4bd42ea39d2d\") " pod="e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8" Oct 30 14:05:37.642783 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:37.642726 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3093abe-9b1e-492a-8837-4bd42ea39d2d-dshm\") pod \"simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8\" (UID: \"f3093abe-9b1e-492a-8837-4bd42ea39d2d\") " pod="e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8" Oct 30 14:05:37.642783 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:37.642760 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3093abe-9b1e-492a-8837-4bd42ea39d2d-tmp-dir\") pod \"simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8\" (UID: \"f3093abe-9b1e-492a-8837-4bd42ea39d2d\") " pod="e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8" Oct 30 14:05:37.642783 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:37.642778 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3093abe-9b1e-492a-8837-4bd42ea39d2d-home\") pod \"simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8\" (UID: \"f3093abe-9b1e-492a-8837-4bd42ea39d2d\") " pod="e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8" Oct 30 14:05:37.642931 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:37.642798 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncp2n\" (UniqueName: \"kubernetes.io/projected/f3093abe-9b1e-492a-8837-4bd42ea39d2d-kube-api-access-ncp2n\") pod \"simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8\" (UID: \"f3093abe-9b1e-492a-8837-4bd42ea39d2d\") " pod="e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8" Oct 30 14:05:37.743690 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:37.743593 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3093abe-9b1e-492a-8837-4bd42ea39d2d-model-cache\") pod \"simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8\" (UID: \"f3093abe-9b1e-492a-8837-4bd42ea39d2d\") " pod="e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8" Oct 30 14:05:37.743690 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:37.743653 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3093abe-9b1e-492a-8837-4bd42ea39d2d-tls-certs\") pod \"simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8\" (UID: \"f3093abe-9b1e-492a-8837-4bd42ea39d2d\") " pod="e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8" Oct 30 14:05:37.743690 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:37.743681 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3093abe-9b1e-492a-8837-4bd42ea39d2d-dshm\") pod \"simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8\" (UID: \"f3093abe-9b1e-492a-8837-4bd42ea39d2d\") " pod="e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8" Oct 30 14:05:37.743977 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:37.743729 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3093abe-9b1e-492a-8837-4bd42ea39d2d-tmp-dir\") pod \"simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8\" (UID: \"f3093abe-9b1e-492a-8837-4bd42ea39d2d\") " pod="e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8" Oct 30 14:05:37.743977 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:37.743753 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3093abe-9b1e-492a-8837-4bd42ea39d2d-home\") pod \"simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8\" (UID: \"f3093abe-9b1e-492a-8837-4bd42ea39d2d\") " pod="e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8" Oct 30 14:05:37.743977 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:37.743777 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncp2n\" (UniqueName: \"kubernetes.io/projected/f3093abe-9b1e-492a-8837-4bd42ea39d2d-kube-api-access-ncp2n\") pod \"simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8\" (UID: \"f3093abe-9b1e-492a-8837-4bd42ea39d2d\") " pod="e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8" Oct 30 14:05:37.744131 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:37.744026 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3093abe-9b1e-492a-8837-4bd42ea39d2d-model-cache\") pod \"simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8\" (UID: \"f3093abe-9b1e-492a-8837-4bd42ea39d2d\") " pod="e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8" Oct 30 14:05:37.744425 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:37.744399 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3093abe-9b1e-492a-8837-4bd42ea39d2d-tmp-dir\") pod \"simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8\" (UID: \"f3093abe-9b1e-492a-8837-4bd42ea39d2d\") " pod="e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8" Oct 30 14:05:37.744614 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:37.744592 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3093abe-9b1e-492a-8837-4bd42ea39d2d-home\") pod \"simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8\" (UID: \"f3093abe-9b1e-492a-8837-4bd42ea39d2d\") " pod="e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8" Oct 30 14:05:37.746585 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:37.746559 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3093abe-9b1e-492a-8837-4bd42ea39d2d-dshm\") pod \"simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8\" (UID: \"f3093abe-9b1e-492a-8837-4bd42ea39d2d\") " pod="e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8" Oct 30 14:05:37.747034 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:37.747007 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3093abe-9b1e-492a-8837-4bd42ea39d2d-tls-certs\") pod \"simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8\" (UID: \"f3093abe-9b1e-492a-8837-4bd42ea39d2d\") " pod="e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8" Oct 30 14:05:37.769842 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:37.769819 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncp2n\" (UniqueName: \"kubernetes.io/projected/f3093abe-9b1e-492a-8837-4bd42ea39d2d-kube-api-access-ncp2n\") pod \"simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8\" (UID: \"f3093abe-9b1e-492a-8837-4bd42ea39d2d\") " pod="e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8" Oct 30 14:05:37.800791 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:37.800770 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8" Oct 30 14:05:38.102888 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:38.102860 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8"] Oct 30 14:05:38.104597 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:05:38.104568 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf3093abe_9b1e_492a_8837_4bd42ea39d2d.slice/crio-2e14dda523d100aa807d63035114e4a81c12777d9ada6aa87fa74d46c32144ef WatchSource:0}: Error finding container 2e14dda523d100aa807d63035114e4a81c12777d9ada6aa87fa74d46c32144ef: Status 404 returned error can't find the container with id 2e14dda523d100aa807d63035114e4a81c12777d9ada6aa87fa74d46c32144ef Oct 30 14:05:38.368035 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:38.367958 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:38.756063 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:38.755966 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8" event={"ID":"f3093abe-9b1e-492a-8837-4bd42ea39d2d","Type":"ContainerStarted","Data":"de198dcfa2afe64c56d7e44af73543b23100debca344cf2a95b7aecc602f296e"} Oct 30 14:05:38.756063 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:38.756020 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8" event={"ID":"f3093abe-9b1e-492a-8837-4bd42ea39d2d","Type":"ContainerStarted","Data":"2e14dda523d100aa807d63035114e4a81c12777d9ada6aa87fa74d46c32144ef"} Oct 30 14:05:38.800892 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:38.800856 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8" Oct 30 14:05:38.802098 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:38.802055 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8" podStartSLOduration=1.802035171 podStartE2EDuration="1.802035171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:05:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:05:38.800737322 +0000 UTC m=+1008.544871393" watchObservedRunningTime="2026-10-30 14:05:38.802035171 +0000 UTC m=+1008.546169214" Oct 30 14:05:39.194853 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:39.194818 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4"] Oct 30 14:05:39.198343 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:39.198321 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4" Oct 30 14:05:39.231288 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:39.231255 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4"] Oct 30 14:05:39.259360 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:39.259324 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1633499-c0e2-4c59-b509-8c8937bc8b49-credential-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4\" (UID: \"e1633499-c0e2-4c59-b509-8c8937bc8b49\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4" Oct 30 14:05:39.259478 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:39.259406 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1633499-c0e2-4c59-b509-8c8937bc8b49-istio-data\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4\" (UID: \"e1633499-c0e2-4c59-b509-8c8937bc8b49\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4" Oct 30 14:05:39.259523 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:39.259484 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bns85\" (UniqueName: \"kubernetes.io/projected/e1633499-c0e2-4c59-b509-8c8937bc8b49-kube-api-access-bns85\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4\" (UID: \"e1633499-c0e2-4c59-b509-8c8937bc8b49\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4" Oct 30 14:05:39.259523 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:39.259507 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1633499-c0e2-4c59-b509-8c8937bc8b49-istio-envoy\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4\" (UID: \"e1633499-c0e2-4c59-b509-8c8937bc8b49\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4" Oct 30 14:05:39.259625 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:39.259603 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1633499-c0e2-4c59-b509-8c8937bc8b49-workload-certs\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4\" (UID: \"e1633499-c0e2-4c59-b509-8c8937bc8b49\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4" Oct 30 14:05:39.259690 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:39.259678 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1633499-c0e2-4c59-b509-8c8937bc8b49-istio-podinfo\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4\" (UID: \"e1633499-c0e2-4c59-b509-8c8937bc8b49\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4" Oct 30 14:05:39.259731 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:39.259720 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1633499-c0e2-4c59-b509-8c8937bc8b49-workload-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4\" (UID: \"e1633499-c0e2-4c59-b509-8c8937bc8b49\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4" Oct 30 14:05:39.259784 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:39.259748 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1633499-c0e2-4c59-b509-8c8937bc8b49-istio-token\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4\" (UID: \"e1633499-c0e2-4c59-b509-8c8937bc8b49\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4" Oct 30 14:05:39.259876 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:39.259861 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1633499-c0e2-4c59-b509-8c8937bc8b49-istiod-ca-cert\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4\" (UID: \"e1633499-c0e2-4c59-b509-8c8937bc8b49\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4" Oct 30 14:05:39.360665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:39.360598 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1633499-c0e2-4c59-b509-8c8937bc8b49-workload-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4\" (UID: \"e1633499-c0e2-4c59-b509-8c8937bc8b49\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4" Oct 30 14:05:39.360853 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:39.360666 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1633499-c0e2-4c59-b509-8c8937bc8b49-istio-token\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4\" (UID: \"e1633499-c0e2-4c59-b509-8c8937bc8b49\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4" Oct 30 14:05:39.360853 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:39.360716 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1633499-c0e2-4c59-b509-8c8937bc8b49-istiod-ca-cert\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4\" (UID: \"e1633499-c0e2-4c59-b509-8c8937bc8b49\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4" Oct 30 14:05:39.360853 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:39.360741 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1633499-c0e2-4c59-b509-8c8937bc8b49-credential-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4\" (UID: \"e1633499-c0e2-4c59-b509-8c8937bc8b49\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4" Oct 30 14:05:39.360853 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:39.360811 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1633499-c0e2-4c59-b509-8c8937bc8b49-istio-data\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4\" (UID: \"e1633499-c0e2-4c59-b509-8c8937bc8b49\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4" Oct 30 14:05:39.361069 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:39.360854 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bns85\" (UniqueName: \"kubernetes.io/projected/e1633499-c0e2-4c59-b509-8c8937bc8b49-kube-api-access-bns85\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4\" (UID: \"e1633499-c0e2-4c59-b509-8c8937bc8b49\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4" Oct 30 14:05:39.361069 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:39.360888 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1633499-c0e2-4c59-b509-8c8937bc8b49-istio-envoy\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4\" (UID: \"e1633499-c0e2-4c59-b509-8c8937bc8b49\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4" Oct 30 14:05:39.361069 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:39.360962 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1633499-c0e2-4c59-b509-8c8937bc8b49-workload-certs\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4\" (UID: \"e1633499-c0e2-4c59-b509-8c8937bc8b49\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4" Oct 30 14:05:39.361069 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:39.361017 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1633499-c0e2-4c59-b509-8c8937bc8b49-istio-podinfo\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4\" (UID: \"e1633499-c0e2-4c59-b509-8c8937bc8b49\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4" Oct 30 14:05:39.361261 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:39.361172 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1633499-c0e2-4c59-b509-8c8937bc8b49-credential-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4\" (UID: \"e1633499-c0e2-4c59-b509-8c8937bc8b49\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4" Oct 30 14:05:39.361261 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:39.361232 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1633499-c0e2-4c59-b509-8c8937bc8b49-istio-data\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4\" (UID: \"e1633499-c0e2-4c59-b509-8c8937bc8b49\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4" Oct 30 14:05:39.361454 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:39.361420 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1633499-c0e2-4c59-b509-8c8937bc8b49-workload-certs\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4\" (UID: \"e1633499-c0e2-4c59-b509-8c8937bc8b49\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4" Oct 30 14:05:39.361687 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:39.361664 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1633499-c0e2-4c59-b509-8c8937bc8b49-workload-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4\" (UID: \"e1633499-c0e2-4c59-b509-8c8937bc8b49\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4" Oct 30 14:05:39.361794 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:39.361759 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1633499-c0e2-4c59-b509-8c8937bc8b49-istiod-ca-cert\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4\" (UID: \"e1633499-c0e2-4c59-b509-8c8937bc8b49\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4" Oct 30 14:05:39.363829 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:39.363806 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1633499-c0e2-4c59-b509-8c8937bc8b49-istio-envoy\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4\" (UID: \"e1633499-c0e2-4c59-b509-8c8937bc8b49\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4" Oct 30 14:05:39.363960 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:39.363940 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1633499-c0e2-4c59-b509-8c8937bc8b49-istio-podinfo\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4\" (UID: \"e1633499-c0e2-4c59-b509-8c8937bc8b49\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4" Oct 30 14:05:39.386466 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:39.386445 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1633499-c0e2-4c59-b509-8c8937bc8b49-istio-token\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4\" (UID: \"e1633499-c0e2-4c59-b509-8c8937bc8b49\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4" Oct 30 14:05:39.386633 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:39.386612 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bns85\" (UniqueName: \"kubernetes.io/projected/e1633499-c0e2-4c59-b509-8c8937bc8b49-kube-api-access-bns85\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4\" (UID: \"e1633499-c0e2-4c59-b509-8c8937bc8b49\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4" Oct 30 14:05:39.513250 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:39.513161 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4" Oct 30 14:05:39.848687 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:39.836363 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4"] Oct 30 14:05:39.848687 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:05:39.837602 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1633499_c0e2_4c59_b509_8c8937bc8b49.slice/crio-f6fd85fca1f20e35124c070b2721ab8b60bffeffe5ed27e15818e5e8cb86cd14 WatchSource:0}: Error finding container f6fd85fca1f20e35124c070b2721ab8b60bffeffe5ed27e15818e5e8cb86cd14: Status 404 returned error can't find the container with id f6fd85fca1f20e35124c070b2721ab8b60bffeffe5ed27e15818e5e8cb86cd14 Oct 30 14:05:39.848687 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:39.840812 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:05:39.848687 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:39.840903 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:05:39.848687 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:39.840946 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:05:40.264967 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:40.264759 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:40.774084 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:40.774042 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4" event={"ID":"e1633499-c0e2-4c59-b509-8c8937bc8b49","Type":"ContainerStarted","Data":"26af3da49249009817c3c279a1cbd0b5cf6e428d2c4a74d387c03df192203a19"} Oct 30 14:05:40.774294 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:40.774108 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4" event={"ID":"e1633499-c0e2-4c59-b509-8c8937bc8b49","Type":"ContainerStarted","Data":"f6fd85fca1f20e35124c070b2721ab8b60bffeffe5ed27e15818e5e8cb86cd14"} Oct 30 14:05:40.855920 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:40.855866 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4" podStartSLOduration=1.855852139 podStartE2EDuration="1.855852139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:05:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:05:40.841604141 +0000 UTC m=+1010.585738174" watchObservedRunningTime="2026-10-30 14:05:40.855852139 +0000 UTC m=+1010.599986181" Oct 30 14:05:41.516668 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:41.515783 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4" Oct 30 14:05:41.520662 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:41.519744 4982 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 30 14:05:41.520662 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:41.519800 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4" podUID="e1633499-c0e2-4c59-b509-8c8937bc8b49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 30 14:05:41.768688 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:41.768553 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:42.514303 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:42.514260 4982 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 30 14:05:42.514861 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:42.514324 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4" podUID="e1633499-c0e2-4c59-b509-8c8937bc8b49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 30 14:05:42.540083 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:42.540051 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:43.514430 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:43.514387 4982 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 30 14:05:43.514890 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:43.514460 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4" podUID="e1633499-c0e2-4c59-b509-8c8937bc8b49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 30 14:05:44.514138 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:44.514097 4982 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 30 14:05:44.514326 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:44.514173 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4" podUID="e1633499-c0e2-4c59-b509-8c8937bc8b49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 30 14:05:44.986893 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:44.985630 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:45.513588 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:45.513547 4982 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 30 14:05:45.513964 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:45.513630 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4" podUID="e1633499-c0e2-4c59-b509-8c8937bc8b49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 30 14:05:46.517865 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:46.517832 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4" Oct 30 14:05:46.518267 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:46.518182 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4" Oct 30 14:05:46.518807 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:46.518785 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4" Oct 30 14:05:47.080119 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:47.080080 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:47.801578 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:47.801547 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8" Oct 30 14:05:47.814113 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:47.814079 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8" Oct 30 14:05:47.839534 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:47.839511 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8" Oct 30 14:05:48.878072 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:48.878036 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:49.138737 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:49.138656 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7"] Oct 30 14:05:49.144205 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:49.144180 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" Oct 30 14:05:49.174622 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:49.174589 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-39529e\"/\"premium-w6-39529e-kserve-self-signed-certs\"" Oct 30 14:05:49.177584 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:49.177557 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7"] Oct 30 14:05:49.203066 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:49.203034 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fa94d86-57ce-4d23-95d5-4efa965f6864-tls-certs\") pod \"premium-w6-39529e-kserve-65fccd67-q8dc7\" (UID: \"4fa94d86-57ce-4d23-95d5-4efa965f6864\") " pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" Oct 30 14:05:49.203199 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:49.203178 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fa94d86-57ce-4d23-95d5-4efa965f6864-model-cache\") pod \"premium-w6-39529e-kserve-65fccd67-q8dc7\" (UID: \"4fa94d86-57ce-4d23-95d5-4efa965f6864\") " pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" Oct 30 14:05:49.203267 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:49.203252 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fa94d86-57ce-4d23-95d5-4efa965f6864-tmp-dir\") pod \"premium-w6-39529e-kserve-65fccd67-q8dc7\" (UID: \"4fa94d86-57ce-4d23-95d5-4efa965f6864\") " pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" Oct 30 14:05:49.203319 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:49.203275 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fa94d86-57ce-4d23-95d5-4efa965f6864-dshm\") pod \"premium-w6-39529e-kserve-65fccd67-q8dc7\" (UID: \"4fa94d86-57ce-4d23-95d5-4efa965f6864\") " pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" Oct 30 14:05:49.203319 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:49.203300 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fa94d86-57ce-4d23-95d5-4efa965f6864-home\") pod \"premium-w6-39529e-kserve-65fccd67-q8dc7\" (UID: \"4fa94d86-57ce-4d23-95d5-4efa965f6864\") " pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" Oct 30 14:05:49.203395 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:49.203354 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nm87\" (UniqueName: \"kubernetes.io/projected/4fa94d86-57ce-4d23-95d5-4efa965f6864-kube-api-access-2nm87\") pod \"premium-w6-39529e-kserve-65fccd67-q8dc7\" (UID: \"4fa94d86-57ce-4d23-95d5-4efa965f6864\") " pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" Oct 30 14:05:49.228023 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:49.227990 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:49.304576 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:49.304534 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fa94d86-57ce-4d23-95d5-4efa965f6864-model-cache\") pod \"premium-w6-39529e-kserve-65fccd67-q8dc7\" (UID: \"4fa94d86-57ce-4d23-95d5-4efa965f6864\") " pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" Oct 30 14:05:49.304755 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:49.304606 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fa94d86-57ce-4d23-95d5-4efa965f6864-tmp-dir\") pod \"premium-w6-39529e-kserve-65fccd67-q8dc7\" (UID: \"4fa94d86-57ce-4d23-95d5-4efa965f6864\") " pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" Oct 30 14:05:49.304755 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:49.304633 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fa94d86-57ce-4d23-95d5-4efa965f6864-dshm\") pod \"premium-w6-39529e-kserve-65fccd67-q8dc7\" (UID: \"4fa94d86-57ce-4d23-95d5-4efa965f6864\") " pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" Oct 30 14:05:49.304755 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:49.304677 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fa94d86-57ce-4d23-95d5-4efa965f6864-home\") pod \"premium-w6-39529e-kserve-65fccd67-q8dc7\" (UID: \"4fa94d86-57ce-4d23-95d5-4efa965f6864\") " pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" Oct 30 14:05:49.304755 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:49.304733 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nm87\" (UniqueName: \"kubernetes.io/projected/4fa94d86-57ce-4d23-95d5-4efa965f6864-kube-api-access-2nm87\") pod \"premium-w6-39529e-kserve-65fccd67-q8dc7\" (UID: \"4fa94d86-57ce-4d23-95d5-4efa965f6864\") " pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" Oct 30 14:05:49.304980 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:49.304807 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fa94d86-57ce-4d23-95d5-4efa965f6864-tls-certs\") pod \"premium-w6-39529e-kserve-65fccd67-q8dc7\" (UID: \"4fa94d86-57ce-4d23-95d5-4efa965f6864\") " pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" Oct 30 14:05:49.305716 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:49.305689 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fa94d86-57ce-4d23-95d5-4efa965f6864-model-cache\") pod \"premium-w6-39529e-kserve-65fccd67-q8dc7\" (UID: \"4fa94d86-57ce-4d23-95d5-4efa965f6864\") " pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" Oct 30 14:05:49.306146 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:49.306123 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fa94d86-57ce-4d23-95d5-4efa965f6864-tmp-dir\") pod \"premium-w6-39529e-kserve-65fccd67-q8dc7\" (UID: \"4fa94d86-57ce-4d23-95d5-4efa965f6864\") " pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" Oct 30 14:05:49.306673 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:49.306652 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fa94d86-57ce-4d23-95d5-4efa965f6864-home\") pod \"premium-w6-39529e-kserve-65fccd67-q8dc7\" (UID: \"4fa94d86-57ce-4d23-95d5-4efa965f6864\") " pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" Oct 30 14:05:49.307984 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:49.307940 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fa94d86-57ce-4d23-95d5-4efa965f6864-dshm\") pod \"premium-w6-39529e-kserve-65fccd67-q8dc7\" (UID: \"4fa94d86-57ce-4d23-95d5-4efa965f6864\") " pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" Oct 30 14:05:49.308409 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:49.308386 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fa94d86-57ce-4d23-95d5-4efa965f6864-tls-certs\") pod \"premium-w6-39529e-kserve-65fccd67-q8dc7\" (UID: \"4fa94d86-57ce-4d23-95d5-4efa965f6864\") " pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" Oct 30 14:05:49.341369 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:49.341338 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nm87\" (UniqueName: \"kubernetes.io/projected/4fa94d86-57ce-4d23-95d5-4efa965f6864-kube-api-access-2nm87\") pod \"premium-w6-39529e-kserve-65fccd67-q8dc7\" (UID: \"4fa94d86-57ce-4d23-95d5-4efa965f6864\") " pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" Oct 30 14:05:49.455557 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:49.455477 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" Oct 30 14:05:49.762415 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:49.762381 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7"] Oct 30 14:05:49.763587 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:05:49.763561 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4fa94d86_57ce_4d23_95d5_4efa965f6864.slice/crio-9529b2dbd152577c43fa8abd5f97ab25dfd5c838563979d7312fb109ceedcf29 WatchSource:0}: Error finding container 9529b2dbd152577c43fa8abd5f97ab25dfd5c838563979d7312fb109ceedcf29: Status 404 returned error can't find the container with id 9529b2dbd152577c43fa8abd5f97ab25dfd5c838563979d7312fb109ceedcf29 Oct 30 14:05:49.841189 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:49.841160 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" event={"ID":"4fa94d86-57ce-4d23-95d5-4efa965f6864","Type":"ContainerStarted","Data":"9529b2dbd152577c43fa8abd5f97ab25dfd5c838563979d7312fb109ceedcf29"} Oct 30 14:05:50.794936 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:50.794899 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:50.847618 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:50.847580 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" event={"ID":"4fa94d86-57ce-4d23-95d5-4efa965f6864","Type":"ContainerStarted","Data":"b24f03de41febe4ea1dd73fe96cebf785a05a1decf34a4f7644408cfa18b506b"} Oct 30 14:05:50.895596 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:50.895546 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" podStartSLOduration=1.895531592 podStartE2EDuration="1.895531592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:05:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:05:50.894736518 +0000 UTC m=+1020.638870560" watchObservedRunningTime="2026-10-30 14:05:50.895531592 +0000 UTC m=+1020.639665634" Oct 30 14:05:51.455760 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:51.455722 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" Oct 30 14:05:52.674176 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:52.674132 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:54.230421 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:54.230383 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb"] Oct 30 14:05:54.235476 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:54.235455 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb" Oct 30 14:05:54.274428 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:54.274402 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb"] Oct 30 14:05:54.368294 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:54.368268 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f29ce71-fed6-4c16-b200-b61362a9a60f-istiod-ca-cert\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb\" (UID: \"9f29ce71-fed6-4c16-b200-b61362a9a60f\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb" Oct 30 14:05:54.368394 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:54.368328 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f29ce71-fed6-4c16-b200-b61362a9a60f-istio-token\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb\" (UID: \"9f29ce71-fed6-4c16-b200-b61362a9a60f\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb" Oct 30 14:05:54.368454 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:54.368412 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f29ce71-fed6-4c16-b200-b61362a9a60f-workload-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb\" (UID: \"9f29ce71-fed6-4c16-b200-b61362a9a60f\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb" Oct 30 14:05:54.368500 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:54.368469 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f29ce71-fed6-4c16-b200-b61362a9a60f-istio-envoy\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb\" (UID: \"9f29ce71-fed6-4c16-b200-b61362a9a60f\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb" Oct 30 14:05:54.368548 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:54.368528 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f29ce71-fed6-4c16-b200-b61362a9a60f-istio-data\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb\" (UID: \"9f29ce71-fed6-4c16-b200-b61362a9a60f\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb" Oct 30 14:05:54.368588 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:54.368573 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f29ce71-fed6-4c16-b200-b61362a9a60f-workload-certs\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb\" (UID: \"9f29ce71-fed6-4c16-b200-b61362a9a60f\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb" Oct 30 14:05:54.368731 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:54.368710 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5hrc\" (UniqueName: \"kubernetes.io/projected/9f29ce71-fed6-4c16-b200-b61362a9a60f-kube-api-access-z5hrc\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb\" (UID: \"9f29ce71-fed6-4c16-b200-b61362a9a60f\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb" Oct 30 14:05:54.368770 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:54.368743 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f29ce71-fed6-4c16-b200-b61362a9a60f-istio-podinfo\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb\" (UID: \"9f29ce71-fed6-4c16-b200-b61362a9a60f\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb" Oct 30 14:05:54.368770 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:54.368762 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f29ce71-fed6-4c16-b200-b61362a9a60f-credential-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb\" (UID: \"9f29ce71-fed6-4c16-b200-b61362a9a60f\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb" Oct 30 14:05:54.469942 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:54.469909 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f29ce71-fed6-4c16-b200-b61362a9a60f-istio-token\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb\" (UID: \"9f29ce71-fed6-4c16-b200-b61362a9a60f\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb" Oct 30 14:05:54.470063 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:54.469952 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f29ce71-fed6-4c16-b200-b61362a9a60f-workload-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb\" (UID: \"9f29ce71-fed6-4c16-b200-b61362a9a60f\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb" Oct 30 14:05:54.470063 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:54.469986 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f29ce71-fed6-4c16-b200-b61362a9a60f-istio-envoy\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb\" (UID: \"9f29ce71-fed6-4c16-b200-b61362a9a60f\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb" Oct 30 14:05:54.470063 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:54.470013 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f29ce71-fed6-4c16-b200-b61362a9a60f-istio-data\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb\" (UID: \"9f29ce71-fed6-4c16-b200-b61362a9a60f\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb" Oct 30 14:05:54.470063 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:54.470051 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f29ce71-fed6-4c16-b200-b61362a9a60f-workload-certs\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb\" (UID: \"9f29ce71-fed6-4c16-b200-b61362a9a60f\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb" Oct 30 14:05:54.470286 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:54.470136 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5hrc\" (UniqueName: \"kubernetes.io/projected/9f29ce71-fed6-4c16-b200-b61362a9a60f-kube-api-access-z5hrc\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb\" (UID: \"9f29ce71-fed6-4c16-b200-b61362a9a60f\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb" Oct 30 14:05:54.470286 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:54.470168 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f29ce71-fed6-4c16-b200-b61362a9a60f-istio-podinfo\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb\" (UID: \"9f29ce71-fed6-4c16-b200-b61362a9a60f\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb" Oct 30 14:05:54.470286 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:54.470204 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f29ce71-fed6-4c16-b200-b61362a9a60f-credential-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb\" (UID: \"9f29ce71-fed6-4c16-b200-b61362a9a60f\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb" Oct 30 14:05:54.470433 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:54.470283 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f29ce71-fed6-4c16-b200-b61362a9a60f-istiod-ca-cert\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb\" (UID: \"9f29ce71-fed6-4c16-b200-b61362a9a60f\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb" Oct 30 14:05:54.470433 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:54.470369 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f29ce71-fed6-4c16-b200-b61362a9a60f-workload-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb\" (UID: \"9f29ce71-fed6-4c16-b200-b61362a9a60f\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb" Oct 30 14:05:54.470554 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:54.470525 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f29ce71-fed6-4c16-b200-b61362a9a60f-workload-certs\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb\" (UID: \"9f29ce71-fed6-4c16-b200-b61362a9a60f\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb" Oct 30 14:05:54.470669 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:54.470633 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f29ce71-fed6-4c16-b200-b61362a9a60f-istio-data\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb\" (UID: \"9f29ce71-fed6-4c16-b200-b61362a9a60f\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb" Oct 30 14:05:54.470885 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:54.470843 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f29ce71-fed6-4c16-b200-b61362a9a60f-credential-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb\" (UID: \"9f29ce71-fed6-4c16-b200-b61362a9a60f\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb" Oct 30 14:05:54.471143 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:54.471115 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f29ce71-fed6-4c16-b200-b61362a9a60f-istiod-ca-cert\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb\" (UID: \"9f29ce71-fed6-4c16-b200-b61362a9a60f\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb" Oct 30 14:05:54.472618 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:54.472593 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f29ce71-fed6-4c16-b200-b61362a9a60f-istio-envoy\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb\" (UID: \"9f29ce71-fed6-4c16-b200-b61362a9a60f\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb" Oct 30 14:05:54.472964 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:54.472944 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f29ce71-fed6-4c16-b200-b61362a9a60f-istio-podinfo\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb\" (UID: \"9f29ce71-fed6-4c16-b200-b61362a9a60f\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb" Oct 30 14:05:54.495416 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:54.495363 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f29ce71-fed6-4c16-b200-b61362a9a60f-istio-token\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb\" (UID: \"9f29ce71-fed6-4c16-b200-b61362a9a60f\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb" Oct 30 14:05:54.495511 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:54.495465 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5hrc\" (UniqueName: \"kubernetes.io/projected/9f29ce71-fed6-4c16-b200-b61362a9a60f-kube-api-access-z5hrc\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb\" (UID: \"9f29ce71-fed6-4c16-b200-b61362a9a60f\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb" Oct 30 14:05:54.513498 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:54.513472 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:54.549201 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:54.549172 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb" Oct 30 14:05:54.745262 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:54.745220 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:54.870109 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:54.870080 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb"] Oct 30 14:05:54.874505 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:05:54.873776 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f29ce71_fed6_4c16_b200_b61362a9a60f.slice/crio-f4a141503f1e20fa1f00aceaf68c6caf7fa763c4ddb83ed41d47716331036d58 WatchSource:0}: Error finding container f4a141503f1e20fa1f00aceaf68c6caf7fa763c4ddb83ed41d47716331036d58: Status 404 returned error can't find the container with id f4a141503f1e20fa1f00aceaf68c6caf7fa763c4ddb83ed41d47716331036d58 Oct 30 14:05:54.877014 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:54.876982 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:05:54.877114 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:54.877057 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:05:54.877114 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:54.877086 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:05:55.887663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:55.884704 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb" event={"ID":"9f29ce71-fed6-4c16-b200-b61362a9a60f","Type":"ContainerStarted","Data":"20125e17829caf31e07b7400dceec76b4f8ee75f9a5b2bba7237cf1049be710f"} Oct 30 14:05:55.887663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:55.884812 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb" event={"ID":"9f29ce71-fed6-4c16-b200-b61362a9a60f","Type":"ContainerStarted","Data":"f4a141503f1e20fa1f00aceaf68c6caf7fa763c4ddb83ed41d47716331036d58"} Oct 30 14:05:55.964500 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:55.964440 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb" podStartSLOduration=1.964421675 podStartE2EDuration="1.964421675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:05:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:05:55.948437463 +0000 UTC m=+1025.692571505" watchObservedRunningTime="2026-10-30 14:05:55.964421675 +0000 UTC m=+1025.708555699" Oct 30 14:05:56.316951 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:56.316920 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:56.550297 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:56.550263 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb" Oct 30 14:05:56.551803 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:56.551777 4982 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 14:05:56.551926 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:56.551824 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb" podUID="9f29ce71-fed6-4c16-b200-b61362a9a60f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 14:05:57.511349 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:57.510693 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9"] Oct 30 14:05:57.517353 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:57.516054 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" Oct 30 14:05:57.530780 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:57.530750 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9"] Oct 30 14:05:57.540265 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:57.540226 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-2df0ab\"/\"kube-root-ca.crt\"" Oct 30 14:05:57.540265 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:57.540247 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-2df0ab\"/\"openshift-service-ca.crt\"" Oct 30 14:05:57.545775 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:57.545748 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-2df0ab\"/\"default-dockercfg-wfvcl\"" Oct 30 14:05:57.545775 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:57.545755 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-2df0ab\"/\"premium-w0-2df0ab-kserve-self-signed-certs\"" Oct 30 14:05:57.550084 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:57.550064 4982 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 14:05:57.550152 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:57.550113 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb" podUID="9f29ce71-fed6-4c16-b200-b61362a9a60f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 14:05:57.605665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:57.604742 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:57.627994 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:57.627959 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/946696b9-91bd-437a-9f23-c16b73a5a9a6-tmp-dir\") pod \"premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9\" (UID: \"946696b9-91bd-437a-9f23-c16b73a5a9a6\") " pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" Oct 30 14:05:57.628149 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:57.628012 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8m8h\" (UniqueName: \"kubernetes.io/projected/946696b9-91bd-437a-9f23-c16b73a5a9a6-kube-api-access-l8m8h\") pod \"premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9\" (UID: \"946696b9-91bd-437a-9f23-c16b73a5a9a6\") " pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" Oct 30 14:05:57.628231 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:57.628145 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/946696b9-91bd-437a-9f23-c16b73a5a9a6-home\") pod \"premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9\" (UID: \"946696b9-91bd-437a-9f23-c16b73a5a9a6\") " pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" Oct 30 14:05:57.628231 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:57.628213 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/946696b9-91bd-437a-9f23-c16b73a5a9a6-model-cache\") pod \"premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9\" (UID: \"946696b9-91bd-437a-9f23-c16b73a5a9a6\") " pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" Oct 30 14:05:57.628362 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:57.628343 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/946696b9-91bd-437a-9f23-c16b73a5a9a6-tls-certs\") pod \"premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9\" (UID: \"946696b9-91bd-437a-9f23-c16b73a5a9a6\") " pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" Oct 30 14:05:57.628424 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:57.628384 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/946696b9-91bd-437a-9f23-c16b73a5a9a6-dshm\") pod \"premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9\" (UID: \"946696b9-91bd-437a-9f23-c16b73a5a9a6\") " pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" Oct 30 14:05:57.734660 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:57.734201 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/946696b9-91bd-437a-9f23-c16b73a5a9a6-model-cache\") pod \"premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9\" (UID: \"946696b9-91bd-437a-9f23-c16b73a5a9a6\") " pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" Oct 30 14:05:57.734660 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:57.734323 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/946696b9-91bd-437a-9f23-c16b73a5a9a6-tls-certs\") pod \"premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9\" (UID: \"946696b9-91bd-437a-9f23-c16b73a5a9a6\") " pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" Oct 30 14:05:57.734660 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:57.734356 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/946696b9-91bd-437a-9f23-c16b73a5a9a6-dshm\") pod \"premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9\" (UID: \"946696b9-91bd-437a-9f23-c16b73a5a9a6\") " pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" Oct 30 14:05:57.734660 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:57.734415 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/946696b9-91bd-437a-9f23-c16b73a5a9a6-tmp-dir\") pod \"premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9\" (UID: \"946696b9-91bd-437a-9f23-c16b73a5a9a6\") " pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" Oct 30 14:05:57.734660 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:57.734457 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8m8h\" (UniqueName: \"kubernetes.io/projected/946696b9-91bd-437a-9f23-c16b73a5a9a6-kube-api-access-l8m8h\") pod \"premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9\" (UID: \"946696b9-91bd-437a-9f23-c16b73a5a9a6\") " pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" Oct 30 14:05:57.734660 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:57.734512 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/946696b9-91bd-437a-9f23-c16b73a5a9a6-home\") pod \"premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9\" (UID: \"946696b9-91bd-437a-9f23-c16b73a5a9a6\") " pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" Oct 30 14:05:57.735123 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:57.734972 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/946696b9-91bd-437a-9f23-c16b73a5a9a6-home\") pod \"premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9\" (UID: \"946696b9-91bd-437a-9f23-c16b73a5a9a6\") " pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" Oct 30 14:05:57.738664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:57.735227 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/946696b9-91bd-437a-9f23-c16b73a5a9a6-model-cache\") pod \"premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9\" (UID: \"946696b9-91bd-437a-9f23-c16b73a5a9a6\") " pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" Oct 30 14:05:57.738664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:57.737046 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/946696b9-91bd-437a-9f23-c16b73a5a9a6-tmp-dir\") pod \"premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9\" (UID: \"946696b9-91bd-437a-9f23-c16b73a5a9a6\") " pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" Oct 30 14:05:57.765911 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:57.761432 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/946696b9-91bd-437a-9f23-c16b73a5a9a6-dshm\") pod \"premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9\" (UID: \"946696b9-91bd-437a-9f23-c16b73a5a9a6\") " pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" Oct 30 14:05:57.765911 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:57.763268 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/946696b9-91bd-437a-9f23-c16b73a5a9a6-tls-certs\") pod \"premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9\" (UID: \"946696b9-91bd-437a-9f23-c16b73a5a9a6\") " pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" Oct 30 14:05:57.769705 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:57.768254 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8m8h\" (UniqueName: \"kubernetes.io/projected/946696b9-91bd-437a-9f23-c16b73a5a9a6-kube-api-access-l8m8h\") pod \"premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9\" (UID: \"946696b9-91bd-437a-9f23-c16b73a5a9a6\") " pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" Oct 30 14:05:57.833423 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:57.833388 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" Oct 30 14:05:58.038450 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:05:58.038359 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:05:58.115083 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:58.115043 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:05:58.162905 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:58.162878 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9"] Oct 30 14:05:58.163851 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:05:58.163823 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod946696b9_91bd_437a_9f23_c16b73a5a9a6.slice/crio-502f1ed5b19d3c740f6785d767cdbfdba64fbe3caeeb768085ed71cc0c6e43cf WatchSource:0}: Error finding container 502f1ed5b19d3c740f6785d767cdbfdba64fbe3caeeb768085ed71cc0c6e43cf: Status 404 returned error can't find the container with id 502f1ed5b19d3c740f6785d767cdbfdba64fbe3caeeb768085ed71cc0c6e43cf Oct 30 14:05:58.549826 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:58.549792 4982 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 14:05:58.550206 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:58.549842 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb" podUID="9f29ce71-fed6-4c16-b200-b61362a9a60f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 14:05:58.921142 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:58.921099 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" event={"ID":"946696b9-91bd-437a-9f23-c16b73a5a9a6","Type":"ContainerStarted","Data":"9f40a74941b4c90cad3417db554d163394dbd2ddd03fc61e55b86e2aec49c0c2"} Oct 30 14:05:58.921142 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:58.921140 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" event={"ID":"946696b9-91bd-437a-9f23-c16b73a5a9a6","Type":"ContainerStarted","Data":"502f1ed5b19d3c740f6785d767cdbfdba64fbe3caeeb768085ed71cc0c6e43cf"} Oct 30 14:05:58.966791 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:58.966745 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" podStartSLOduration=1.966732472 podStartE2EDuration="1.966732472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:05:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:05:58.964508001 +0000 UTC m=+1028.708642043" watchObservedRunningTime="2026-10-30 14:05:58.966732472 +0000 UTC m=+1028.710866514" Oct 30 14:05:59.455924 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:59.455885 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" Oct 30 14:05:59.468531 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:59.468499 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" Oct 30 14:05:59.549941 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:59.549907 4982 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 14:05:59.550293 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:59.549956 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb" podUID="9f29ce71-fed6-4c16-b200-b61362a9a60f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 14:05:59.834254 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:59.834220 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" Oct 30 14:05:59.942238 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:05:59.942208 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" Oct 30 14:06:00.049327 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:00.049273 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:00.550212 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:00.550177 4982 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 14:06:00.550583 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:00.550233 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb" podUID="9f29ce71-fed6-4c16-b200-b61362a9a60f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 14:06:01.554000 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:01.553967 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb" Oct 30 14:06:01.554464 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:01.554423 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb" Oct 30 14:06:01.555279 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:01.555253 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb" Oct 30 14:06:01.853566 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:01.853509 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:02.801763 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:02.801610 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:04.403010 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:04.402970 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:06.226904 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:06.226863 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:07.546202 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:07.546164 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:07.833536 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:07.833504 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" Oct 30 14:06:07.846929 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:07.846908 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" Oct 30 14:06:07.996239 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:07.996203 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" Oct 30 14:06:08.061972 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:08.061936 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:08.347164 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:08.347129 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:10.039799 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:10.039766 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:11.826028 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:11.825976 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:13.697444 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:13.697414 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:15.860859 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:15.860781 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:16.556163 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:16.556102 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:17.747885 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:17.747836 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:19.568034 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:19.567993 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:20.404484 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:20.404445 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn"] Oct 30 14:06:20.409612 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:20.409587 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" Oct 30 14:06:20.428961 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:20.428927 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn"] Oct 30 14:06:20.435439 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:20.435408 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-ca229c\"/\"premium-w1-ca229c-kserve-self-signed-certs\"" Oct 30 14:06:20.495241 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:20.495206 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:20.515278 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:20.515225 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdbl4\" (UniqueName: \"kubernetes.io/projected/cbd754bd-771d-4007-8ac0-7e413dc24915-kube-api-access-qdbl4\") pod \"premium-w1-ca229c-kserve-545bb5d7b9-q2gvn\" (UID: \"cbd754bd-771d-4007-8ac0-7e413dc24915\") " pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" Oct 30 14:06:20.515478 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:20.515358 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbd754bd-771d-4007-8ac0-7e413dc24915-dshm\") pod \"premium-w1-ca229c-kserve-545bb5d7b9-q2gvn\" (UID: \"cbd754bd-771d-4007-8ac0-7e413dc24915\") " pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" Oct 30 14:06:20.515478 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:20.515422 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbd754bd-771d-4007-8ac0-7e413dc24915-tls-certs\") pod \"premium-w1-ca229c-kserve-545bb5d7b9-q2gvn\" (UID: \"cbd754bd-771d-4007-8ac0-7e413dc24915\") " pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" Oct 30 14:06:20.515614 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:20.515527 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbd754bd-771d-4007-8ac0-7e413dc24915-home\") pod \"premium-w1-ca229c-kserve-545bb5d7b9-q2gvn\" (UID: \"cbd754bd-771d-4007-8ac0-7e413dc24915\") " pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" Oct 30 14:06:20.515701 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:20.515687 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbd754bd-771d-4007-8ac0-7e413dc24915-model-cache\") pod \"premium-w1-ca229c-kserve-545bb5d7b9-q2gvn\" (UID: \"cbd754bd-771d-4007-8ac0-7e413dc24915\") " pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" Oct 30 14:06:20.515778 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:20.515755 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbd754bd-771d-4007-8ac0-7e413dc24915-tmp-dir\") pod \"premium-w1-ca229c-kserve-545bb5d7b9-q2gvn\" (UID: \"cbd754bd-771d-4007-8ac0-7e413dc24915\") " pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" Oct 30 14:06:20.616795 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:20.616745 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbd754bd-771d-4007-8ac0-7e413dc24915-model-cache\") pod \"premium-w1-ca229c-kserve-545bb5d7b9-q2gvn\" (UID: \"cbd754bd-771d-4007-8ac0-7e413dc24915\") " pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" Oct 30 14:06:20.617296 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:20.616815 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbd754bd-771d-4007-8ac0-7e413dc24915-tmp-dir\") pod \"premium-w1-ca229c-kserve-545bb5d7b9-q2gvn\" (UID: \"cbd754bd-771d-4007-8ac0-7e413dc24915\") " pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" Oct 30 14:06:20.617296 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:20.616883 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdbl4\" (UniqueName: \"kubernetes.io/projected/cbd754bd-771d-4007-8ac0-7e413dc24915-kube-api-access-qdbl4\") pod \"premium-w1-ca229c-kserve-545bb5d7b9-q2gvn\" (UID: \"cbd754bd-771d-4007-8ac0-7e413dc24915\") " pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" Oct 30 14:06:20.617423 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:20.617335 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbd754bd-771d-4007-8ac0-7e413dc24915-tmp-dir\") pod \"premium-w1-ca229c-kserve-545bb5d7b9-q2gvn\" (UID: \"cbd754bd-771d-4007-8ac0-7e413dc24915\") " pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" Oct 30 14:06:20.617423 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:20.617383 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbd754bd-771d-4007-8ac0-7e413dc24915-model-cache\") pod \"premium-w1-ca229c-kserve-545bb5d7b9-q2gvn\" (UID: \"cbd754bd-771d-4007-8ac0-7e413dc24915\") " pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" Oct 30 14:06:20.617528 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:20.617464 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbd754bd-771d-4007-8ac0-7e413dc24915-dshm\") pod \"premium-w1-ca229c-kserve-545bb5d7b9-q2gvn\" (UID: \"cbd754bd-771d-4007-8ac0-7e413dc24915\") " pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" Oct 30 14:06:20.617528 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:20.617512 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbd754bd-771d-4007-8ac0-7e413dc24915-tls-certs\") pod \"premium-w1-ca229c-kserve-545bb5d7b9-q2gvn\" (UID: \"cbd754bd-771d-4007-8ac0-7e413dc24915\") " pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" Oct 30 14:06:20.617631 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:20.617551 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbd754bd-771d-4007-8ac0-7e413dc24915-home\") pod \"premium-w1-ca229c-kserve-545bb5d7b9-q2gvn\" (UID: \"cbd754bd-771d-4007-8ac0-7e413dc24915\") " pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" Oct 30 14:06:20.618006 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:20.617985 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbd754bd-771d-4007-8ac0-7e413dc24915-home\") pod \"premium-w1-ca229c-kserve-545bb5d7b9-q2gvn\" (UID: \"cbd754bd-771d-4007-8ac0-7e413dc24915\") " pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" Oct 30 14:06:20.620863 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:20.620832 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbd754bd-771d-4007-8ac0-7e413dc24915-dshm\") pod \"premium-w1-ca229c-kserve-545bb5d7b9-q2gvn\" (UID: \"cbd754bd-771d-4007-8ac0-7e413dc24915\") " pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" Oct 30 14:06:20.623046 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:20.623019 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbd754bd-771d-4007-8ac0-7e413dc24915-tls-certs\") pod \"premium-w1-ca229c-kserve-545bb5d7b9-q2gvn\" (UID: \"cbd754bd-771d-4007-8ac0-7e413dc24915\") " pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" Oct 30 14:06:20.645770 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:20.645734 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdbl4\" (UniqueName: \"kubernetes.io/projected/cbd754bd-771d-4007-8ac0-7e413dc24915-kube-api-access-qdbl4\") pod \"premium-w1-ca229c-kserve-545bb5d7b9-q2gvn\" (UID: \"cbd754bd-771d-4007-8ac0-7e413dc24915\") " pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" Oct 30 14:06:20.728717 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:20.728598 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" Oct 30 14:06:21.036269 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:21.036239 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn"] Oct 30 14:06:21.037421 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:06:21.037395 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcbd754bd_771d_4007_8ac0_7e413dc24915.slice/crio-b90ef2db5eefd1a9a37d1eee9825c3ca6559195829ad5a703bb1176e08f76ab2 WatchSource:0}: Error finding container b90ef2db5eefd1a9a37d1eee9825c3ca6559195829ad5a703bb1176e08f76ab2: Status 404 returned error can't find the container with id b90ef2db5eefd1a9a37d1eee9825c3ca6559195829ad5a703bb1176e08f76ab2 Oct 30 14:06:21.056894 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:21.056858 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" event={"ID":"cbd754bd-771d-4007-8ac0-7e413dc24915","Type":"ContainerStarted","Data":"b90ef2db5eefd1a9a37d1eee9825c3ca6559195829ad5a703bb1176e08f76ab2"} Oct 30 14:06:21.420923 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:21.420887 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:22.063808 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:22.063769 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" event={"ID":"cbd754bd-771d-4007-8ac0-7e413dc24915","Type":"ContainerStarted","Data":"b08205185b49f8f91e2fac0f1ee1b8adb70afe7d6201a1a78cf3b6718ac81bb6"} Oct 30 14:06:22.109097 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:22.109035 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" podStartSLOduration=2.109018277 podStartE2EDuration="2.109018277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:06:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:06:22.108123314 +0000 UTC m=+1051.852257357" watchObservedRunningTime="2026-10-30 14:06:22.109018277 +0000 UTC m=+1051.853152318" Oct 30 14:06:22.729527 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:22.729491 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" Oct 30 14:06:23.226229 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:23.226188 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:24.030125 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:24.030083 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8"] Oct 30 14:06:24.034066 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:24.034040 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8" Oct 30 14:06:24.065290 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:24.065264 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-2df0ab-openshift-default-dockercfg-z8znz\"" Oct 30 14:06:24.068765 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:24.068735 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8"] Oct 30 14:06:24.162634 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:24.162603 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00eefe61-57f5-4d1c-b9a5-8dd54db2922e-istio-token\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8\" (UID: \"00eefe61-57f5-4d1c-b9a5-8dd54db2922e\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8" Oct 30 14:06:24.162802 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:24.162748 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00eefe61-57f5-4d1c-b9a5-8dd54db2922e-credential-socket\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8\" (UID: \"00eefe61-57f5-4d1c-b9a5-8dd54db2922e\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8" Oct 30 14:06:24.162802 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:24.162785 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00eefe61-57f5-4d1c-b9a5-8dd54db2922e-workload-certs\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8\" (UID: \"00eefe61-57f5-4d1c-b9a5-8dd54db2922e\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8" Oct 30 14:06:24.162930 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:24.162904 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00eefe61-57f5-4d1c-b9a5-8dd54db2922e-istio-envoy\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8\" (UID: \"00eefe61-57f5-4d1c-b9a5-8dd54db2922e\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8" Oct 30 14:06:24.162984 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:24.162964 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00eefe61-57f5-4d1c-b9a5-8dd54db2922e-workload-socket\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8\" (UID: \"00eefe61-57f5-4d1c-b9a5-8dd54db2922e\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8" Oct 30 14:06:24.163019 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:24.162994 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vvkl\" (UniqueName: \"kubernetes.io/projected/00eefe61-57f5-4d1c-b9a5-8dd54db2922e-kube-api-access-8vvkl\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8\" (UID: \"00eefe61-57f5-4d1c-b9a5-8dd54db2922e\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8" Oct 30 14:06:24.163019 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:24.163013 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00eefe61-57f5-4d1c-b9a5-8dd54db2922e-istio-data\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8\" (UID: \"00eefe61-57f5-4d1c-b9a5-8dd54db2922e\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8" Oct 30 14:06:24.163094 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:24.163083 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00eefe61-57f5-4d1c-b9a5-8dd54db2922e-istiod-ca-cert\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8\" (UID: \"00eefe61-57f5-4d1c-b9a5-8dd54db2922e\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8" Oct 30 14:06:24.163126 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:24.163110 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00eefe61-57f5-4d1c-b9a5-8dd54db2922e-istio-podinfo\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8\" (UID: \"00eefe61-57f5-4d1c-b9a5-8dd54db2922e\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8" Oct 30 14:06:24.264703 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:24.264672 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00eefe61-57f5-4d1c-b9a5-8dd54db2922e-istio-token\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8\" (UID: \"00eefe61-57f5-4d1c-b9a5-8dd54db2922e\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8" Oct 30 14:06:24.265086 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:24.264729 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00eefe61-57f5-4d1c-b9a5-8dd54db2922e-credential-socket\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8\" (UID: \"00eefe61-57f5-4d1c-b9a5-8dd54db2922e\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8" Oct 30 14:06:24.265086 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:24.264770 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00eefe61-57f5-4d1c-b9a5-8dd54db2922e-workload-certs\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8\" (UID: \"00eefe61-57f5-4d1c-b9a5-8dd54db2922e\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8" Oct 30 14:06:24.265086 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:24.264829 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00eefe61-57f5-4d1c-b9a5-8dd54db2922e-istio-envoy\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8\" (UID: \"00eefe61-57f5-4d1c-b9a5-8dd54db2922e\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8" Oct 30 14:06:24.265086 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:24.264870 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00eefe61-57f5-4d1c-b9a5-8dd54db2922e-workload-socket\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8\" (UID: \"00eefe61-57f5-4d1c-b9a5-8dd54db2922e\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8" Oct 30 14:06:24.265086 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:24.264991 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vvkl\" (UniqueName: \"kubernetes.io/projected/00eefe61-57f5-4d1c-b9a5-8dd54db2922e-kube-api-access-8vvkl\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8\" (UID: \"00eefe61-57f5-4d1c-b9a5-8dd54db2922e\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8" Oct 30 14:06:24.265086 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:24.265038 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00eefe61-57f5-4d1c-b9a5-8dd54db2922e-istio-data\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8\" (UID: \"00eefe61-57f5-4d1c-b9a5-8dd54db2922e\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8" Oct 30 14:06:24.265427 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:24.265091 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00eefe61-57f5-4d1c-b9a5-8dd54db2922e-istiod-ca-cert\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8\" (UID: \"00eefe61-57f5-4d1c-b9a5-8dd54db2922e\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8" Oct 30 14:06:24.265427 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:24.265134 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00eefe61-57f5-4d1c-b9a5-8dd54db2922e-istio-podinfo\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8\" (UID: \"00eefe61-57f5-4d1c-b9a5-8dd54db2922e\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8" Oct 30 14:06:24.265427 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:24.265185 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00eefe61-57f5-4d1c-b9a5-8dd54db2922e-credential-socket\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8\" (UID: \"00eefe61-57f5-4d1c-b9a5-8dd54db2922e\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8" Oct 30 14:06:24.265427 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:24.265229 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00eefe61-57f5-4d1c-b9a5-8dd54db2922e-workload-certs\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8\" (UID: \"00eefe61-57f5-4d1c-b9a5-8dd54db2922e\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8" Oct 30 14:06:24.265656 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:24.265421 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00eefe61-57f5-4d1c-b9a5-8dd54db2922e-workload-socket\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8\" (UID: \"00eefe61-57f5-4d1c-b9a5-8dd54db2922e\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8" Oct 30 14:06:24.265758 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:24.265739 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00eefe61-57f5-4d1c-b9a5-8dd54db2922e-istio-data\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8\" (UID: \"00eefe61-57f5-4d1c-b9a5-8dd54db2922e\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8" Oct 30 14:06:24.265809 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:24.265790 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00eefe61-57f5-4d1c-b9a5-8dd54db2922e-istiod-ca-cert\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8\" (UID: \"00eefe61-57f5-4d1c-b9a5-8dd54db2922e\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8" Oct 30 14:06:24.267759 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:24.267729 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00eefe61-57f5-4d1c-b9a5-8dd54db2922e-istio-envoy\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8\" (UID: \"00eefe61-57f5-4d1c-b9a5-8dd54db2922e\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8" Oct 30 14:06:24.267924 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:24.267907 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00eefe61-57f5-4d1c-b9a5-8dd54db2922e-istio-podinfo\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8\" (UID: \"00eefe61-57f5-4d1c-b9a5-8dd54db2922e\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8" Oct 30 14:06:24.291633 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:24.291569 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vvkl\" (UniqueName: \"kubernetes.io/projected/00eefe61-57f5-4d1c-b9a5-8dd54db2922e-kube-api-access-8vvkl\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8\" (UID: \"00eefe61-57f5-4d1c-b9a5-8dd54db2922e\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8" Oct 30 14:06:24.291777 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:24.291630 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00eefe61-57f5-4d1c-b9a5-8dd54db2922e-istio-token\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8\" (UID: \"00eefe61-57f5-4d1c-b9a5-8dd54db2922e\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8" Oct 30 14:06:24.348900 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:24.348871 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8" Oct 30 14:06:24.457824 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:24.457781 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml"] Oct 30 14:06:24.462531 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:24.462508 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml" Oct 30 14:06:24.481455 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:24.481432 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-da78c4-openshift-default-dockercfg-jkr4w\"" Oct 30 14:06:24.489065 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:24.488992 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml"] Oct 30 14:06:24.568393 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:24.568356 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6687b0d-4a6b-4fe8-9c62-47057042db79-istio-token\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml\" (UID: \"a6687b0d-4a6b-4fe8-9c62-47057042db79\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml" Oct 30 14:06:24.568591 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:24.568412 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6687b0d-4a6b-4fe8-9c62-47057042db79-istio-podinfo\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml\" (UID: \"a6687b0d-4a6b-4fe8-9c62-47057042db79\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml" Oct 30 14:06:24.568591 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:24.568494 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6687b0d-4a6b-4fe8-9c62-47057042db79-istio-envoy\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml\" (UID: \"a6687b0d-4a6b-4fe8-9c62-47057042db79\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml" Oct 30 14:06:24.568735 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:24.568591 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjdkt\" (UniqueName: \"kubernetes.io/projected/a6687b0d-4a6b-4fe8-9c62-47057042db79-kube-api-access-qjdkt\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml\" (UID: \"a6687b0d-4a6b-4fe8-9c62-47057042db79\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml" Oct 30 14:06:24.568735 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:24.568636 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6687b0d-4a6b-4fe8-9c62-47057042db79-workload-socket\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml\" (UID: \"a6687b0d-4a6b-4fe8-9c62-47057042db79\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml" Oct 30 14:06:24.568831 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:24.568742 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6687b0d-4a6b-4fe8-9c62-47057042db79-istio-data\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml\" (UID: \"a6687b0d-4a6b-4fe8-9c62-47057042db79\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml" Oct 30 14:06:24.568831 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:24.568773 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6687b0d-4a6b-4fe8-9c62-47057042db79-credential-socket\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml\" (UID: \"a6687b0d-4a6b-4fe8-9c62-47057042db79\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml" Oct 30 14:06:24.568831 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:24.568796 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6687b0d-4a6b-4fe8-9c62-47057042db79-istiod-ca-cert\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml\" (UID: \"a6687b0d-4a6b-4fe8-9c62-47057042db79\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml" Oct 30 14:06:24.568964 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:24.568870 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6687b0d-4a6b-4fe8-9c62-47057042db79-workload-certs\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml\" (UID: \"a6687b0d-4a6b-4fe8-9c62-47057042db79\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml" Oct 30 14:06:24.670465 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:24.670424 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6687b0d-4a6b-4fe8-9c62-47057042db79-istio-podinfo\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml\" (UID: \"a6687b0d-4a6b-4fe8-9c62-47057042db79\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml" Oct 30 14:06:24.670666 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:24.670486 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6687b0d-4a6b-4fe8-9c62-47057042db79-istio-envoy\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml\" (UID: \"a6687b0d-4a6b-4fe8-9c62-47057042db79\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml" Oct 30 14:06:24.670666 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:24.670546 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjdkt\" (UniqueName: \"kubernetes.io/projected/a6687b0d-4a6b-4fe8-9c62-47057042db79-kube-api-access-qjdkt\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml\" (UID: \"a6687b0d-4a6b-4fe8-9c62-47057042db79\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml" Oct 30 14:06:24.670666 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:24.670593 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6687b0d-4a6b-4fe8-9c62-47057042db79-workload-socket\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml\" (UID: \"a6687b0d-4a6b-4fe8-9c62-47057042db79\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml" Oct 30 14:06:24.670666 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:24.670621 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6687b0d-4a6b-4fe8-9c62-47057042db79-istio-data\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml\" (UID: \"a6687b0d-4a6b-4fe8-9c62-47057042db79\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml" Oct 30 14:06:24.670909 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:24.670690 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6687b0d-4a6b-4fe8-9c62-47057042db79-credential-socket\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml\" (UID: \"a6687b0d-4a6b-4fe8-9c62-47057042db79\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml" Oct 30 14:06:24.670909 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:24.670724 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6687b0d-4a6b-4fe8-9c62-47057042db79-istiod-ca-cert\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml\" (UID: \"a6687b0d-4a6b-4fe8-9c62-47057042db79\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml" Oct 30 14:06:24.670909 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:24.670757 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6687b0d-4a6b-4fe8-9c62-47057042db79-workload-certs\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml\" (UID: \"a6687b0d-4a6b-4fe8-9c62-47057042db79\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml" Oct 30 14:06:24.670909 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:24.670826 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6687b0d-4a6b-4fe8-9c62-47057042db79-istio-token\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml\" (UID: \"a6687b0d-4a6b-4fe8-9c62-47057042db79\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml" Oct 30 14:06:24.671113 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:24.671067 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6687b0d-4a6b-4fe8-9c62-47057042db79-istio-data\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml\" (UID: \"a6687b0d-4a6b-4fe8-9c62-47057042db79\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml" Oct 30 14:06:24.671518 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:24.671287 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6687b0d-4a6b-4fe8-9c62-47057042db79-workload-certs\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml\" (UID: \"a6687b0d-4a6b-4fe8-9c62-47057042db79\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml" Oct 30 14:06:24.671518 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:24.671471 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6687b0d-4a6b-4fe8-9c62-47057042db79-workload-socket\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml\" (UID: \"a6687b0d-4a6b-4fe8-9c62-47057042db79\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml" Oct 30 14:06:24.671784 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:24.671580 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6687b0d-4a6b-4fe8-9c62-47057042db79-credential-socket\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml\" (UID: \"a6687b0d-4a6b-4fe8-9c62-47057042db79\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml" Oct 30 14:06:24.673341 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:24.673310 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6687b0d-4a6b-4fe8-9c62-47057042db79-istio-podinfo\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml\" (UID: \"a6687b0d-4a6b-4fe8-9c62-47057042db79\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml" Oct 30 14:06:24.673434 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:24.673383 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6687b0d-4a6b-4fe8-9c62-47057042db79-istio-envoy\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml\" (UID: \"a6687b0d-4a6b-4fe8-9c62-47057042db79\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml" Oct 30 14:06:24.673971 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:24.673946 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6687b0d-4a6b-4fe8-9c62-47057042db79-istiod-ca-cert\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml\" (UID: \"a6687b0d-4a6b-4fe8-9c62-47057042db79\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml" Oct 30 14:06:24.674484 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:24.674464 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8"] Oct 30 14:06:24.674928 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:06:24.674904 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00eefe61_57f5_4d1c_b9a5_8dd54db2922e.slice/crio-c4fe3274541b6f22597a5219e5bfc984e20b32f3174e644a1b5f2516078ce3c0 WatchSource:0}: Error finding container c4fe3274541b6f22597a5219e5bfc984e20b32f3174e644a1b5f2516078ce3c0: Status 404 returned error can't find the container with id c4fe3274541b6f22597a5219e5bfc984e20b32f3174e644a1b5f2516078ce3c0 Oct 30 14:06:24.677138 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:24.677106 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:06:24.677233 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:24.677180 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:06:24.677233 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:24.677226 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:06:24.697269 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:24.697243 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6687b0d-4a6b-4fe8-9c62-47057042db79-istio-token\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml\" (UID: \"a6687b0d-4a6b-4fe8-9c62-47057042db79\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml" Oct 30 14:06:24.697445 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:24.697421 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjdkt\" (UniqueName: \"kubernetes.io/projected/a6687b0d-4a6b-4fe8-9c62-47057042db79-kube-api-access-qjdkt\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml\" (UID: \"a6687b0d-4a6b-4fe8-9c62-47057042db79\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml" Oct 30 14:06:24.780800 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:24.780567 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml" Oct 30 14:06:25.084927 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:25.084823 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8" event={"ID":"00eefe61-57f5-4d1c-b9a5-8dd54db2922e","Type":"ContainerStarted","Data":"797831e639462afe18671a43b4d40589e0afcb8b3fe4208cb8d075134fa2a194"} Oct 30 14:06:25.084927 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:25.084896 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8" event={"ID":"00eefe61-57f5-4d1c-b9a5-8dd54db2922e","Type":"ContainerStarted","Data":"c4fe3274541b6f22597a5219e5bfc984e20b32f3174e644a1b5f2516078ce3c0"} Oct 30 14:06:25.104720 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:25.104686 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml"] Oct 30 14:06:25.105493 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:06:25.105462 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6687b0d_4a6b_4fe8_9c62_47057042db79.slice/crio-9aeeaf69c10a8aa26ef60c3924725c9c2cf671800963ded9a5a9f9cadd272aa1 WatchSource:0}: Error finding container 9aeeaf69c10a8aa26ef60c3924725c9c2cf671800963ded9a5a9f9cadd272aa1: Status 404 returned error can't find the container with id 9aeeaf69c10a8aa26ef60c3924725c9c2cf671800963ded9a5a9f9cadd272aa1 Oct 30 14:06:25.108062 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:25.108032 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:06:25.108140 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:25.108094 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:06:25.108140 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:25.108125 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:06:25.134608 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:25.134565 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8" podStartSLOduration=1.134552443 podStartE2EDuration="1.134552443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:06:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:06:25.134483887 +0000 UTC m=+1054.878617930" watchObservedRunningTime="2026-10-30 14:06:25.134552443 +0000 UTC m=+1054.878686484" Oct 30 14:06:25.174670 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:25.171122 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:25.349555 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:25.349471 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8" Oct 30 14:06:25.351233 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:25.351204 4982 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 14:06:25.351358 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:25.351256 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8" podUID="00eefe61-57f5-4d1c-b9a5-8dd54db2922e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 14:06:26.137666 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:26.130124 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml" event={"ID":"a6687b0d-4a6b-4fe8-9c62-47057042db79","Type":"ContainerStarted","Data":"f28c75d166598ecf32a358d61a8bcaf00ed385e2d78c966ace93a5c61b896556"} Oct 30 14:06:26.137666 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:26.130184 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml" event={"ID":"a6687b0d-4a6b-4fe8-9c62-47057042db79","Type":"ContainerStarted","Data":"9aeeaf69c10a8aa26ef60c3924725c9c2cf671800963ded9a5a9f9cadd272aa1"} Oct 30 14:06:26.216681 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:26.216602 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml" podStartSLOduration=2.216578475 podStartE2EDuration="2.216578475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:06:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:06:26.198101788 +0000 UTC m=+1055.942235832" watchObservedRunningTime="2026-10-30 14:06:26.216578475 +0000 UTC m=+1055.960712518" Oct 30 14:06:26.350328 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:26.350298 4982 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 14:06:26.350813 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:26.350778 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8" podUID="00eefe61-57f5-4d1c-b9a5-8dd54db2922e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 14:06:26.785671 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:26.785619 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml" Oct 30 14:06:26.786010 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:26.785985 4982 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 14:06:26.786172 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:26.786145 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml" podUID="a6687b0d-4a6b-4fe8-9c62-47057042db79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 14:06:26.843893 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:26.843855 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:27.001780 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:27.001746 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:27.151091 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:27.151047 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:27.349660 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:27.349595 4982 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 14:06:27.349828 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:27.349687 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8" podUID="00eefe61-57f5-4d1c-b9a5-8dd54db2922e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 14:06:27.790666 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:27.783989 4982 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 14:06:27.790666 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:27.784056 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml" podUID="a6687b0d-4a6b-4fe8-9c62-47057042db79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 14:06:28.059585 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:06:28.059548 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:06:28.353732 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:28.352128 4982 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 14:06:28.353732 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:28.352198 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8" podUID="00eefe61-57f5-4d1c-b9a5-8dd54db2922e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 14:06:28.784518 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:28.784006 4982 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 14:06:28.784518 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:28.784077 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml" podUID="a6687b0d-4a6b-4fe8-9c62-47057042db79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 14:06:28.937427 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:28.937347 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:29.350254 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:29.350216 4982 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 14:06:29.350456 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:29.350276 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8" podUID="00eefe61-57f5-4d1c-b9a5-8dd54db2922e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 14:06:29.788722 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:29.783990 4982 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 14:06:29.788722 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:29.784051 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml" podUID="a6687b0d-4a6b-4fe8-9c62-47057042db79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 14:06:30.349425 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:30.349383 4982 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 14:06:30.349866 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:30.349445 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8" podUID="00eefe61-57f5-4d1c-b9a5-8dd54db2922e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 14:06:30.729983 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:30.729880 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" Oct 30 14:06:30.744991 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:30.744940 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:30.748488 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:30.748461 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" Oct 30 14:06:30.796792 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:30.796756 4982 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 14:06:30.797090 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:30.797061 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml" podUID="a6687b0d-4a6b-4fe8-9c62-47057042db79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 14:06:31.174463 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:31.174431 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" Oct 30 14:06:31.349759 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:31.349723 4982 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 14:06:31.350218 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:31.349782 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8" podUID="00eefe61-57f5-4d1c-b9a5-8dd54db2922e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 14:06:32.075572 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:32.075539 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml" Oct 30 14:06:32.076043 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:32.075979 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml" Oct 30 14:06:32.077070 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:32.077046 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml" Oct 30 14:06:32.354482 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:32.354395 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8" Oct 30 14:06:32.354958 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:32.354673 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8" Oct 30 14:06:32.355363 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:32.355347 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8" Oct 30 14:06:32.547896 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:32.547840 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:34.550973 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:34.550902 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:37.215065 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:37.215013 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:39.180500 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:39.180458 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:39.189393 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:39.189365 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4"] Oct 30 14:06:39.193700 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:39.193622 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4" Oct 30 14:06:39.229456 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:39.229424 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-a7bb0c-openshift-default-dockercfg-25sds\"" Oct 30 14:06:39.231939 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:39.231668 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4"] Oct 30 14:06:39.268534 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:39.268497 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de285ba3-daf9-4154-98b0-95b642020499-istio-data\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4\" (UID: \"de285ba3-daf9-4154-98b0-95b642020499\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4" Oct 30 14:06:39.268768 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:39.268554 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de285ba3-daf9-4154-98b0-95b642020499-workload-socket\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4\" (UID: \"de285ba3-daf9-4154-98b0-95b642020499\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4" Oct 30 14:06:39.268768 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:39.268634 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de285ba3-daf9-4154-98b0-95b642020499-istio-envoy\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4\" (UID: \"de285ba3-daf9-4154-98b0-95b642020499\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4" Oct 30 14:06:39.268768 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:39.268713 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de285ba3-daf9-4154-98b0-95b642020499-workload-certs\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4\" (UID: \"de285ba3-daf9-4154-98b0-95b642020499\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4" Oct 30 14:06:39.268906 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:39.268777 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de285ba3-daf9-4154-98b0-95b642020499-credential-socket\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4\" (UID: \"de285ba3-daf9-4154-98b0-95b642020499\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4" Oct 30 14:06:39.268906 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:39.268795 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de285ba3-daf9-4154-98b0-95b642020499-istio-podinfo\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4\" (UID: \"de285ba3-daf9-4154-98b0-95b642020499\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4" Oct 30 14:06:39.268906 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:39.268838 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de285ba3-daf9-4154-98b0-95b642020499-istio-token\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4\" (UID: \"de285ba3-daf9-4154-98b0-95b642020499\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4" Oct 30 14:06:39.269043 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:39.269006 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkxzb\" (UniqueName: \"kubernetes.io/projected/de285ba3-daf9-4154-98b0-95b642020499-kube-api-access-zkxzb\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4\" (UID: \"de285ba3-daf9-4154-98b0-95b642020499\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4" Oct 30 14:06:39.269043 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:39.269033 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de285ba3-daf9-4154-98b0-95b642020499-istiod-ca-cert\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4\" (UID: \"de285ba3-daf9-4154-98b0-95b642020499\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4" Oct 30 14:06:39.369883 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:39.369847 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de285ba3-daf9-4154-98b0-95b642020499-istio-token\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4\" (UID: \"de285ba3-daf9-4154-98b0-95b642020499\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4" Oct 30 14:06:39.370081 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:39.369958 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkxzb\" (UniqueName: \"kubernetes.io/projected/de285ba3-daf9-4154-98b0-95b642020499-kube-api-access-zkxzb\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4\" (UID: \"de285ba3-daf9-4154-98b0-95b642020499\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4" Oct 30 14:06:39.370081 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:39.370000 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de285ba3-daf9-4154-98b0-95b642020499-istiod-ca-cert\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4\" (UID: \"de285ba3-daf9-4154-98b0-95b642020499\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4" Oct 30 14:06:39.370081 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:39.370052 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de285ba3-daf9-4154-98b0-95b642020499-istio-data\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4\" (UID: \"de285ba3-daf9-4154-98b0-95b642020499\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4" Oct 30 14:06:39.370261 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:39.370084 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de285ba3-daf9-4154-98b0-95b642020499-workload-socket\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4\" (UID: \"de285ba3-daf9-4154-98b0-95b642020499\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4" Oct 30 14:06:39.370310 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:39.370267 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de285ba3-daf9-4154-98b0-95b642020499-istio-envoy\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4\" (UID: \"de285ba3-daf9-4154-98b0-95b642020499\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4" Oct 30 14:06:39.370364 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:39.370309 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de285ba3-daf9-4154-98b0-95b642020499-workload-certs\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4\" (UID: \"de285ba3-daf9-4154-98b0-95b642020499\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4" Oct 30 14:06:39.370364 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:39.370346 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de285ba3-daf9-4154-98b0-95b642020499-credential-socket\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4\" (UID: \"de285ba3-daf9-4154-98b0-95b642020499\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4" Oct 30 14:06:39.370474 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:39.370373 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de285ba3-daf9-4154-98b0-95b642020499-istio-podinfo\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4\" (UID: \"de285ba3-daf9-4154-98b0-95b642020499\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4" Oct 30 14:06:39.370525 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:39.370512 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de285ba3-daf9-4154-98b0-95b642020499-istio-data\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4\" (UID: \"de285ba3-daf9-4154-98b0-95b642020499\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4" Oct 30 14:06:39.370617 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:39.370598 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de285ba3-daf9-4154-98b0-95b642020499-workload-socket\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4\" (UID: \"de285ba3-daf9-4154-98b0-95b642020499\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4" Oct 30 14:06:39.370798 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:39.370776 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de285ba3-daf9-4154-98b0-95b642020499-workload-certs\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4\" (UID: \"de285ba3-daf9-4154-98b0-95b642020499\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4" Oct 30 14:06:39.370798 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:39.370789 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de285ba3-daf9-4154-98b0-95b642020499-credential-socket\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4\" (UID: \"de285ba3-daf9-4154-98b0-95b642020499\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4" Oct 30 14:06:39.370963 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:39.370945 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de285ba3-daf9-4154-98b0-95b642020499-istiod-ca-cert\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4\" (UID: \"de285ba3-daf9-4154-98b0-95b642020499\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4" Oct 30 14:06:39.372531 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:39.372507 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de285ba3-daf9-4154-98b0-95b642020499-istio-envoy\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4\" (UID: \"de285ba3-daf9-4154-98b0-95b642020499\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4" Oct 30 14:06:39.372855 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:39.372837 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de285ba3-daf9-4154-98b0-95b642020499-istio-podinfo\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4\" (UID: \"de285ba3-daf9-4154-98b0-95b642020499\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4" Oct 30 14:06:39.396084 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:39.396047 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de285ba3-daf9-4154-98b0-95b642020499-istio-token\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4\" (UID: \"de285ba3-daf9-4154-98b0-95b642020499\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4" Oct 30 14:06:39.396255 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:39.396220 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkxzb\" (UniqueName: \"kubernetes.io/projected/de285ba3-daf9-4154-98b0-95b642020499-kube-api-access-zkxzb\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4\" (UID: \"de285ba3-daf9-4154-98b0-95b642020499\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4" Oct 30 14:06:39.511376 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:39.511261 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4" Oct 30 14:06:39.636411 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:39.636370 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls"] Oct 30 14:06:39.640519 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:39.640495 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls" Oct 30 14:06:39.659712 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:39.659683 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-46b13b-openshift-default-dockercfg-922c6\"" Oct 30 14:06:39.664952 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:39.664927 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls"] Oct 30 14:06:39.776265 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:39.776185 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a20886d-6388-4460-ae47-12a5a28e0eb6-workload-certs\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls\" (UID: \"2a20886d-6388-4460-ae47-12a5a28e0eb6\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls" Oct 30 14:06:39.776265 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:39.776242 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a20886d-6388-4460-ae47-12a5a28e0eb6-istio-envoy\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls\" (UID: \"2a20886d-6388-4460-ae47-12a5a28e0eb6\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls" Oct 30 14:06:39.776441 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:39.776334 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a20886d-6388-4460-ae47-12a5a28e0eb6-istio-podinfo\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls\" (UID: \"2a20886d-6388-4460-ae47-12a5a28e0eb6\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls" Oct 30 14:06:39.776482 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:39.776432 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a20886d-6388-4460-ae47-12a5a28e0eb6-istio-data\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls\" (UID: \"2a20886d-6388-4460-ae47-12a5a28e0eb6\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls" Oct 30 14:06:39.776482 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:39.776476 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a20886d-6388-4460-ae47-12a5a28e0eb6-credential-socket\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls\" (UID: \"2a20886d-6388-4460-ae47-12a5a28e0eb6\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls" Oct 30 14:06:39.776554 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:39.776508 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a20886d-6388-4460-ae47-12a5a28e0eb6-workload-socket\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls\" (UID: \"2a20886d-6388-4460-ae47-12a5a28e0eb6\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls" Oct 30 14:06:39.776554 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:39.776540 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4jth\" (UniqueName: \"kubernetes.io/projected/2a20886d-6388-4460-ae47-12a5a28e0eb6-kube-api-access-j4jth\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls\" (UID: \"2a20886d-6388-4460-ae47-12a5a28e0eb6\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls" Oct 30 14:06:39.776696 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:39.776679 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a20886d-6388-4460-ae47-12a5a28e0eb6-istio-token\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls\" (UID: \"2a20886d-6388-4460-ae47-12a5a28e0eb6\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls" Oct 30 14:06:39.776776 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:39.776756 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a20886d-6388-4460-ae47-12a5a28e0eb6-istiod-ca-cert\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls\" (UID: \"2a20886d-6388-4460-ae47-12a5a28e0eb6\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls" Oct 30 14:06:39.844873 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:06:39.844845 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde285ba3_daf9_4154_98b0_95b642020499.slice/crio-c75d4e0aa30d8ca859e34c3a2aa1676dc6a35a9a2e5e5f5d8d6a27e12c086ad1 WatchSource:0}: Error finding container c75d4e0aa30d8ca859e34c3a2aa1676dc6a35a9a2e5e5f5d8d6a27e12c086ad1: Status 404 returned error can't find the container with id c75d4e0aa30d8ca859e34c3a2aa1676dc6a35a9a2e5e5f5d8d6a27e12c086ad1 Oct 30 14:06:39.845157 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:39.845131 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4"] Oct 30 14:06:39.847498 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:39.847463 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:06:39.847594 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:39.847528 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:06:39.847594 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:39.847557 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:06:39.878319 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:39.878293 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a20886d-6388-4460-ae47-12a5a28e0eb6-workload-certs\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls\" (UID: \"2a20886d-6388-4460-ae47-12a5a28e0eb6\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls" Oct 30 14:06:39.878418 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:39.878341 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a20886d-6388-4460-ae47-12a5a28e0eb6-istio-envoy\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls\" (UID: \"2a20886d-6388-4460-ae47-12a5a28e0eb6\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls" Oct 30 14:06:39.878493 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:39.878470 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a20886d-6388-4460-ae47-12a5a28e0eb6-istio-podinfo\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls\" (UID: \"2a20886d-6388-4460-ae47-12a5a28e0eb6\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls" Oct 30 14:06:39.878595 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:39.878581 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a20886d-6388-4460-ae47-12a5a28e0eb6-istio-data\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls\" (UID: \"2a20886d-6388-4460-ae47-12a5a28e0eb6\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls" Oct 30 14:06:39.878666 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:39.878633 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a20886d-6388-4460-ae47-12a5a28e0eb6-credential-socket\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls\" (UID: \"2a20886d-6388-4460-ae47-12a5a28e0eb6\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls" Oct 30 14:06:39.878727 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:39.878690 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a20886d-6388-4460-ae47-12a5a28e0eb6-workload-certs\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls\" (UID: \"2a20886d-6388-4460-ae47-12a5a28e0eb6\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls" Oct 30 14:06:39.878727 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:39.878692 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a20886d-6388-4460-ae47-12a5a28e0eb6-workload-socket\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls\" (UID: \"2a20886d-6388-4460-ae47-12a5a28e0eb6\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls" Oct 30 14:06:39.878854 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:39.878789 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4jth\" (UniqueName: \"kubernetes.io/projected/2a20886d-6388-4460-ae47-12a5a28e0eb6-kube-api-access-j4jth\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls\" (UID: \"2a20886d-6388-4460-ae47-12a5a28e0eb6\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls" Oct 30 14:06:39.878854 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:39.878843 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a20886d-6388-4460-ae47-12a5a28e0eb6-istio-token\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls\" (UID: \"2a20886d-6388-4460-ae47-12a5a28e0eb6\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls" Oct 30 14:06:39.878965 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:39.878923 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a20886d-6388-4460-ae47-12a5a28e0eb6-istiod-ca-cert\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls\" (UID: \"2a20886d-6388-4460-ae47-12a5a28e0eb6\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls" Oct 30 14:06:39.879037 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:39.879009 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a20886d-6388-4460-ae47-12a5a28e0eb6-istio-data\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls\" (UID: \"2a20886d-6388-4460-ae47-12a5a28e0eb6\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls" Oct 30 14:06:39.879112 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:39.879089 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a20886d-6388-4460-ae47-12a5a28e0eb6-credential-socket\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls\" (UID: \"2a20886d-6388-4460-ae47-12a5a28e0eb6\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls" Oct 30 14:06:39.879362 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:39.879334 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a20886d-6388-4460-ae47-12a5a28e0eb6-workload-socket\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls\" (UID: \"2a20886d-6388-4460-ae47-12a5a28e0eb6\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls" Oct 30 14:06:39.879772 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:39.879750 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a20886d-6388-4460-ae47-12a5a28e0eb6-istiod-ca-cert\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls\" (UID: \"2a20886d-6388-4460-ae47-12a5a28e0eb6\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls" Oct 30 14:06:39.880630 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:39.880611 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a20886d-6388-4460-ae47-12a5a28e0eb6-istio-envoy\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls\" (UID: \"2a20886d-6388-4460-ae47-12a5a28e0eb6\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls" Oct 30 14:06:39.881430 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:39.881397 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a20886d-6388-4460-ae47-12a5a28e0eb6-istio-podinfo\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls\" (UID: \"2a20886d-6388-4460-ae47-12a5a28e0eb6\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls" Oct 30 14:06:39.905611 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:39.905591 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a20886d-6388-4460-ae47-12a5a28e0eb6-istio-token\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls\" (UID: \"2a20886d-6388-4460-ae47-12a5a28e0eb6\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls" Oct 30 14:06:39.906242 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:39.906217 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4jth\" (UniqueName: \"kubernetes.io/projected/2a20886d-6388-4460-ae47-12a5a28e0eb6-kube-api-access-j4jth\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls\" (UID: \"2a20886d-6388-4460-ae47-12a5a28e0eb6\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls" Oct 30 14:06:39.953954 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:39.953909 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls" Oct 30 14:06:40.236441 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:40.236398 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4" event={"ID":"de285ba3-daf9-4154-98b0-95b642020499","Type":"ContainerStarted","Data":"72bbd3796327d2a8db0fa250d6d3f36b2d26da2f5b2e56bc84458ad2137dae23"} Oct 30 14:06:40.236922 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:40.236468 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4" event={"ID":"de285ba3-daf9-4154-98b0-95b642020499","Type":"ContainerStarted","Data":"c75d4e0aa30d8ca859e34c3a2aa1676dc6a35a9a2e5e5f5d8d6a27e12c086ad1"} Oct 30 14:06:40.286378 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:40.286350 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls"] Oct 30 14:06:40.287267 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:06:40.287241 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a20886d_6388_4460_ae47_12a5a28e0eb6.slice/crio-a1bf02e796c442589821ada77100a9de2010f7449918f565b46fa42ec34ba0c1 WatchSource:0}: Error finding container a1bf02e796c442589821ada77100a9de2010f7449918f565b46fa42ec34ba0c1: Status 404 returned error can't find the container with id a1bf02e796c442589821ada77100a9de2010f7449918f565b46fa42ec34ba0c1 Oct 30 14:06:40.290136 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:40.290104 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:06:40.290237 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:40.290164 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:06:40.290237 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:40.290199 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:06:40.299494 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:40.299442 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4" podStartSLOduration=1.29942622 podStartE2EDuration="1.29942622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:06:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:06:40.290115164 +0000 UTC m=+1070.034249205" watchObservedRunningTime="2026-10-30 14:06:40.29942622 +0000 UTC m=+1070.043560263" Oct 30 14:06:40.512059 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:40.511913 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4" Oct 30 14:06:40.514354 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:40.514301 4982 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 30 14:06:40.514500 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:40.514372 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4" podUID="de285ba3-daf9-4154-98b0-95b642020499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 30 14:06:41.243682 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:41.243629 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls" event={"ID":"2a20886d-6388-4460-ae47-12a5a28e0eb6","Type":"ContainerStarted","Data":"ba609330d711f7655ff170ef31402eaac26bea8c9f5f1ecc33e14d6ba9d81a0a"} Oct 30 14:06:41.244109 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:41.243739 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls" event={"ID":"2a20886d-6388-4460-ae47-12a5a28e0eb6","Type":"ContainerStarted","Data":"a1bf02e796c442589821ada77100a9de2010f7449918f565b46fa42ec34ba0c1"} Oct 30 14:06:41.269611 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:41.269579 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:41.311673 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:41.309116 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls" podStartSLOduration=2.309094617 podStartE2EDuration="2.309094617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:06:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:06:41.306169897 +0000 UTC m=+1071.050303940" watchObservedRunningTime="2026-10-30 14:06:41.309094617 +0000 UTC m=+1071.053228660" Oct 30 14:06:41.518725 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:41.512095 4982 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 30 14:06:41.518725 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:41.512160 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4" podUID="de285ba3-daf9-4154-98b0-95b642020499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 30 14:06:41.954544 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:41.954513 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls" Oct 30 14:06:41.955772 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:41.955750 4982 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 14:06:41.955943 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:41.955920 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls" podUID="2a20886d-6388-4460-ae47-12a5a28e0eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 14:06:41.978447 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:41.978408 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:42.522061 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:42.522022 4982 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 30 14:06:42.522615 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:42.522581 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4" podUID="de285ba3-daf9-4154-98b0-95b642020499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 30 14:06:42.955004 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:42.954959 4982 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 14:06:42.955302 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:42.955023 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls" podUID="2a20886d-6388-4460-ae47-12a5a28e0eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 14:06:43.193663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:43.193195 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:43.513665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:43.513008 4982 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 30 14:06:43.513665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:43.513071 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4" podUID="de285ba3-daf9-4154-98b0-95b642020499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 30 14:06:43.955074 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:43.955037 4982 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 14:06:43.955461 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:43.955094 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls" podUID="2a20886d-6388-4460-ae47-12a5a28e0eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 14:06:44.521662 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:44.518273 4982 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 30 14:06:44.521662 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:44.518352 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4" podUID="de285ba3-daf9-4154-98b0-95b642020499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 30 14:06:44.954904 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:44.954855 4982 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 14:06:44.955096 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:44.954919 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls" podUID="2a20886d-6388-4460-ae47-12a5a28e0eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 14:06:45.041992 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:45.041945 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:45.512219 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:45.512163 4982 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 30 14:06:45.512536 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:45.512491 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4" podUID="de285ba3-daf9-4154-98b0-95b642020499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 30 14:06:45.954807 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:45.954767 4982 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 14:06:45.955011 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:45.954839 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls" podUID="2a20886d-6388-4460-ae47-12a5a28e0eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 14:06:46.703263 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:46.701061 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4" Oct 30 14:06:46.703263 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:46.702689 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4" Oct 30 14:06:46.706023 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:46.705990 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4" Oct 30 14:06:46.955167 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:46.955087 4982 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 14:06:46.955320 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:46.955151 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls" podUID="2a20886d-6388-4460-ae47-12a5a28e0eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 14:06:47.134774 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:47.134732 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:47.958387 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:47.958357 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls" Oct 30 14:06:47.958868 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:47.958848 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls" Oct 30 14:06:47.959588 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:47.959567 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls" Oct 30 14:06:49.015103 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:49.015063 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:51.038366 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:51.038299 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:52.851889 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:52.851849 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:54.037593 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:54.037555 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf"] Oct 30 14:06:54.041850 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:54.041827 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf" Oct 30 14:06:54.066378 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:54.066349 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf"] Oct 30 14:06:54.073990 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:54.073966 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-00926c-openshift-default-dockercfg-2jkx6\"" Oct 30 14:06:54.143093 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:54.143052 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/952e947d-d7c5-476b-a066-e4f171894acc-istiod-ca-cert\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf\" (UID: \"952e947d-d7c5-476b-a066-e4f171894acc\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf" Oct 30 14:06:54.143276 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:54.143177 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/952e947d-d7c5-476b-a066-e4f171894acc-workload-socket\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf\" (UID: \"952e947d-d7c5-476b-a066-e4f171894acc\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf" Oct 30 14:06:54.143350 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:54.143283 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/952e947d-d7c5-476b-a066-e4f171894acc-istio-token\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf\" (UID: \"952e947d-d7c5-476b-a066-e4f171894acc\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf" Oct 30 14:06:54.143417 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:54.143318 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/952e947d-d7c5-476b-a066-e4f171894acc-istio-envoy\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf\" (UID: \"952e947d-d7c5-476b-a066-e4f171894acc\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf" Oct 30 14:06:54.143478 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:54.143458 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/952e947d-d7c5-476b-a066-e4f171894acc-workload-certs\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf\" (UID: \"952e947d-d7c5-476b-a066-e4f171894acc\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf" Oct 30 14:06:54.143533 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:54.143509 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/952e947d-d7c5-476b-a066-e4f171894acc-credential-socket\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf\" (UID: \"952e947d-d7c5-476b-a066-e4f171894acc\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf" Oct 30 14:06:54.143615 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:54.143600 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/952e947d-d7c5-476b-a066-e4f171894acc-istio-podinfo\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf\" (UID: \"952e947d-d7c5-476b-a066-e4f171894acc\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf" Oct 30 14:06:54.143695 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:54.143634 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/952e947d-d7c5-476b-a066-e4f171894acc-istio-data\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf\" (UID: \"952e947d-d7c5-476b-a066-e4f171894acc\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf" Oct 30 14:06:54.143794 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:54.143776 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbrm8\" (UniqueName: \"kubernetes.io/projected/952e947d-d7c5-476b-a066-e4f171894acc-kube-api-access-rbrm8\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf\" (UID: \"952e947d-d7c5-476b-a066-e4f171894acc\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf" Oct 30 14:06:54.245028 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:54.244991 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbrm8\" (UniqueName: \"kubernetes.io/projected/952e947d-d7c5-476b-a066-e4f171894acc-kube-api-access-rbrm8\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf\" (UID: \"952e947d-d7c5-476b-a066-e4f171894acc\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf" Oct 30 14:06:54.245214 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:54.245039 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/952e947d-d7c5-476b-a066-e4f171894acc-istiod-ca-cert\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf\" (UID: \"952e947d-d7c5-476b-a066-e4f171894acc\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf" Oct 30 14:06:54.245214 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:54.245083 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/952e947d-d7c5-476b-a066-e4f171894acc-workload-socket\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf\" (UID: \"952e947d-d7c5-476b-a066-e4f171894acc\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf" Oct 30 14:06:54.245214 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:54.245140 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/952e947d-d7c5-476b-a066-e4f171894acc-istio-token\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf\" (UID: \"952e947d-d7c5-476b-a066-e4f171894acc\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf" Oct 30 14:06:54.245214 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:54.245168 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/952e947d-d7c5-476b-a066-e4f171894acc-istio-envoy\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf\" (UID: \"952e947d-d7c5-476b-a066-e4f171894acc\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf" Oct 30 14:06:54.245482 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:54.245224 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/952e947d-d7c5-476b-a066-e4f171894acc-workload-certs\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf\" (UID: \"952e947d-d7c5-476b-a066-e4f171894acc\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf" Oct 30 14:06:54.245482 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:54.245270 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/952e947d-d7c5-476b-a066-e4f171894acc-credential-socket\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf\" (UID: \"952e947d-d7c5-476b-a066-e4f171894acc\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf" Oct 30 14:06:54.245482 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:54.245340 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/952e947d-d7c5-476b-a066-e4f171894acc-istio-podinfo\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf\" (UID: \"952e947d-d7c5-476b-a066-e4f171894acc\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf" Oct 30 14:06:54.245482 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:54.245371 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/952e947d-d7c5-476b-a066-e4f171894acc-istio-data\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf\" (UID: \"952e947d-d7c5-476b-a066-e4f171894acc\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf" Oct 30 14:06:54.245694 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:54.245592 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/952e947d-d7c5-476b-a066-e4f171894acc-workload-socket\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf\" (UID: \"952e947d-d7c5-476b-a066-e4f171894acc\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf" Oct 30 14:06:54.245761 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:54.245732 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/952e947d-d7c5-476b-a066-e4f171894acc-credential-socket\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf\" (UID: \"952e947d-d7c5-476b-a066-e4f171894acc\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf" Oct 30 14:06:54.245820 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:54.245735 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/952e947d-d7c5-476b-a066-e4f171894acc-workload-certs\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf\" (UID: \"952e947d-d7c5-476b-a066-e4f171894acc\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf" Oct 30 14:06:54.245890 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:54.245866 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/952e947d-d7c5-476b-a066-e4f171894acc-istio-data\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf\" (UID: \"952e947d-d7c5-476b-a066-e4f171894acc\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf" Oct 30 14:06:54.246234 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:54.246210 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/952e947d-d7c5-476b-a066-e4f171894acc-istiod-ca-cert\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf\" (UID: \"952e947d-d7c5-476b-a066-e4f171894acc\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf" Oct 30 14:06:54.247684 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:54.247662 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/952e947d-d7c5-476b-a066-e4f171894acc-istio-envoy\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf\" (UID: \"952e947d-d7c5-476b-a066-e4f171894acc\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf" Oct 30 14:06:54.247849 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:54.247830 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/952e947d-d7c5-476b-a066-e4f171894acc-istio-podinfo\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf\" (UID: \"952e947d-d7c5-476b-a066-e4f171894acc\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf" Oct 30 14:06:54.271588 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:54.271563 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbrm8\" (UniqueName: \"kubernetes.io/projected/952e947d-d7c5-476b-a066-e4f171894acc-kube-api-access-rbrm8\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf\" (UID: \"952e947d-d7c5-476b-a066-e4f171894acc\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf" Oct 30 14:06:54.271722 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:54.271561 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/952e947d-d7c5-476b-a066-e4f171894acc-istio-token\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf\" (UID: \"952e947d-d7c5-476b-a066-e4f171894acc\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf" Oct 30 14:06:54.354397 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:54.354355 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf" Oct 30 14:06:54.672954 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:54.672922 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf"] Oct 30 14:06:54.673056 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:06:54.672978 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod952e947d_d7c5_476b_a066_e4f171894acc.slice/crio-917545c31707f3302f5d57d98ba312d65d195db40382d1d51e3c8ad862e091cd WatchSource:0}: Error finding container 917545c31707f3302f5d57d98ba312d65d195db40382d1d51e3c8ad862e091cd: Status 404 returned error can't find the container with id 917545c31707f3302f5d57d98ba312d65d195db40382d1d51e3c8ad862e091cd Oct 30 14:06:54.675461 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:54.675429 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:06:54.675543 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:54.675491 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:06:54.675543 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:54.675521 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:06:54.859420 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:54.859356 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:55.331595 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:55.331552 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf" event={"ID":"952e947d-d7c5-476b-a066-e4f171894acc","Type":"ContainerStarted","Data":"108bed20f473d35b9ba53c60245d40492657338b15c2ea115fc6e016f322acf4"} Oct 30 14:06:55.332026 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:55.331619 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf" event={"ID":"952e947d-d7c5-476b-a066-e4f171894acc","Type":"ContainerStarted","Data":"917545c31707f3302f5d57d98ba312d65d195db40382d1d51e3c8ad862e091cd"} Oct 30 14:06:55.354889 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:55.354855 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf" Oct 30 14:06:55.356466 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:55.356437 4982 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 30 14:06:55.356583 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:55.356491 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf" podUID="952e947d-d7c5-476b-a066-e4f171894acc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 30 14:06:55.409003 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:55.408939 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf" podStartSLOduration=1.408921503 podStartE2EDuration="1.408921503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:06:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:06:55.395162507 +0000 UTC m=+1085.139296550" watchObservedRunningTime="2026-10-30 14:06:55.408921503 +0000 UTC m=+1085.153055547" Oct 30 14:06:56.355622 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:56.355581 4982 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 30 14:06:56.356042 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:56.355651 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf" podUID="952e947d-d7c5-476b-a066-e4f171894acc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 30 14:06:56.726070 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:56.725967 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:57.355869 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:57.355815 4982 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 30 14:06:57.356398 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:57.355884 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf" podUID="952e947d-d7c5-476b-a066-e4f171894acc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 30 14:06:58.038068 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:06:58.038030 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:06:58.355235 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:58.355201 4982 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 30 14:06:58.355424 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:58.355255 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf" podUID="952e947d-d7c5-476b-a066-e4f171894acc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 30 14:06:58.756373 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:58.756284 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:06:59.355540 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:59.355499 4982 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 30 14:06:59.355713 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:06:59.355552 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf" podUID="952e947d-d7c5-476b-a066-e4f171894acc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 30 14:07:00.354885 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:00.354842 4982 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 30 14:07:00.355383 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:00.354904 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf" podUID="952e947d-d7c5-476b-a066-e4f171894acc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 30 14:07:00.939987 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:00.939950 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:01.581927 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:01.581892 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf" Oct 30 14:07:01.583316 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:01.583282 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf" Oct 30 14:07:01.587877 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:01.587848 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf" Oct 30 14:07:02.967360 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:02.967322 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:05.162380 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:05.162332 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:07.026122 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:07.026077 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:08.926761 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:08.926728 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:10.773723 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:10.773679 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:12.649417 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:12.649379 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:14.552269 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:14.552232 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:17.198665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:17.198407 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:19.484134 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:19.484087 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:21.323208 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:21.323178 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:23.307818 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:23.307779 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:25.353814 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:25.353772 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:27.345091 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:27.345058 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:28.016391 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:07:28.016349 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:07:29.246032 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:29.245995 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:30.075364 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:30.075316 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:30.223289 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:30.223254 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:31.159664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:31.159419 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:32.879263 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:32.879222 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:33.128445 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:33.128364 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-56mz2"] Oct 30 14:07:33.133782 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:33.133725 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-56mz2" Oct 30 14:07:33.170626 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:33.170595 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-56mz2"] Oct 30 14:07:33.246874 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:33.246839 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c389875f-cfd2-4fe3-b659-d4f96c6b549f-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-56mz2\" (UID: \"c389875f-cfd2-4fe3-b659-d4f96c6b549f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-56mz2" Oct 30 14:07:33.247038 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:33.246961 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87cmz\" (UniqueName: \"kubernetes.io/projected/c389875f-cfd2-4fe3-b659-d4f96c6b549f-kube-api-access-87cmz\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-56mz2\" (UID: \"c389875f-cfd2-4fe3-b659-d4f96c6b549f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-56mz2" Oct 30 14:07:33.247038 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:33.247003 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c389875f-cfd2-4fe3-b659-d4f96c6b549f-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-56mz2\" (UID: \"c389875f-cfd2-4fe3-b659-d4f96c6b549f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-56mz2" Oct 30 14:07:33.247161 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:33.247036 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c389875f-cfd2-4fe3-b659-d4f96c6b549f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-56mz2\" (UID: \"c389875f-cfd2-4fe3-b659-d4f96c6b549f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-56mz2" Oct 30 14:07:33.247161 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:33.247061 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c389875f-cfd2-4fe3-b659-d4f96c6b549f-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-56mz2\" (UID: \"c389875f-cfd2-4fe3-b659-d4f96c6b549f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-56mz2" Oct 30 14:07:33.247161 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:33.247079 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c389875f-cfd2-4fe3-b659-d4f96c6b549f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-56mz2\" (UID: \"c389875f-cfd2-4fe3-b659-d4f96c6b549f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-56mz2" Oct 30 14:07:33.247321 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:33.247258 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c389875f-cfd2-4fe3-b659-d4f96c6b549f-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-56mz2\" (UID: \"c389875f-cfd2-4fe3-b659-d4f96c6b549f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-56mz2" Oct 30 14:07:33.247379 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:33.247339 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c389875f-cfd2-4fe3-b659-d4f96c6b549f-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-56mz2\" (UID: \"c389875f-cfd2-4fe3-b659-d4f96c6b549f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-56mz2" Oct 30 14:07:33.247418 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:33.247386 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c389875f-cfd2-4fe3-b659-d4f96c6b549f-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-56mz2\" (UID: \"c389875f-cfd2-4fe3-b659-d4f96c6b549f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-56mz2" Oct 30 14:07:33.285079 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:33.285046 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:33.348815 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:33.348776 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c389875f-cfd2-4fe3-b659-d4f96c6b549f-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-56mz2\" (UID: \"c389875f-cfd2-4fe3-b659-d4f96c6b549f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-56mz2" Oct 30 14:07:33.349027 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:33.348828 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c389875f-cfd2-4fe3-b659-d4f96c6b549f-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-56mz2\" (UID: \"c389875f-cfd2-4fe3-b659-d4f96c6b549f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-56mz2" Oct 30 14:07:33.349027 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:33.348896 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87cmz\" (UniqueName: \"kubernetes.io/projected/c389875f-cfd2-4fe3-b659-d4f96c6b549f-kube-api-access-87cmz\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-56mz2\" (UID: \"c389875f-cfd2-4fe3-b659-d4f96c6b549f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-56mz2" Oct 30 14:07:33.349027 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:33.348930 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c389875f-cfd2-4fe3-b659-d4f96c6b549f-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-56mz2\" (UID: \"c389875f-cfd2-4fe3-b659-d4f96c6b549f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-56mz2" Oct 30 14:07:33.349027 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:33.348970 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c389875f-cfd2-4fe3-b659-d4f96c6b549f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-56mz2\" (UID: \"c389875f-cfd2-4fe3-b659-d4f96c6b549f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-56mz2" Oct 30 14:07:33.349027 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:33.349004 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c389875f-cfd2-4fe3-b659-d4f96c6b549f-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-56mz2\" (UID: \"c389875f-cfd2-4fe3-b659-d4f96c6b549f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-56mz2" Oct 30 14:07:33.349340 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:33.349026 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c389875f-cfd2-4fe3-b659-d4f96c6b549f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-56mz2\" (UID: \"c389875f-cfd2-4fe3-b659-d4f96c6b549f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-56mz2" Oct 30 14:07:33.349340 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:33.349125 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c389875f-cfd2-4fe3-b659-d4f96c6b549f-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-56mz2\" (UID: \"c389875f-cfd2-4fe3-b659-d4f96c6b549f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-56mz2" Oct 30 14:07:33.349340 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:33.349187 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c389875f-cfd2-4fe3-b659-d4f96c6b549f-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-56mz2\" (UID: \"c389875f-cfd2-4fe3-b659-d4f96c6b549f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-56mz2" Oct 30 14:07:33.349788 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:33.349755 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c389875f-cfd2-4fe3-b659-d4f96c6b549f-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-56mz2\" (UID: \"c389875f-cfd2-4fe3-b659-d4f96c6b549f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-56mz2" Oct 30 14:07:33.349942 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:33.349880 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c389875f-cfd2-4fe3-b659-d4f96c6b549f-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-56mz2\" (UID: \"c389875f-cfd2-4fe3-b659-d4f96c6b549f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-56mz2" Oct 30 14:07:33.350150 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:33.350128 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c389875f-cfd2-4fe3-b659-d4f96c6b549f-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-56mz2\" (UID: \"c389875f-cfd2-4fe3-b659-d4f96c6b549f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-56mz2" Oct 30 14:07:33.350416 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:33.350395 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c389875f-cfd2-4fe3-b659-d4f96c6b549f-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-56mz2\" (UID: \"c389875f-cfd2-4fe3-b659-d4f96c6b549f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-56mz2" Oct 30 14:07:33.350456 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:33.350418 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c389875f-cfd2-4fe3-b659-d4f96c6b549f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-56mz2\" (UID: \"c389875f-cfd2-4fe3-b659-d4f96c6b549f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-56mz2" Oct 30 14:07:33.352215 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:33.352178 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c389875f-cfd2-4fe3-b659-d4f96c6b549f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-56mz2\" (UID: \"c389875f-cfd2-4fe3-b659-d4f96c6b549f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-56mz2" Oct 30 14:07:33.352617 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:33.352590 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c389875f-cfd2-4fe3-b659-d4f96c6b549f-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-56mz2\" (UID: \"c389875f-cfd2-4fe3-b659-d4f96c6b549f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-56mz2" Oct 30 14:07:33.374498 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:33.374469 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c389875f-cfd2-4fe3-b659-d4f96c6b549f-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-56mz2\" (UID: \"c389875f-cfd2-4fe3-b659-d4f96c6b549f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-56mz2" Oct 30 14:07:33.374892 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:33.374874 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87cmz\" (UniqueName: \"kubernetes.io/projected/c389875f-cfd2-4fe3-b659-d4f96c6b549f-kube-api-access-87cmz\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-56mz2\" (UID: \"c389875f-cfd2-4fe3-b659-d4f96c6b549f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-56mz2" Oct 30 14:07:33.448823 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:33.448765 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-56mz2" Oct 30 14:07:33.774356 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:07:33.774325 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc389875f_cfd2_4fe3_b659_d4f96c6b549f.slice/crio-6e2cd4ae0615c729c0a254e5d87e9ba172faf345ab5ea9538289ca4e1f1263f5 WatchSource:0}: Error finding container 6e2cd4ae0615c729c0a254e5d87e9ba172faf345ab5ea9538289ca4e1f1263f5: Status 404 returned error can't find the container with id 6e2cd4ae0615c729c0a254e5d87e9ba172faf345ab5ea9538289ca4e1f1263f5 Oct 30 14:07:33.774532 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:33.774511 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-56mz2"] Oct 30 14:07:33.776470 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:33.776442 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:07:33.776548 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:33.776498 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:07:33.776548 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:33.776528 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:07:34.707683 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:34.705830 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-56mz2" event={"ID":"c389875f-cfd2-4fe3-b659-d4f96c6b549f","Type":"ContainerStarted","Data":"b7722336c6102861fa699a5dc63bdc1df944baf7c9b5948ede06a4b61c3e1c1a"} Oct 30 14:07:34.707683 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:34.705993 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-56mz2" event={"ID":"c389875f-cfd2-4fe3-b659-d4f96c6b549f","Type":"ContainerStarted","Data":"6e2cd4ae0615c729c0a254e5d87e9ba172faf345ab5ea9538289ca4e1f1263f5"} Oct 30 14:07:34.769476 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:34.769411 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-56mz2" podStartSLOduration=1.76939778 podStartE2EDuration="1.76939778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:07:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:07:34.767728178 +0000 UTC m=+1124.511862226" watchObservedRunningTime="2026-10-30 14:07:34.76939778 +0000 UTC m=+1124.513531822" Oct 30 14:07:35.398244 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:35.398208 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:35.450141 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:35.450110 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-56mz2" Oct 30 14:07:35.452066 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:35.452040 4982 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-56mz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 30 14:07:35.452264 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:35.452245 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-56mz2" podUID="c389875f-cfd2-4fe3-b659-d4f96c6b549f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 30 14:07:36.449112 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:36.449080 4982 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-56mz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 30 14:07:36.449542 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:36.449144 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-56mz2" podUID="c389875f-cfd2-4fe3-b659-d4f96c6b549f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 30 14:07:37.419597 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:37.419558 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:37.449140 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:37.449106 4982 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-56mz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 30 14:07:37.449526 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:37.449155 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-56mz2" podUID="c389875f-cfd2-4fe3-b659-d4f96c6b549f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 30 14:07:38.449367 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:38.449330 4982 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-56mz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 30 14:07:38.449810 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:38.449383 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-56mz2" podUID="c389875f-cfd2-4fe3-b659-d4f96c6b549f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 30 14:07:39.271766 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:39.271733 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp"] Oct 30 14:07:39.275508 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:39.275480 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp" Oct 30 14:07:39.330476 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:39.330371 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp"] Oct 30 14:07:39.414233 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:39.414141 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:39.443532 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:39.443492 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ae22b83-f2b9-47c9-b51b-f70a43f699b5-istio-envoy\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp\" (UID: \"2ae22b83-f2b9-47c9-b51b-f70a43f699b5\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp" Oct 30 14:07:39.443760 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:39.443544 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ae22b83-f2b9-47c9-b51b-f70a43f699b5-istio-podinfo\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp\" (UID: \"2ae22b83-f2b9-47c9-b51b-f70a43f699b5\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp" Oct 30 14:07:39.443760 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:39.443618 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ae22b83-f2b9-47c9-b51b-f70a43f699b5-workload-certs\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp\" (UID: \"2ae22b83-f2b9-47c9-b51b-f70a43f699b5\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp" Oct 30 14:07:39.443760 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:39.443694 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae22b83-f2b9-47c9-b51b-f70a43f699b5-credential-socket\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp\" (UID: \"2ae22b83-f2b9-47c9-b51b-f70a43f699b5\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp" Oct 30 14:07:39.443760 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:39.443738 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ae22b83-f2b9-47c9-b51b-f70a43f699b5-istio-token\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp\" (UID: \"2ae22b83-f2b9-47c9-b51b-f70a43f699b5\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp" Oct 30 14:07:39.443987 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:39.443762 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae22b83-f2b9-47c9-b51b-f70a43f699b5-workload-socket\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp\" (UID: \"2ae22b83-f2b9-47c9-b51b-f70a43f699b5\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp" Oct 30 14:07:39.443987 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:39.443889 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ae22b83-f2b9-47c9-b51b-f70a43f699b5-istiod-ca-cert\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp\" (UID: \"2ae22b83-f2b9-47c9-b51b-f70a43f699b5\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp" Oct 30 14:07:39.443987 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:39.443930 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjmrk\" (UniqueName: \"kubernetes.io/projected/2ae22b83-f2b9-47c9-b51b-f70a43f699b5-kube-api-access-sjmrk\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp\" (UID: \"2ae22b83-f2b9-47c9-b51b-f70a43f699b5\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp" Oct 30 14:07:39.444130 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:39.444088 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ae22b83-f2b9-47c9-b51b-f70a43f699b5-istio-data\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp\" (UID: \"2ae22b83-f2b9-47c9-b51b-f70a43f699b5\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp" Oct 30 14:07:39.450214 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:39.450182 4982 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-56mz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 30 14:07:39.450995 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:39.450960 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-56mz2" podUID="c389875f-cfd2-4fe3-b659-d4f96c6b549f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 30 14:07:39.545821 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:39.545737 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ae22b83-f2b9-47c9-b51b-f70a43f699b5-istiod-ca-cert\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp\" (UID: \"2ae22b83-f2b9-47c9-b51b-f70a43f699b5\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp" Oct 30 14:07:39.546019 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:39.546004 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjmrk\" (UniqueName: \"kubernetes.io/projected/2ae22b83-f2b9-47c9-b51b-f70a43f699b5-kube-api-access-sjmrk\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp\" (UID: \"2ae22b83-f2b9-47c9-b51b-f70a43f699b5\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp" Oct 30 14:07:39.546198 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:39.546187 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ae22b83-f2b9-47c9-b51b-f70a43f699b5-istio-data\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp\" (UID: \"2ae22b83-f2b9-47c9-b51b-f70a43f699b5\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp" Oct 30 14:07:39.546314 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:39.546302 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ae22b83-f2b9-47c9-b51b-f70a43f699b5-istio-envoy\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp\" (UID: \"2ae22b83-f2b9-47c9-b51b-f70a43f699b5\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp" Oct 30 14:07:39.546430 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:39.546415 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ae22b83-f2b9-47c9-b51b-f70a43f699b5-istio-podinfo\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp\" (UID: \"2ae22b83-f2b9-47c9-b51b-f70a43f699b5\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp" Oct 30 14:07:39.546534 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:39.546508 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ae22b83-f2b9-47c9-b51b-f70a43f699b5-istiod-ca-cert\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp\" (UID: \"2ae22b83-f2b9-47c9-b51b-f70a43f699b5\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp" Oct 30 14:07:39.546596 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:39.546525 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ae22b83-f2b9-47c9-b51b-f70a43f699b5-istio-data\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp\" (UID: \"2ae22b83-f2b9-47c9-b51b-f70a43f699b5\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp" Oct 30 14:07:39.546709 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:39.546692 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ae22b83-f2b9-47c9-b51b-f70a43f699b5-workload-certs\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp\" (UID: \"2ae22b83-f2b9-47c9-b51b-f70a43f699b5\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp" Oct 30 14:07:39.546817 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:39.546804 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae22b83-f2b9-47c9-b51b-f70a43f699b5-credential-socket\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp\" (UID: \"2ae22b83-f2b9-47c9-b51b-f70a43f699b5\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp" Oct 30 14:07:39.546888 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:39.546875 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ae22b83-f2b9-47c9-b51b-f70a43f699b5-istio-token\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp\" (UID: \"2ae22b83-f2b9-47c9-b51b-f70a43f699b5\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp" Oct 30 14:07:39.546968 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:39.546959 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae22b83-f2b9-47c9-b51b-f70a43f699b5-workload-socket\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp\" (UID: \"2ae22b83-f2b9-47c9-b51b-f70a43f699b5\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp" Oct 30 14:07:39.547115 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:39.546839 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ae22b83-f2b9-47c9-b51b-f70a43f699b5-workload-certs\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp\" (UID: \"2ae22b83-f2b9-47c9-b51b-f70a43f699b5\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp" Oct 30 14:07:39.547174 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:39.547157 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae22b83-f2b9-47c9-b51b-f70a43f699b5-credential-socket\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp\" (UID: \"2ae22b83-f2b9-47c9-b51b-f70a43f699b5\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp" Oct 30 14:07:39.547403 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:39.547388 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae22b83-f2b9-47c9-b51b-f70a43f699b5-workload-socket\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp\" (UID: \"2ae22b83-f2b9-47c9-b51b-f70a43f699b5\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp" Oct 30 14:07:39.548814 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:39.548792 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ae22b83-f2b9-47c9-b51b-f70a43f699b5-istio-envoy\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp\" (UID: \"2ae22b83-f2b9-47c9-b51b-f70a43f699b5\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp" Oct 30 14:07:39.549121 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:39.549100 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ae22b83-f2b9-47c9-b51b-f70a43f699b5-istio-podinfo\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp\" (UID: \"2ae22b83-f2b9-47c9-b51b-f70a43f699b5\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp" Oct 30 14:07:39.571930 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:39.571897 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ae22b83-f2b9-47c9-b51b-f70a43f699b5-istio-token\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp\" (UID: \"2ae22b83-f2b9-47c9-b51b-f70a43f699b5\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp" Oct 30 14:07:39.572376 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:39.572352 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjmrk\" (UniqueName: \"kubernetes.io/projected/2ae22b83-f2b9-47c9-b51b-f70a43f699b5-kube-api-access-sjmrk\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp\" (UID: \"2ae22b83-f2b9-47c9-b51b-f70a43f699b5\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp" Oct 30 14:07:39.634825 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:39.634785 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp" Oct 30 14:07:40.003991 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:40.002494 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp"] Oct 30 14:07:40.020618 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:40.020564 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:07:40.020826 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:40.020688 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:07:40.020826 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:40.020751 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:07:40.021786 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:40.021758 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx"] Oct 30 14:07:40.027443 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:40.027418 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx" Oct 30 14:07:40.047170 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:40.047140 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx"] Oct 30 14:07:40.056210 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:40.056136 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-ca229c-openshift-default-dockercfg-52bwp\"" Oct 30 14:07:40.159600 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:40.159131 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5162bf6-1ca4-4925-aa7d-0275e3881ce8-istio-data\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx\" (UID: \"d5162bf6-1ca4-4925-aa7d-0275e3881ce8\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx" Oct 30 14:07:40.159600 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:40.159204 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48cmt\" (UniqueName: \"kubernetes.io/projected/d5162bf6-1ca4-4925-aa7d-0275e3881ce8-kube-api-access-48cmt\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx\" (UID: \"d5162bf6-1ca4-4925-aa7d-0275e3881ce8\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx" Oct 30 14:07:40.159600 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:40.159302 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5162bf6-1ca4-4925-aa7d-0275e3881ce8-credential-socket\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx\" (UID: \"d5162bf6-1ca4-4925-aa7d-0275e3881ce8\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx" Oct 30 14:07:40.159600 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:40.159340 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5162bf6-1ca4-4925-aa7d-0275e3881ce8-istio-token\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx\" (UID: \"d5162bf6-1ca4-4925-aa7d-0275e3881ce8\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx" Oct 30 14:07:40.159600 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:40.159443 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5162bf6-1ca4-4925-aa7d-0275e3881ce8-istio-podinfo\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx\" (UID: \"d5162bf6-1ca4-4925-aa7d-0275e3881ce8\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx" Oct 30 14:07:40.159600 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:40.159481 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5162bf6-1ca4-4925-aa7d-0275e3881ce8-istiod-ca-cert\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx\" (UID: \"d5162bf6-1ca4-4925-aa7d-0275e3881ce8\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx" Oct 30 14:07:40.159600 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:40.159522 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5162bf6-1ca4-4925-aa7d-0275e3881ce8-istio-envoy\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx\" (UID: \"d5162bf6-1ca4-4925-aa7d-0275e3881ce8\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx" Oct 30 14:07:40.159600 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:40.159577 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5162bf6-1ca4-4925-aa7d-0275e3881ce8-workload-socket\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx\" (UID: \"d5162bf6-1ca4-4925-aa7d-0275e3881ce8\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx" Oct 30 14:07:40.160337 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:40.159675 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5162bf6-1ca4-4925-aa7d-0275e3881ce8-workload-certs\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx\" (UID: \"d5162bf6-1ca4-4925-aa7d-0275e3881ce8\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx" Oct 30 14:07:40.260980 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:40.260852 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5162bf6-1ca4-4925-aa7d-0275e3881ce8-istio-data\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx\" (UID: \"d5162bf6-1ca4-4925-aa7d-0275e3881ce8\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx" Oct 30 14:07:40.260980 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:40.260903 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48cmt\" (UniqueName: \"kubernetes.io/projected/d5162bf6-1ca4-4925-aa7d-0275e3881ce8-kube-api-access-48cmt\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx\" (UID: \"d5162bf6-1ca4-4925-aa7d-0275e3881ce8\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx" Oct 30 14:07:40.260980 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:40.260938 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5162bf6-1ca4-4925-aa7d-0275e3881ce8-credential-socket\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx\" (UID: \"d5162bf6-1ca4-4925-aa7d-0275e3881ce8\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx" Oct 30 14:07:40.260980 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:40.260958 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5162bf6-1ca4-4925-aa7d-0275e3881ce8-istio-token\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx\" (UID: \"d5162bf6-1ca4-4925-aa7d-0275e3881ce8\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx" Oct 30 14:07:40.261282 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:40.261016 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5162bf6-1ca4-4925-aa7d-0275e3881ce8-istio-podinfo\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx\" (UID: \"d5162bf6-1ca4-4925-aa7d-0275e3881ce8\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx" Oct 30 14:07:40.261282 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:40.261060 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5162bf6-1ca4-4925-aa7d-0275e3881ce8-istiod-ca-cert\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx\" (UID: \"d5162bf6-1ca4-4925-aa7d-0275e3881ce8\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx" Oct 30 14:07:40.261282 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:40.261130 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5162bf6-1ca4-4925-aa7d-0275e3881ce8-istio-envoy\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx\" (UID: \"d5162bf6-1ca4-4925-aa7d-0275e3881ce8\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx" Oct 30 14:07:40.261282 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:40.261197 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5162bf6-1ca4-4925-aa7d-0275e3881ce8-workload-socket\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx\" (UID: \"d5162bf6-1ca4-4925-aa7d-0275e3881ce8\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx" Oct 30 14:07:40.261282 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:40.261269 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5162bf6-1ca4-4925-aa7d-0275e3881ce8-workload-certs\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx\" (UID: \"d5162bf6-1ca4-4925-aa7d-0275e3881ce8\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx" Oct 30 14:07:40.261433 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:40.261346 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5162bf6-1ca4-4925-aa7d-0275e3881ce8-istio-data\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx\" (UID: \"d5162bf6-1ca4-4925-aa7d-0275e3881ce8\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx" Oct 30 14:07:40.261692 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:40.261624 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5162bf6-1ca4-4925-aa7d-0275e3881ce8-workload-certs\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx\" (UID: \"d5162bf6-1ca4-4925-aa7d-0275e3881ce8\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx" Oct 30 14:07:40.261979 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:40.261946 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5162bf6-1ca4-4925-aa7d-0275e3881ce8-workload-socket\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx\" (UID: \"d5162bf6-1ca4-4925-aa7d-0275e3881ce8\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx" Oct 30 14:07:40.262213 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:40.262192 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5162bf6-1ca4-4925-aa7d-0275e3881ce8-credential-socket\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx\" (UID: \"d5162bf6-1ca4-4925-aa7d-0275e3881ce8\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx" Oct 30 14:07:40.262306 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:40.262243 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5162bf6-1ca4-4925-aa7d-0275e3881ce8-istiod-ca-cert\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx\" (UID: \"d5162bf6-1ca4-4925-aa7d-0275e3881ce8\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx" Oct 30 14:07:40.264113 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:40.264089 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5162bf6-1ca4-4925-aa7d-0275e3881ce8-istio-envoy\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx\" (UID: \"d5162bf6-1ca4-4925-aa7d-0275e3881ce8\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx" Oct 30 14:07:40.264451 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:40.264429 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5162bf6-1ca4-4925-aa7d-0275e3881ce8-istio-podinfo\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx\" (UID: \"d5162bf6-1ca4-4925-aa7d-0275e3881ce8\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx" Oct 30 14:07:40.287469 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:40.287430 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5162bf6-1ca4-4925-aa7d-0275e3881ce8-istio-token\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx\" (UID: \"d5162bf6-1ca4-4925-aa7d-0275e3881ce8\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx" Oct 30 14:07:40.287632 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:40.287487 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48cmt\" (UniqueName: \"kubernetes.io/projected/d5162bf6-1ca4-4925-aa7d-0275e3881ce8-kube-api-access-48cmt\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx\" (UID: \"d5162bf6-1ca4-4925-aa7d-0275e3881ce8\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx" Oct 30 14:07:40.349028 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:40.348987 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx" Oct 30 14:07:40.449364 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:40.449330 4982 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-56mz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 30 14:07:40.449551 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:40.449385 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-56mz2" podUID="c389875f-cfd2-4fe3-b659-d4f96c6b549f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 30 14:07:40.699166 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:40.698096 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx"] Oct 30 14:07:40.731956 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:40.731248 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:07:40.731956 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:40.731339 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:07:40.731956 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:40.731394 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:07:40.758123 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:40.758076 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp" event={"ID":"2ae22b83-f2b9-47c9-b51b-f70a43f699b5","Type":"ContainerStarted","Data":"8ff6d043b111d16c1d416b1b59c2487fd3cf6462c7087b77548acf73b286ed95"} Oct 30 14:07:40.758285 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:40.758146 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp" event={"ID":"2ae22b83-f2b9-47c9-b51b-f70a43f699b5","Type":"ContainerStarted","Data":"6ea1d8ed918c17e3f334e5ec877477ec529e2294c8a51cb3c51f4703fc6455b1"} Oct 30 14:07:40.759968 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:40.759934 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx" event={"ID":"d5162bf6-1ca4-4925-aa7d-0275e3881ce8","Type":"ContainerStarted","Data":"11cdf8ce2ec488e806f0680f9b134a1e9996400801468b2dc6914b5c6c0c589f"} Oct 30 14:07:40.813664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:40.813048 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp" podStartSLOduration=1.813025499 podStartE2EDuration="1.813025499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:07:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:07:40.810933269 +0000 UTC m=+1130.555067312" watchObservedRunningTime="2026-10-30 14:07:40.813025499 +0000 UTC m=+1130.557159543" Oct 30 14:07:41.449615 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:41.449571 4982 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-56mz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 30 14:07:41.449832 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:41.449636 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-56mz2" podUID="c389875f-cfd2-4fe3-b659-d4f96c6b549f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 30 14:07:41.502668 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:41.502540 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:41.642697 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:41.642048 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp" Oct 30 14:07:41.642697 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:41.642201 4982 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 30 14:07:41.642697 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:41.642247 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp" podUID="2ae22b83-f2b9-47c9-b51b-f70a43f699b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 30 14:07:41.772158 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:41.772063 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx" event={"ID":"d5162bf6-1ca4-4925-aa7d-0275e3881ce8","Type":"ContainerStarted","Data":"95a2c47dd418c2eb1f7033531d53ca7a70bb428f60d42f89cca9e09040d90380"} Oct 30 14:07:42.243676 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:42.239033 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:42.350665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:42.350074 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx" Oct 30 14:07:42.356665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:42.354252 4982 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 30 14:07:42.356665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:42.354314 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx" podUID="d5162bf6-1ca4-4925-aa7d-0275e3881ce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 30 14:07:42.645667 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:42.643102 4982 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 30 14:07:42.645667 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:42.643180 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp" podUID="2ae22b83-f2b9-47c9-b51b-f70a43f699b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 30 14:07:43.374666 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:43.368796 4982 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 30 14:07:43.374666 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:43.368856 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx" podUID="d5162bf6-1ca4-4925-aa7d-0275e3881ce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 30 14:07:43.568700 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:43.545741 4982 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-56mz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:07:43.568700 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:43.545807 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-56mz2" podUID="c389875f-cfd2-4fe3-b659-d4f96c6b549f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:07:43.640357 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:43.640259 4982 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 30 14:07:43.640622 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:43.640584 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp" podUID="2ae22b83-f2b9-47c9-b51b-f70a43f699b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 30 14:07:43.866880 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:43.866841 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-56mz2" Oct 30 14:07:43.867991 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:43.867965 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-56mz2" Oct 30 14:07:43.869079 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:43.869060 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-56mz2" Oct 30 14:07:43.948018 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:43.943707 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx" podStartSLOduration=4.943680164 podStartE2EDuration="4.943680164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:07:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:07:41.839151157 +0000 UTC m=+1131.583285193" watchObservedRunningTime="2026-10-30 14:07:43.943680164 +0000 UTC m=+1133.687814231" Oct 30 14:07:44.030666 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:44.025607 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:44.350312 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:44.350263 4982 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 30 14:07:44.350500 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:44.350323 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx" podUID="d5162bf6-1ca4-4925-aa7d-0275e3881ce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 30 14:07:44.641424 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:44.639067 4982 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 30 14:07:44.641424 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:44.639137 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp" podUID="2ae22b83-f2b9-47c9-b51b-f70a43f699b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 30 14:07:45.351675 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:45.350049 4982 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 30 14:07:45.351675 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:45.350115 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx" podUID="d5162bf6-1ca4-4925-aa7d-0275e3881ce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 30 14:07:45.639887 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:45.639779 4982 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 30 14:07:45.640144 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:45.640113 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp" podUID="2ae22b83-f2b9-47c9-b51b-f70a43f699b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 30 14:07:46.357214 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:46.357178 4982 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 30 14:07:46.357781 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:46.357749 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx" podUID="d5162bf6-1ca4-4925-aa7d-0275e3881ce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 30 14:07:46.512142 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:46.512103 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:46.636038 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:46.635945 4982 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 30 14:07:46.636038 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:46.635998 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp" podUID="2ae22b83-f2b9-47c9-b51b-f70a43f699b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 30 14:07:47.350108 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:47.350049 4982 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 30 14:07:47.350282 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:47.350135 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx" podUID="d5162bf6-1ca4-4925-aa7d-0275e3881ce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 30 14:07:47.643746 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:47.643621 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp" Oct 30 14:07:47.646442 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:47.646417 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp" Oct 30 14:07:47.651789 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:47.651763 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp" Oct 30 14:07:48.353840 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:48.353804 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx" Oct 30 14:07:48.354181 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:48.354153 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx" Oct 30 14:07:48.354976 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:48.354955 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx" Oct 30 14:07:48.525598 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:48.525553 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:50.557526 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:50.557475 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:52.170430 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:52.170396 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:52.794761 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:52.794704 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:54.739734 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:54.739697 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84"] Oct 30 14:07:54.744437 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:54.744414 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84" Oct 30 14:07:54.754943 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:54.754911 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:54.777755 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:54.777724 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84"] Oct 30 14:07:54.848632 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:54.848595 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0991b98-e891-4582-a9ce-88846fcaad92-credential-socket\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84\" (UID: \"e0991b98-e891-4582-a9ce-88846fcaad92\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84" Oct 30 14:07:54.848632 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:54.848652 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0991b98-e891-4582-a9ce-88846fcaad92-istiod-ca-cert\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84\" (UID: \"e0991b98-e891-4582-a9ce-88846fcaad92\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84" Oct 30 14:07:54.848909 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:54.848745 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnmtb\" (UniqueName: \"kubernetes.io/projected/e0991b98-e891-4582-a9ce-88846fcaad92-kube-api-access-wnmtb\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84\" (UID: \"e0991b98-e891-4582-a9ce-88846fcaad92\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84" Oct 30 14:07:54.848909 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:54.848786 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0991b98-e891-4582-a9ce-88846fcaad92-istio-token\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84\" (UID: \"e0991b98-e891-4582-a9ce-88846fcaad92\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84" Oct 30 14:07:54.849020 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:54.848911 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0991b98-e891-4582-a9ce-88846fcaad92-istio-podinfo\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84\" (UID: \"e0991b98-e891-4582-a9ce-88846fcaad92\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84" Oct 30 14:07:54.849077 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:54.849020 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0991b98-e891-4582-a9ce-88846fcaad92-workload-socket\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84\" (UID: \"e0991b98-e891-4582-a9ce-88846fcaad92\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84" Oct 30 14:07:54.849077 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:54.849063 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0991b98-e891-4582-a9ce-88846fcaad92-workload-certs\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84\" (UID: \"e0991b98-e891-4582-a9ce-88846fcaad92\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84" Oct 30 14:07:54.849224 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:54.849207 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0991b98-e891-4582-a9ce-88846fcaad92-istio-envoy\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84\" (UID: \"e0991b98-e891-4582-a9ce-88846fcaad92\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84" Oct 30 14:07:54.849286 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:54.849266 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0991b98-e891-4582-a9ce-88846fcaad92-istio-data\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84\" (UID: \"e0991b98-e891-4582-a9ce-88846fcaad92\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84" Oct 30 14:07:54.905110 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:54.905065 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w"] Oct 30 14:07:54.909368 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:54.909343 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w" Oct 30 14:07:54.937969 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:54.937942 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w"] Oct 30 14:07:54.950432 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:54.950405 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0991b98-e891-4582-a9ce-88846fcaad92-istio-envoy\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84\" (UID: \"e0991b98-e891-4582-a9ce-88846fcaad92\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84" Oct 30 14:07:54.950571 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:54.950457 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0991b98-e891-4582-a9ce-88846fcaad92-istio-data\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84\" (UID: \"e0991b98-e891-4582-a9ce-88846fcaad92\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84" Oct 30 14:07:54.950571 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:54.950486 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0991b98-e891-4582-a9ce-88846fcaad92-credential-socket\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84\" (UID: \"e0991b98-e891-4582-a9ce-88846fcaad92\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84" Oct 30 14:07:54.950571 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:54.950502 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0991b98-e891-4582-a9ce-88846fcaad92-istiod-ca-cert\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84\" (UID: \"e0991b98-e891-4582-a9ce-88846fcaad92\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84" Oct 30 14:07:54.950571 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:54.950526 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnmtb\" (UniqueName: \"kubernetes.io/projected/e0991b98-e891-4582-a9ce-88846fcaad92-kube-api-access-wnmtb\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84\" (UID: \"e0991b98-e891-4582-a9ce-88846fcaad92\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84" Oct 30 14:07:54.950571 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:54.950563 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0991b98-e891-4582-a9ce-88846fcaad92-istio-token\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84\" (UID: \"e0991b98-e891-4582-a9ce-88846fcaad92\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84" Oct 30 14:07:54.950866 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:54.950594 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw65m\" (UniqueName: \"kubernetes.io/projected/7db57691-2261-4a03-abe3-a0fa7bb45429-kube-api-access-kw65m\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w\" (UID: \"7db57691-2261-4a03-abe3-a0fa7bb45429\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w" Oct 30 14:07:54.950866 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:54.950627 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7db57691-2261-4a03-abe3-a0fa7bb45429-workload-certs\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w\" (UID: \"7db57691-2261-4a03-abe3-a0fa7bb45429\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w" Oct 30 14:07:54.950866 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:54.950665 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7db57691-2261-4a03-abe3-a0fa7bb45429-istiod-ca-cert\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w\" (UID: \"7db57691-2261-4a03-abe3-a0fa7bb45429\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w" Oct 30 14:07:54.950866 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:54.950709 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7db57691-2261-4a03-abe3-a0fa7bb45429-istio-envoy\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w\" (UID: \"7db57691-2261-4a03-abe3-a0fa7bb45429\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w" Oct 30 14:07:54.950866 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:54.950733 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7db57691-2261-4a03-abe3-a0fa7bb45429-istio-data\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w\" (UID: \"7db57691-2261-4a03-abe3-a0fa7bb45429\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w" Oct 30 14:07:54.950866 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:54.950792 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0991b98-e891-4582-a9ce-88846fcaad92-istio-podinfo\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84\" (UID: \"e0991b98-e891-4582-a9ce-88846fcaad92\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84" Oct 30 14:07:54.950866 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:54.950829 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7db57691-2261-4a03-abe3-a0fa7bb45429-workload-socket\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w\" (UID: \"7db57691-2261-4a03-abe3-a0fa7bb45429\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w" Oct 30 14:07:54.951209 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:54.950895 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0991b98-e891-4582-a9ce-88846fcaad92-workload-socket\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84\" (UID: \"e0991b98-e891-4582-a9ce-88846fcaad92\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84" Oct 30 14:07:54.951209 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:54.950937 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0991b98-e891-4582-a9ce-88846fcaad92-workload-certs\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84\" (UID: \"e0991b98-e891-4582-a9ce-88846fcaad92\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84" Oct 30 14:07:54.951209 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:54.950963 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7db57691-2261-4a03-abe3-a0fa7bb45429-istio-podinfo\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w\" (UID: \"7db57691-2261-4a03-abe3-a0fa7bb45429\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w" Oct 30 14:07:54.952006 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:54.951418 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0991b98-e891-4582-a9ce-88846fcaad92-workload-socket\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84\" (UID: \"e0991b98-e891-4582-a9ce-88846fcaad92\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84" Oct 30 14:07:54.952006 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:54.951476 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7db57691-2261-4a03-abe3-a0fa7bb45429-istio-token\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w\" (UID: \"7db57691-2261-4a03-abe3-a0fa7bb45429\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w" Oct 30 14:07:54.952006 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:54.951710 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0991b98-e891-4582-a9ce-88846fcaad92-workload-certs\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84\" (UID: \"e0991b98-e891-4582-a9ce-88846fcaad92\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84" Oct 30 14:07:54.952006 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:54.951938 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7db57691-2261-4a03-abe3-a0fa7bb45429-credential-socket\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w\" (UID: \"7db57691-2261-4a03-abe3-a0fa7bb45429\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w" Oct 30 14:07:54.952533 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:54.952505 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0991b98-e891-4582-a9ce-88846fcaad92-istiod-ca-cert\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84\" (UID: \"e0991b98-e891-4582-a9ce-88846fcaad92\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84" Oct 30 14:07:54.952850 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:54.952808 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0991b98-e891-4582-a9ce-88846fcaad92-istio-data\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84\" (UID: \"e0991b98-e891-4582-a9ce-88846fcaad92\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84" Oct 30 14:07:54.953182 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:54.953159 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0991b98-e891-4582-a9ce-88846fcaad92-credential-socket\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84\" (UID: \"e0991b98-e891-4582-a9ce-88846fcaad92\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84" Oct 30 14:07:54.953375 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:54.953359 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0991b98-e891-4582-a9ce-88846fcaad92-istio-envoy\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84\" (UID: \"e0991b98-e891-4582-a9ce-88846fcaad92\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84" Oct 30 14:07:54.953764 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:54.953747 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0991b98-e891-4582-a9ce-88846fcaad92-istio-podinfo\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84\" (UID: \"e0991b98-e891-4582-a9ce-88846fcaad92\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84" Oct 30 14:07:54.982275 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:54.982246 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0991b98-e891-4582-a9ce-88846fcaad92-istio-token\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84\" (UID: \"e0991b98-e891-4582-a9ce-88846fcaad92\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84" Oct 30 14:07:54.982382 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:54.982359 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnmtb\" (UniqueName: \"kubernetes.io/projected/e0991b98-e891-4582-a9ce-88846fcaad92-kube-api-access-wnmtb\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84\" (UID: \"e0991b98-e891-4582-a9ce-88846fcaad92\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84" Oct 30 14:07:55.055455 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:55.054035 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7db57691-2261-4a03-abe3-a0fa7bb45429-istio-podinfo\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w\" (UID: \"7db57691-2261-4a03-abe3-a0fa7bb45429\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w" Oct 30 14:07:55.055455 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:55.054127 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7db57691-2261-4a03-abe3-a0fa7bb45429-istio-token\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w\" (UID: \"7db57691-2261-4a03-abe3-a0fa7bb45429\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w" Oct 30 14:07:55.055455 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:55.054204 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7db57691-2261-4a03-abe3-a0fa7bb45429-credential-socket\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w\" (UID: \"7db57691-2261-4a03-abe3-a0fa7bb45429\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w" Oct 30 14:07:55.055455 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:55.054305 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw65m\" (UniqueName: \"kubernetes.io/projected/7db57691-2261-4a03-abe3-a0fa7bb45429-kube-api-access-kw65m\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w\" (UID: \"7db57691-2261-4a03-abe3-a0fa7bb45429\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w" Oct 30 14:07:55.055455 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:55.054339 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7db57691-2261-4a03-abe3-a0fa7bb45429-workload-certs\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w\" (UID: \"7db57691-2261-4a03-abe3-a0fa7bb45429\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w" Oct 30 14:07:55.055455 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:55.054367 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7db57691-2261-4a03-abe3-a0fa7bb45429-istiod-ca-cert\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w\" (UID: \"7db57691-2261-4a03-abe3-a0fa7bb45429\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w" Oct 30 14:07:55.055455 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:55.054413 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7db57691-2261-4a03-abe3-a0fa7bb45429-istio-envoy\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w\" (UID: \"7db57691-2261-4a03-abe3-a0fa7bb45429\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w" Oct 30 14:07:55.055455 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:55.054445 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7db57691-2261-4a03-abe3-a0fa7bb45429-istio-data\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w\" (UID: \"7db57691-2261-4a03-abe3-a0fa7bb45429\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w" Oct 30 14:07:55.055455 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:55.054517 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7db57691-2261-4a03-abe3-a0fa7bb45429-workload-socket\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w\" (UID: \"7db57691-2261-4a03-abe3-a0fa7bb45429\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w" Oct 30 14:07:55.055455 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:55.055015 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7db57691-2261-4a03-abe3-a0fa7bb45429-workload-socket\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w\" (UID: \"7db57691-2261-4a03-abe3-a0fa7bb45429\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w" Oct 30 14:07:55.060478 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:55.058504 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7db57691-2261-4a03-abe3-a0fa7bb45429-istio-podinfo\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w\" (UID: \"7db57691-2261-4a03-abe3-a0fa7bb45429\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w" Oct 30 14:07:55.060478 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:55.059094 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7db57691-2261-4a03-abe3-a0fa7bb45429-credential-socket\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w\" (UID: \"7db57691-2261-4a03-abe3-a0fa7bb45429\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w" Oct 30 14:07:55.060478 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:55.059570 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7db57691-2261-4a03-abe3-a0fa7bb45429-workload-certs\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w\" (UID: \"7db57691-2261-4a03-abe3-a0fa7bb45429\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w" Oct 30 14:07:55.060478 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:55.060180 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7db57691-2261-4a03-abe3-a0fa7bb45429-istiod-ca-cert\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w\" (UID: \"7db57691-2261-4a03-abe3-a0fa7bb45429\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w" Oct 30 14:07:55.062454 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:55.062432 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84" Oct 30 14:07:55.062703 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:55.062537 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7db57691-2261-4a03-abe3-a0fa7bb45429-istio-data\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w\" (UID: \"7db57691-2261-4a03-abe3-a0fa7bb45429\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w" Oct 30 14:07:55.063072 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:55.063043 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7db57691-2261-4a03-abe3-a0fa7bb45429-istio-envoy\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w\" (UID: \"7db57691-2261-4a03-abe3-a0fa7bb45429\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w" Oct 30 14:07:55.112375 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:55.112339 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7db57691-2261-4a03-abe3-a0fa7bb45429-istio-token\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w\" (UID: \"7db57691-2261-4a03-abe3-a0fa7bb45429\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w" Oct 30 14:07:55.126630 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:55.126560 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw65m\" (UniqueName: \"kubernetes.io/projected/7db57691-2261-4a03-abe3-a0fa7bb45429-kube-api-access-kw65m\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w\" (UID: \"7db57691-2261-4a03-abe3-a0fa7bb45429\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w" Oct 30 14:07:55.227113 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:55.227080 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w" Oct 30 14:07:55.392303 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:07:55.392270 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0991b98_e891_4582_a9ce_88846fcaad92.slice/crio-388eec4e88ceff7d7e268019b0547558f5091644c60dc23897d6420fafe35db8 WatchSource:0}: Error finding container 388eec4e88ceff7d7e268019b0547558f5091644c60dc23897d6420fafe35db8: Status 404 returned error can't find the container with id 388eec4e88ceff7d7e268019b0547558f5091644c60dc23897d6420fafe35db8 Oct 30 14:07:55.393955 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:55.393929 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84"] Oct 30 14:07:55.395094 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:55.395060 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:07:55.395192 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:55.395138 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:07:55.395192 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:55.395180 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:07:55.535845 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:55.535820 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w"] Oct 30 14:07:55.537265 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:07:55.537240 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7db57691_2261_4a03_abe3_a0fa7bb45429.slice/crio-aabd41ca0b34aa9ab038e58a9e1238e15df738b4bf34e16910e95056576a5392 WatchSource:0}: Error finding container aabd41ca0b34aa9ab038e58a9e1238e15df738b4bf34e16910e95056576a5392: Status 404 returned error can't find the container with id aabd41ca0b34aa9ab038e58a9e1238e15df738b4bf34e16910e95056576a5392 Oct 30 14:07:55.540764 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:55.540732 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:07:55.540861 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:55.540809 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:07:55.540924 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:55.540857 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:07:55.921207 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:55.920916 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w" event={"ID":"7db57691-2261-4a03-abe3-a0fa7bb45429","Type":"ContainerStarted","Data":"2a2859a9d0d837ecc3a1e65a024a68b3cc7954ef8cce3bc1b7fb288607668961"} Oct 30 14:07:55.921207 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:55.920980 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w" event={"ID":"7db57691-2261-4a03-abe3-a0fa7bb45429","Type":"ContainerStarted","Data":"aabd41ca0b34aa9ab038e58a9e1238e15df738b4bf34e16910e95056576a5392"} Oct 30 14:07:55.922774 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:55.922747 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84" event={"ID":"e0991b98-e891-4582-a9ce-88846fcaad92","Type":"ContainerStarted","Data":"f50889a5471bfa929aca7f9a76117b150a51172237b53872b17585b70b801543"} Oct 30 14:07:55.922900 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:55.922790 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84" event={"ID":"e0991b98-e891-4582-a9ce-88846fcaad92","Type":"ContainerStarted","Data":"388eec4e88ceff7d7e268019b0547558f5091644c60dc23897d6420fafe35db8"} Oct 30 14:07:55.972740 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:55.972689 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w" podStartSLOduration=1.972675248 podStartE2EDuration="1.972675248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:07:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:07:55.970089282 +0000 UTC m=+1145.714223326" watchObservedRunningTime="2026-10-30 14:07:55.972675248 +0000 UTC m=+1145.716809290" Oct 30 14:07:56.019622 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:56.019575 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84" podStartSLOduration=2.019561168 podStartE2EDuration="2.019561168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:07:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:07:56.018229882 +0000 UTC m=+1145.762363924" watchObservedRunningTime="2026-10-30 14:07:56.019561168 +0000 UTC m=+1145.763695209" Oct 30 14:07:56.063582 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:56.063550 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84" Oct 30 14:07:56.064955 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:56.064927 4982 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Oct 30 14:07:56.065070 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:56.064976 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84" podUID="e0991b98-e891-4582-a9ce-88846fcaad92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Oct 30 14:07:56.228139 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:56.228031 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w" Oct 30 14:07:56.230898 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:56.230857 4982 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 30 14:07:56.231080 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:56.230923 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w" podUID="7db57691-2261-4a03-abe3-a0fa7bb45429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 30 14:07:56.759452 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:56.759067 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:57.062834 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:57.062802 4982 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Oct 30 14:07:57.063197 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:57.062850 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84" podUID="e0991b98-e891-4582-a9ce-88846fcaad92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Oct 30 14:07:57.231670 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:57.231031 4982 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 30 14:07:57.231670 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:57.231095 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w" podUID="7db57691-2261-4a03-abe3-a0fa7bb45429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 30 14:07:58.063280 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:58.063237 4982 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Oct 30 14:07:58.063661 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:58.063411 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84" podUID="e0991b98-e891-4582-a9ce-88846fcaad92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Oct 30 14:07:58.067131 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:07:58.067104 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:07:58.228664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:58.228553 4982 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 30 14:07:58.228664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:58.228616 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w" podUID="7db57691-2261-4a03-abe3-a0fa7bb45429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 30 14:07:58.750813 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:58.750776 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:07:59.062983 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:59.062847 4982 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Oct 30 14:07:59.062983 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:59.062897 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84" podUID="e0991b98-e891-4582-a9ce-88846fcaad92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Oct 30 14:07:59.230855 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:59.230280 4982 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 30 14:07:59.230855 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:07:59.230358 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w" podUID="7db57691-2261-4a03-abe3-a0fa7bb45429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 30 14:08:00.063522 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:00.063483 4982 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Oct 30 14:08:00.063698 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:00.063544 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84" podUID="e0991b98-e891-4582-a9ce-88846fcaad92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Oct 30 14:08:00.228103 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:00.228063 4982 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 30 14:08:00.228412 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:00.228366 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w" podUID="7db57691-2261-4a03-abe3-a0fa7bb45429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 30 14:08:00.897133 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:00.897095 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:01.063217 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:01.063166 4982 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Oct 30 14:08:01.063412 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:01.063223 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84" podUID="e0991b98-e891-4582-a9ce-88846fcaad92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Oct 30 14:08:01.233121 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:01.233023 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:01.233606 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:01.233585 4982 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 30 14:08:01.233803 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:01.233778 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w" podUID="7db57691-2261-4a03-abe3-a0fa7bb45429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 30 14:08:02.063924 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:02.063873 4982 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Oct 30 14:08:02.064412 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:02.063943 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84" podUID="e0991b98-e891-4582-a9ce-88846fcaad92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Oct 30 14:08:02.227535 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:02.227494 4982 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 30 14:08:02.227744 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:02.227565 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w" podUID="7db57691-2261-4a03-abe3-a0fa7bb45429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 30 14:08:02.727621 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:02.727583 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:03.067701 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:03.067670 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84" Oct 30 14:08:03.068182 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:03.068041 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84" Oct 30 14:08:03.069025 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:03.069003 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84" Oct 30 14:08:03.232258 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:03.232222 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w" Oct 30 14:08:03.232636 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:03.232612 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w" Oct 30 14:08:03.233314 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:03.233293 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w" Oct 30 14:08:04.625614 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:04.625572 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:06.425904 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:06.425861 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:08.488386 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:08.488343 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:10.961269 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:10.961203 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:12.904974 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:12.904940 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:14.742342 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:14.742306 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:16.766818 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:16.766785 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:18.780218 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:18.780172 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:20.596161 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:20.596118 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:20.740358 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:20.740321 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:22.642443 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:22.642412 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:24.532338 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:24.532295 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4"] Oct 30 14:08:24.536801 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:24.536776 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4" Oct 30 14:08:24.571009 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:24.570968 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4"] Oct 30 14:08:24.638898 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:24.638853 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b14bb682-071a-4c22-ad2d-cab7434a4192-workload-socket\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4\" (UID: \"b14bb682-071a-4c22-ad2d-cab7434a4192\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4" Oct 30 14:08:24.639113 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:24.638907 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b14bb682-071a-4c22-ad2d-cab7434a4192-credential-socket\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4\" (UID: \"b14bb682-071a-4c22-ad2d-cab7434a4192\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4" Oct 30 14:08:24.639113 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:24.639006 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b14bb682-071a-4c22-ad2d-cab7434a4192-istiod-ca-cert\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4\" (UID: \"b14bb682-071a-4c22-ad2d-cab7434a4192\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4" Oct 30 14:08:24.639113 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:24.639035 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b14bb682-071a-4c22-ad2d-cab7434a4192-istio-envoy\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4\" (UID: \"b14bb682-071a-4c22-ad2d-cab7434a4192\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4" Oct 30 14:08:24.639113 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:24.639069 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b14bb682-071a-4c22-ad2d-cab7434a4192-istio-token\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4\" (UID: \"b14bb682-071a-4c22-ad2d-cab7434a4192\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4" Oct 30 14:08:24.639413 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:24.639159 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b14bb682-071a-4c22-ad2d-cab7434a4192-istio-data\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4\" (UID: \"b14bb682-071a-4c22-ad2d-cab7434a4192\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4" Oct 30 14:08:24.639413 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:24.639244 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b14bb682-071a-4c22-ad2d-cab7434a4192-workload-certs\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4\" (UID: \"b14bb682-071a-4c22-ad2d-cab7434a4192\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4" Oct 30 14:08:24.639413 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:24.639307 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jk4t\" (UniqueName: \"kubernetes.io/projected/b14bb682-071a-4c22-ad2d-cab7434a4192-kube-api-access-9jk4t\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4\" (UID: \"b14bb682-071a-4c22-ad2d-cab7434a4192\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4" Oct 30 14:08:24.639574 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:24.639428 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b14bb682-071a-4c22-ad2d-cab7434a4192-istio-podinfo\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4\" (UID: \"b14bb682-071a-4c22-ad2d-cab7434a4192\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4" Oct 30 14:08:24.640792 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:24.640761 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:24.741384 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:24.741343 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b14bb682-071a-4c22-ad2d-cab7434a4192-istio-token\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4\" (UID: \"b14bb682-071a-4c22-ad2d-cab7434a4192\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4" Oct 30 14:08:24.741566 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:24.741403 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b14bb682-071a-4c22-ad2d-cab7434a4192-istio-data\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4\" (UID: \"b14bb682-071a-4c22-ad2d-cab7434a4192\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4" Oct 30 14:08:24.741566 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:24.741493 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b14bb682-071a-4c22-ad2d-cab7434a4192-workload-certs\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4\" (UID: \"b14bb682-071a-4c22-ad2d-cab7434a4192\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4" Oct 30 14:08:24.741742 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:24.741575 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jk4t\" (UniqueName: \"kubernetes.io/projected/b14bb682-071a-4c22-ad2d-cab7434a4192-kube-api-access-9jk4t\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4\" (UID: \"b14bb682-071a-4c22-ad2d-cab7434a4192\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4" Oct 30 14:08:24.741742 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:24.741656 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b14bb682-071a-4c22-ad2d-cab7434a4192-istio-podinfo\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4\" (UID: \"b14bb682-071a-4c22-ad2d-cab7434a4192\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4" Oct 30 14:08:24.741742 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:24.741721 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b14bb682-071a-4c22-ad2d-cab7434a4192-workload-socket\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4\" (UID: \"b14bb682-071a-4c22-ad2d-cab7434a4192\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4" Oct 30 14:08:24.741893 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:24.741769 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b14bb682-071a-4c22-ad2d-cab7434a4192-credential-socket\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4\" (UID: \"b14bb682-071a-4c22-ad2d-cab7434a4192\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4" Oct 30 14:08:24.741893 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:24.741875 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b14bb682-071a-4c22-ad2d-cab7434a4192-istiod-ca-cert\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4\" (UID: \"b14bb682-071a-4c22-ad2d-cab7434a4192\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4" Oct 30 14:08:24.741996 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:24.741917 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b14bb682-071a-4c22-ad2d-cab7434a4192-istio-envoy\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4\" (UID: \"b14bb682-071a-4c22-ad2d-cab7434a4192\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4" Oct 30 14:08:24.742979 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:24.742944 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b14bb682-071a-4c22-ad2d-cab7434a4192-istio-data\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4\" (UID: \"b14bb682-071a-4c22-ad2d-cab7434a4192\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4" Oct 30 14:08:24.744403 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:24.744353 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b14bb682-071a-4c22-ad2d-cab7434a4192-credential-socket\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4\" (UID: \"b14bb682-071a-4c22-ad2d-cab7434a4192\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4" Oct 30 14:08:24.744836 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:24.744809 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b14bb682-071a-4c22-ad2d-cab7434a4192-workload-socket\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4\" (UID: \"b14bb682-071a-4c22-ad2d-cab7434a4192\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4" Oct 30 14:08:24.744976 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:24.744910 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b14bb682-071a-4c22-ad2d-cab7434a4192-istiod-ca-cert\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4\" (UID: \"b14bb682-071a-4c22-ad2d-cab7434a4192\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4" Oct 30 14:08:24.745233 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:24.745210 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b14bb682-071a-4c22-ad2d-cab7434a4192-workload-certs\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4\" (UID: \"b14bb682-071a-4c22-ad2d-cab7434a4192\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4" Oct 30 14:08:24.746895 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:24.746870 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b14bb682-071a-4c22-ad2d-cab7434a4192-istio-envoy\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4\" (UID: \"b14bb682-071a-4c22-ad2d-cab7434a4192\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4" Oct 30 14:08:24.747918 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:24.747891 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b14bb682-071a-4c22-ad2d-cab7434a4192-istio-podinfo\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4\" (UID: \"b14bb682-071a-4c22-ad2d-cab7434a4192\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4" Oct 30 14:08:24.767569 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:24.767528 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b14bb682-071a-4c22-ad2d-cab7434a4192-istio-token\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4\" (UID: \"b14bb682-071a-4c22-ad2d-cab7434a4192\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4" Oct 30 14:08:24.767903 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:24.767877 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jk4t\" (UniqueName: \"kubernetes.io/projected/b14bb682-071a-4c22-ad2d-cab7434a4192-kube-api-access-9jk4t\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4\" (UID: \"b14bb682-071a-4c22-ad2d-cab7434a4192\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4" Oct 30 14:08:24.854592 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:24.854555 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4" Oct 30 14:08:25.186682 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:25.186622 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4"] Oct 30 14:08:25.187034 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:08:25.187011 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb14bb682_071a_4c22_ad2d_cab7434a4192.slice/crio-66448dafa9336847254cbac81cfebb15676c158f26fe8d7b542cd300f017184c WatchSource:0}: Error finding container 66448dafa9336847254cbac81cfebb15676c158f26fe8d7b542cd300f017184c: Status 404 returned error can't find the container with id 66448dafa9336847254cbac81cfebb15676c158f26fe8d7b542cd300f017184c Oct 30 14:08:25.189457 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:25.189420 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:08:25.189518 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:25.189491 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:08:25.189563 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:25.189520 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:08:26.174977 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:26.174907 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4" event={"ID":"b14bb682-071a-4c22-ad2d-cab7434a4192","Type":"ContainerStarted","Data":"31e08d6c89814d4efd0c371849a1e652cd204be7f967ab5a584e85e3d4e9ecd5"} Oct 30 14:08:26.175501 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:26.174995 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4" event={"ID":"b14bb682-071a-4c22-ad2d-cab7434a4192","Type":"ContainerStarted","Data":"66448dafa9336847254cbac81cfebb15676c158f26fe8d7b542cd300f017184c"} Oct 30 14:08:26.267795 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:26.267732 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4" podStartSLOduration=2.2677189159999998 podStartE2EDuration="2.267718916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:08:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:08:26.262244436 +0000 UTC m=+1176.006378480" watchObservedRunningTime="2026-10-30 14:08:26.267718916 +0000 UTC m=+1176.011852957" Oct 30 14:08:26.610679 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:26.609951 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:26.855526 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:26.855483 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4" Oct 30 14:08:26.856851 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:26.856820 4982 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 30 14:08:26.856982 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:26.856874 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4" podUID="b14bb682-071a-4c22-ad2d-cab7434a4192" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 30 14:08:27.855295 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:27.855256 4982 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 30 14:08:27.855822 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:27.855319 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4" podUID="b14bb682-071a-4c22-ad2d-cab7434a4192" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 30 14:08:28.033472 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:08:28.033366 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:08:28.706671 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:28.706361 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:28.855713 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:28.855669 4982 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 30 14:08:28.856107 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:28.855728 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4" podUID="b14bb682-071a-4c22-ad2d-cab7434a4192" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 30 14:08:29.855492 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:29.855452 4982 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 30 14:08:29.855701 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:29.855523 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4" podUID="b14bb682-071a-4c22-ad2d-cab7434a4192" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 30 14:08:30.778060 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:30.778026 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:30.855040 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:30.855004 4982 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 30 14:08:30.855345 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:30.855312 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4" podUID="b14bb682-071a-4c22-ad2d-cab7434a4192" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 30 14:08:31.855311 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:31.855266 4982 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 30 14:08:31.855789 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:31.855335 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4" podUID="b14bb682-071a-4c22-ad2d-cab7434a4192" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 30 14:08:32.859961 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:32.859930 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4" Oct 30 14:08:32.860458 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:32.860438 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4" Oct 30 14:08:32.861077 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:32.861055 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4" Oct 30 14:08:32.935311 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:32.935221 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:35.316009 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:35.315977 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:37.293372 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:37.293334 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:39.128628 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:39.128583 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:39.566378 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:39.566344 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk"] Oct 30 14:08:39.570477 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:39.570454 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk" Oct 30 14:08:39.605820 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:39.605780 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk"] Oct 30 14:08:39.681811 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:39.681777 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8fa477f-0e45-42fa-9941-514ea90d1094-istio-token\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk\" (UID: \"e8fa477f-0e45-42fa-9941-514ea90d1094\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk" Oct 30 14:08:39.681962 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:39.681876 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8fa477f-0e45-42fa-9941-514ea90d1094-istio-data\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk\" (UID: \"e8fa477f-0e45-42fa-9941-514ea90d1094\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk" Oct 30 14:08:39.681962 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:39.681934 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8fa477f-0e45-42fa-9941-514ea90d1094-credential-socket\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk\" (UID: \"e8fa477f-0e45-42fa-9941-514ea90d1094\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk" Oct 30 14:08:39.682049 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:39.681998 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8fa477f-0e45-42fa-9941-514ea90d1094-istio-envoy\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk\" (UID: \"e8fa477f-0e45-42fa-9941-514ea90d1094\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk" Oct 30 14:08:39.682084 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:39.682048 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8fa477f-0e45-42fa-9941-514ea90d1094-workload-socket\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk\" (UID: \"e8fa477f-0e45-42fa-9941-514ea90d1094\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk" Oct 30 14:08:39.682124 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:39.682093 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6gz4\" (UniqueName: \"kubernetes.io/projected/e8fa477f-0e45-42fa-9941-514ea90d1094-kube-api-access-c6gz4\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk\" (UID: \"e8fa477f-0e45-42fa-9941-514ea90d1094\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk" Oct 30 14:08:39.682189 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:39.682172 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8fa477f-0e45-42fa-9941-514ea90d1094-istiod-ca-cert\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk\" (UID: \"e8fa477f-0e45-42fa-9941-514ea90d1094\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk" Oct 30 14:08:39.682226 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:39.682205 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8fa477f-0e45-42fa-9941-514ea90d1094-workload-certs\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk\" (UID: \"e8fa477f-0e45-42fa-9941-514ea90d1094\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk" Oct 30 14:08:39.682352 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:39.682335 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8fa477f-0e45-42fa-9941-514ea90d1094-istio-podinfo\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk\" (UID: \"e8fa477f-0e45-42fa-9941-514ea90d1094\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk" Oct 30 14:08:39.783429 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:39.783395 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8fa477f-0e45-42fa-9941-514ea90d1094-istio-token\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk\" (UID: \"e8fa477f-0e45-42fa-9941-514ea90d1094\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk" Oct 30 14:08:39.783615 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:39.783444 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8fa477f-0e45-42fa-9941-514ea90d1094-istio-data\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk\" (UID: \"e8fa477f-0e45-42fa-9941-514ea90d1094\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk" Oct 30 14:08:39.783615 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:39.783530 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8fa477f-0e45-42fa-9941-514ea90d1094-credential-socket\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk\" (UID: \"e8fa477f-0e45-42fa-9941-514ea90d1094\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk" Oct 30 14:08:39.783615 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:39.783586 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8fa477f-0e45-42fa-9941-514ea90d1094-istio-envoy\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk\" (UID: \"e8fa477f-0e45-42fa-9941-514ea90d1094\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk" Oct 30 14:08:39.783812 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:39.783650 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8fa477f-0e45-42fa-9941-514ea90d1094-workload-socket\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk\" (UID: \"e8fa477f-0e45-42fa-9941-514ea90d1094\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk" Oct 30 14:08:39.783812 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:39.783704 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6gz4\" (UniqueName: \"kubernetes.io/projected/e8fa477f-0e45-42fa-9941-514ea90d1094-kube-api-access-c6gz4\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk\" (UID: \"e8fa477f-0e45-42fa-9941-514ea90d1094\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk" Oct 30 14:08:39.783812 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:39.783739 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8fa477f-0e45-42fa-9941-514ea90d1094-istiod-ca-cert\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk\" (UID: \"e8fa477f-0e45-42fa-9941-514ea90d1094\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk" Oct 30 14:08:39.783812 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:39.783771 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8fa477f-0e45-42fa-9941-514ea90d1094-workload-certs\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk\" (UID: \"e8fa477f-0e45-42fa-9941-514ea90d1094\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk" Oct 30 14:08:39.784005 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:39.783849 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8fa477f-0e45-42fa-9941-514ea90d1094-istio-data\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk\" (UID: \"e8fa477f-0e45-42fa-9941-514ea90d1094\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk" Oct 30 14:08:39.784005 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:39.783936 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8fa477f-0e45-42fa-9941-514ea90d1094-istio-podinfo\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk\" (UID: \"e8fa477f-0e45-42fa-9941-514ea90d1094\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk" Oct 30 14:08:39.784554 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:39.784454 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8fa477f-0e45-42fa-9941-514ea90d1094-istiod-ca-cert\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk\" (UID: \"e8fa477f-0e45-42fa-9941-514ea90d1094\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk" Oct 30 14:08:39.787589 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:39.785629 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8fa477f-0e45-42fa-9941-514ea90d1094-credential-socket\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk\" (UID: \"e8fa477f-0e45-42fa-9941-514ea90d1094\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk" Oct 30 14:08:39.787589 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:39.785849 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8fa477f-0e45-42fa-9941-514ea90d1094-workload-certs\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk\" (UID: \"e8fa477f-0e45-42fa-9941-514ea90d1094\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk" Oct 30 14:08:39.787589 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:39.786036 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8fa477f-0e45-42fa-9941-514ea90d1094-workload-socket\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk\" (UID: \"e8fa477f-0e45-42fa-9941-514ea90d1094\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk" Oct 30 14:08:39.787589 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:39.786925 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8fa477f-0e45-42fa-9941-514ea90d1094-istio-envoy\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk\" (UID: \"e8fa477f-0e45-42fa-9941-514ea90d1094\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk" Oct 30 14:08:39.802795 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:39.802772 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8fa477f-0e45-42fa-9941-514ea90d1094-istio-podinfo\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk\" (UID: \"e8fa477f-0e45-42fa-9941-514ea90d1094\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk" Oct 30 14:08:39.809795 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:39.809772 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8fa477f-0e45-42fa-9941-514ea90d1094-istio-token\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk\" (UID: \"e8fa477f-0e45-42fa-9941-514ea90d1094\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk" Oct 30 14:08:39.810363 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:39.810342 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6gz4\" (UniqueName: \"kubernetes.io/projected/e8fa477f-0e45-42fa-9941-514ea90d1094-kube-api-access-c6gz4\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk\" (UID: \"e8fa477f-0e45-42fa-9941-514ea90d1094\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk" Oct 30 14:08:39.885062 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:39.884810 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk" Oct 30 14:08:40.211363 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:40.211288 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk"] Oct 30 14:08:40.215417 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:08:40.215379 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8fa477f_0e45_42fa_9941_514ea90d1094.slice/crio-385b43eb2169b20ea3142273f63625efa674b2b904eb69ea3e077192b12d055e WatchSource:0}: Error finding container 385b43eb2169b20ea3142273f63625efa674b2b904eb69ea3e077192b12d055e: Status 404 returned error can't find the container with id 385b43eb2169b20ea3142273f63625efa674b2b904eb69ea3e077192b12d055e Oct 30 14:08:40.218078 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:40.218039 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:08:40.218181 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:40.218122 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:08:40.218181 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:40.218167 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:08:40.300766 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:40.300738 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk" event={"ID":"e8fa477f-0e45-42fa-9941-514ea90d1094","Type":"ContainerStarted","Data":"5ccc3fa1dfa138b1c81421115baf983c445eca622ac209bdd53cd57bbf935daa"} Oct 30 14:08:40.300866 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:40.300798 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk" event={"ID":"e8fa477f-0e45-42fa-9941-514ea90d1094","Type":"ContainerStarted","Data":"385b43eb2169b20ea3142273f63625efa674b2b904eb69ea3e077192b12d055e"} Oct 30 14:08:40.351234 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:40.351164 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk" podStartSLOduration=1.3511450950000001 podStartE2EDuration="1.351145095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:08:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:08:40.348355337 +0000 UTC m=+1190.092489390" watchObservedRunningTime="2026-10-30 14:08:40.351145095 +0000 UTC m=+1190.095279143" Oct 30 14:08:40.786105 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:40.786065 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:40.886769 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:40.885728 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk" Oct 30 14:08:40.960694 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:40.960659 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:41.906816 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:41.906779 4982 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 14:08:41.907442 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:41.907399 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk" podUID="e8fa477f-0e45-42fa-9941-514ea90d1094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 14:08:42.330050 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:42.330015 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:42.898665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:42.890989 4982 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 14:08:42.898665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:42.891047 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk" podUID="e8fa477f-0e45-42fa-9941-514ea90d1094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 14:08:42.949358 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:42.949318 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:43.888664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:43.886938 4982 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 14:08:43.888664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:43.886996 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk" podUID="e8fa477f-0e45-42fa-9941-514ea90d1094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 14:08:44.761007 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:44.760969 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:44.895660 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:44.888044 4982 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 14:08:44.895660 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:44.888107 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk" podUID="e8fa477f-0e45-42fa-9941-514ea90d1094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 14:08:45.886660 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:45.885994 4982 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 14:08:45.886660 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:45.886062 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk" podUID="e8fa477f-0e45-42fa-9941-514ea90d1094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 14:08:46.546262 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:46.546227 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:46.885822 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:46.885784 4982 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 14:08:46.885997 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:46.885856 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk" podUID="e8fa477f-0e45-42fa-9941-514ea90d1094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 14:08:48.096774 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:48.096742 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk" Oct 30 14:08:48.098682 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:48.098221 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk" Oct 30 14:08:48.106278 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:48.106253 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk" Oct 30 14:08:48.544020 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:48.543942 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:50.346116 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:50.346079 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:50.537280 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:50.537247 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ntlqq_dc967875-1546-40bb-8376-fde09416c204/console-operator/2.log" Oct 30 14:08:50.539210 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:50.539189 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ntlqq_dc967875-1546-40bb-8376-fde09416c204/console-operator/2.log" Oct 30 14:08:50.542226 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:50.542206 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhxjf_aca47fe8-7068-4743-b6ab-283ee0874f66/ovn-acl-logging/0.log" Oct 30 14:08:50.543915 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:50.543891 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhxjf_aca47fe8-7068-4743-b6ab-283ee0874f66/ovn-acl-logging/0.log" Oct 30 14:08:52.176109 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:52.176077 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:54.023335 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:54.023285 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:54.944493 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:54.944446 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk"] Oct 30 14:08:54.949559 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:54.949533 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk" Oct 30 14:08:54.981491 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:54.981456 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk"] Oct 30 14:08:55.089842 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:55.089803 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99cbf845-7193-4e4d-8315-b5b0614cfd33-istio-envoy\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk\" (UID: \"99cbf845-7193-4e4d-8315-b5b0614cfd33\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk" Oct 30 14:08:55.090813 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:55.090781 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99cbf845-7193-4e4d-8315-b5b0614cfd33-credential-socket\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk\" (UID: \"99cbf845-7193-4e4d-8315-b5b0614cfd33\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk" Oct 30 14:08:55.091090 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:55.091070 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99cbf845-7193-4e4d-8315-b5b0614cfd33-workload-socket\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk\" (UID: \"99cbf845-7193-4e4d-8315-b5b0614cfd33\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk" Oct 30 14:08:55.091257 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:55.091236 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99cbf845-7193-4e4d-8315-b5b0614cfd33-istiod-ca-cert\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk\" (UID: \"99cbf845-7193-4e4d-8315-b5b0614cfd33\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk" Oct 30 14:08:55.091448 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:55.091429 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99cbf845-7193-4e4d-8315-b5b0614cfd33-istio-data\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk\" (UID: \"99cbf845-7193-4e4d-8315-b5b0614cfd33\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk" Oct 30 14:08:55.091622 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:55.091606 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99cbf845-7193-4e4d-8315-b5b0614cfd33-istio-podinfo\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk\" (UID: \"99cbf845-7193-4e4d-8315-b5b0614cfd33\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk" Oct 30 14:08:55.091835 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:55.091817 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99cbf845-7193-4e4d-8315-b5b0614cfd33-istio-token\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk\" (UID: \"99cbf845-7193-4e4d-8315-b5b0614cfd33\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk" Oct 30 14:08:55.092042 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:55.092023 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htlnw\" (UniqueName: \"kubernetes.io/projected/99cbf845-7193-4e4d-8315-b5b0614cfd33-kube-api-access-htlnw\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk\" (UID: \"99cbf845-7193-4e4d-8315-b5b0614cfd33\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk" Oct 30 14:08:55.092226 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:55.092208 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99cbf845-7193-4e4d-8315-b5b0614cfd33-workload-certs\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk\" (UID: \"99cbf845-7193-4e4d-8315-b5b0614cfd33\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk" Oct 30 14:08:55.193682 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:55.193607 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99cbf845-7193-4e4d-8315-b5b0614cfd33-workload-socket\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk\" (UID: \"99cbf845-7193-4e4d-8315-b5b0614cfd33\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk" Oct 30 14:08:55.194302 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:55.194280 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99cbf845-7193-4e4d-8315-b5b0614cfd33-istiod-ca-cert\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk\" (UID: \"99cbf845-7193-4e4d-8315-b5b0614cfd33\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk" Oct 30 14:08:55.195303 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:55.195233 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99cbf845-7193-4e4d-8315-b5b0614cfd33-istio-data\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk\" (UID: \"99cbf845-7193-4e4d-8315-b5b0614cfd33\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk" Oct 30 14:08:55.195303 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:55.195289 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99cbf845-7193-4e4d-8315-b5b0614cfd33-istio-podinfo\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk\" (UID: \"99cbf845-7193-4e4d-8315-b5b0614cfd33\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk" Oct 30 14:08:55.195477 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:55.195345 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99cbf845-7193-4e4d-8315-b5b0614cfd33-istio-token\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk\" (UID: \"99cbf845-7193-4e4d-8315-b5b0614cfd33\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk" Oct 30 14:08:55.195477 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:55.195425 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htlnw\" (UniqueName: \"kubernetes.io/projected/99cbf845-7193-4e4d-8315-b5b0614cfd33-kube-api-access-htlnw\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk\" (UID: \"99cbf845-7193-4e4d-8315-b5b0614cfd33\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk" Oct 30 14:08:55.195585 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:55.195480 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99cbf845-7193-4e4d-8315-b5b0614cfd33-workload-certs\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk\" (UID: \"99cbf845-7193-4e4d-8315-b5b0614cfd33\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk" Oct 30 14:08:55.195585 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:55.195572 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99cbf845-7193-4e4d-8315-b5b0614cfd33-istio-envoy\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk\" (UID: \"99cbf845-7193-4e4d-8315-b5b0614cfd33\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk" Oct 30 14:08:55.195719 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:55.195625 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99cbf845-7193-4e4d-8315-b5b0614cfd33-credential-socket\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk\" (UID: \"99cbf845-7193-4e4d-8315-b5b0614cfd33\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk" Oct 30 14:08:55.196766 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:55.195160 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99cbf845-7193-4e4d-8315-b5b0614cfd33-istiod-ca-cert\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk\" (UID: \"99cbf845-7193-4e4d-8315-b5b0614cfd33\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk" Oct 30 14:08:55.196766 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:55.194222 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99cbf845-7193-4e4d-8315-b5b0614cfd33-workload-socket\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk\" (UID: \"99cbf845-7193-4e4d-8315-b5b0614cfd33\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk" Oct 30 14:08:55.196766 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:55.196531 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99cbf845-7193-4e4d-8315-b5b0614cfd33-workload-certs\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk\" (UID: \"99cbf845-7193-4e4d-8315-b5b0614cfd33\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk" Oct 30 14:08:55.197028 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:55.196967 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99cbf845-7193-4e4d-8315-b5b0614cfd33-credential-socket\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk\" (UID: \"99cbf845-7193-4e4d-8315-b5b0614cfd33\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk" Oct 30 14:08:55.197289 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:55.197260 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99cbf845-7193-4e4d-8315-b5b0614cfd33-istio-data\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk\" (UID: \"99cbf845-7193-4e4d-8315-b5b0614cfd33\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk" Oct 30 14:08:55.205941 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:55.205894 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99cbf845-7193-4e4d-8315-b5b0614cfd33-istio-envoy\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk\" (UID: \"99cbf845-7193-4e4d-8315-b5b0614cfd33\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk" Oct 30 14:08:55.207414 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:55.207390 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99cbf845-7193-4e4d-8315-b5b0614cfd33-istio-podinfo\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk\" (UID: \"99cbf845-7193-4e4d-8315-b5b0614cfd33\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk" Oct 30 14:08:55.228950 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:55.228903 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99cbf845-7193-4e4d-8315-b5b0614cfd33-istio-token\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk\" (UID: \"99cbf845-7193-4e4d-8315-b5b0614cfd33\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk" Oct 30 14:08:55.230674 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:55.230596 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htlnw\" (UniqueName: \"kubernetes.io/projected/99cbf845-7193-4e4d-8315-b5b0614cfd33-kube-api-access-htlnw\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk\" (UID: \"99cbf845-7193-4e4d-8315-b5b0614cfd33\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk" Oct 30 14:08:55.267977 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:55.267952 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk" Oct 30 14:08:55.725282 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:55.725248 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk"] Oct 30 14:08:55.739136 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:55.738974 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:08:55.739136 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:55.739098 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:08:55.739323 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:55.739140 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:08:56.209665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:56.207218 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:56.455848 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:56.455806 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk" event={"ID":"99cbf845-7193-4e4d-8315-b5b0614cfd33","Type":"ContainerStarted","Data":"c2e6c1f8067dbeaa782c8a4627a999a6a47e555d4161a0f79cd9ee848e98e031"} Oct 30 14:08:56.455990 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:56.455869 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk" event={"ID":"99cbf845-7193-4e4d-8315-b5b0614cfd33","Type":"ContainerStarted","Data":"f4ca797ff3c768528fc0a7a9163bc7b42968753f6b6258312dfa376e7093693e"} Oct 30 14:08:56.521712 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:56.520863 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk" podStartSLOduration=2.520841464 podStartE2EDuration="2.520841464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:08:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:08:56.517965197 +0000 UTC m=+1206.262099244" watchObservedRunningTime="2026-10-30 14:08:56.520841464 +0000 UTC m=+1206.264975508" Oct 30 14:08:57.269083 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:57.269041 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk" Oct 30 14:08:57.270631 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:57.270602 4982 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 14:08:57.270779 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:57.270670 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk" podUID="99cbf845-7193-4e4d-8315-b5b0614cfd33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 14:08:58.033665 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:08:58.033607 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:08:58.089557 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:58.089521 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:08:58.268532 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:58.268431 4982 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 14:08:58.268762 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:58.268557 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk" podUID="99cbf845-7193-4e4d-8315-b5b0614cfd33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 14:08:59.269192 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:59.269152 4982 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 14:08:59.269591 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:08:59.269219 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk" podUID="99cbf845-7193-4e4d-8315-b5b0614cfd33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 14:09:00.006177 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:00.006134 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:00.269177 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:00.269095 4982 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 14:09:00.269177 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:00.269149 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk" podUID="99cbf845-7193-4e4d-8315-b5b0614cfd33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 14:09:01.268488 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:01.268450 4982 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 14:09:01.268685 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:01.268508 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk" podUID="99cbf845-7193-4e4d-8315-b5b0614cfd33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 14:09:02.081799 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:02.081747 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:02.269142 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:02.269110 4982 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 14:09:02.269312 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:02.269161 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk" podUID="99cbf845-7193-4e4d-8315-b5b0614cfd33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 14:09:03.273176 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:03.273096 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk" Oct 30 14:09:03.273576 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:03.273563 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk" Oct 30 14:09:03.274193 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:03.274175 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk" Oct 30 14:09:03.911033 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:03.910996 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:05.775189 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:05.775133 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:07.648486 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:07.648443 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" podUID="cc3a3a28-d723-459a-93d0-ff115eac479a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:09:07.699754 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:07.699711 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" podUID="ab822fe6-4349-4051-b2fd-c6e5ceb50d08" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded" Oct 30 14:09:07.729106 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:07.729062 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" podUID="15e7b7f9-2570-410a-a398-4c07c2e465ea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:09:08.119322 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:08.119279 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:10.056166 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:10.056120 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:11.889178 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:11.889145 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:13.254984 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:13.254945 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf"] Oct 30 14:09:13.261881 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:13.261853 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" Oct 30 14:09:13.273421 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:13.273381 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf"] Oct 30 14:09:13.282912 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:13.282887 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-da78c4\"/\"distinct-2-w2-da78c4-kserve-self-signed-certs\"" Oct 30 14:09:13.294680 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:13.294650 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00fdf663-9bc9-4349-a565-6a8dc01c0842-model-cache\") pod \"distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf\" (UID: \"00fdf663-9bc9-4349-a565-6a8dc01c0842\") " pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" Oct 30 14:09:13.294853 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:13.294796 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs6km\" (UniqueName: \"kubernetes.io/projected/00fdf663-9bc9-4349-a565-6a8dc01c0842-kube-api-access-bs6km\") pod \"distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf\" (UID: \"00fdf663-9bc9-4349-a565-6a8dc01c0842\") " pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" Oct 30 14:09:13.294853 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:13.294828 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00fdf663-9bc9-4349-a565-6a8dc01c0842-tls-certs\") pod \"distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf\" (UID: \"00fdf663-9bc9-4349-a565-6a8dc01c0842\") " pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" Oct 30 14:09:13.294853 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:13.294849 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00fdf663-9bc9-4349-a565-6a8dc01c0842-home\") pod \"distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf\" (UID: \"00fdf663-9bc9-4349-a565-6a8dc01c0842\") " pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" Oct 30 14:09:13.295185 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:13.294869 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00fdf663-9bc9-4349-a565-6a8dc01c0842-tmp-dir\") pod \"distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf\" (UID: \"00fdf663-9bc9-4349-a565-6a8dc01c0842\") " pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" Oct 30 14:09:13.295185 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:13.294912 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00fdf663-9bc9-4349-a565-6a8dc01c0842-dshm\") pod \"distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf\" (UID: \"00fdf663-9bc9-4349-a565-6a8dc01c0842\") " pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" Oct 30 14:09:13.331540 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:13.331508 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:13.396588 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:13.396558 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs6km\" (UniqueName: \"kubernetes.io/projected/00fdf663-9bc9-4349-a565-6a8dc01c0842-kube-api-access-bs6km\") pod \"distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf\" (UID: \"00fdf663-9bc9-4349-a565-6a8dc01c0842\") " pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" Oct 30 14:09:13.396762 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:13.396614 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00fdf663-9bc9-4349-a565-6a8dc01c0842-tls-certs\") pod \"distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf\" (UID: \"00fdf663-9bc9-4349-a565-6a8dc01c0842\") " pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" Oct 30 14:09:13.396762 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:13.396664 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00fdf663-9bc9-4349-a565-6a8dc01c0842-home\") pod \"distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf\" (UID: \"00fdf663-9bc9-4349-a565-6a8dc01c0842\") " pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" Oct 30 14:09:13.396762 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:13.396696 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00fdf663-9bc9-4349-a565-6a8dc01c0842-tmp-dir\") pod \"distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf\" (UID: \"00fdf663-9bc9-4349-a565-6a8dc01c0842\") " pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" Oct 30 14:09:13.396920 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:13.396767 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00fdf663-9bc9-4349-a565-6a8dc01c0842-dshm\") pod \"distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf\" (UID: \"00fdf663-9bc9-4349-a565-6a8dc01c0842\") " pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" Oct 30 14:09:13.396920 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:13.396848 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00fdf663-9bc9-4349-a565-6a8dc01c0842-model-cache\") pod \"distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf\" (UID: \"00fdf663-9bc9-4349-a565-6a8dc01c0842\") " pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" Oct 30 14:09:13.397245 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:13.397189 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00fdf663-9bc9-4349-a565-6a8dc01c0842-tmp-dir\") pod \"distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf\" (UID: \"00fdf663-9bc9-4349-a565-6a8dc01c0842\") " pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" Oct 30 14:09:13.397364 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:13.397295 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00fdf663-9bc9-4349-a565-6a8dc01c0842-model-cache\") pod \"distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf\" (UID: \"00fdf663-9bc9-4349-a565-6a8dc01c0842\") " pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" Oct 30 14:09:13.397787 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:13.397768 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00fdf663-9bc9-4349-a565-6a8dc01c0842-home\") pod \"distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf\" (UID: \"00fdf663-9bc9-4349-a565-6a8dc01c0842\") " pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" Oct 30 14:09:13.399829 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:13.399802 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00fdf663-9bc9-4349-a565-6a8dc01c0842-dshm\") pod \"distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf\" (UID: \"00fdf663-9bc9-4349-a565-6a8dc01c0842\") " pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" Oct 30 14:09:13.401424 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:13.401242 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00fdf663-9bc9-4349-a565-6a8dc01c0842-tls-certs\") pod \"distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf\" (UID: \"00fdf663-9bc9-4349-a565-6a8dc01c0842\") " pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" Oct 30 14:09:13.423401 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:13.423357 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs6km\" (UniqueName: \"kubernetes.io/projected/00fdf663-9bc9-4349-a565-6a8dc01c0842-kube-api-access-bs6km\") pod \"distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf\" (UID: \"00fdf663-9bc9-4349-a565-6a8dc01c0842\") " pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" Oct 30 14:09:13.579936 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:13.579901 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" Oct 30 14:09:13.803619 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:13.803577 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:13.910283 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:13.910251 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf"] Oct 30 14:09:13.911855 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:09:13.911811 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod00fdf663_9bc9_4349_a565_6a8dc01c0842.slice/crio-84639024ad5ef879eb97ecd486b3c3eda580e47e54b45acd81d71781efb54a0a WatchSource:0}: Error finding container 84639024ad5ef879eb97ecd486b3c3eda580e47e54b45acd81d71781efb54a0a: Status 404 returned error can't find the container with id 84639024ad5ef879eb97ecd486b3c3eda580e47e54b45acd81d71781efb54a0a Oct 30 14:09:14.635228 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:14.635193 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" event={"ID":"00fdf663-9bc9-4349-a565-6a8dc01c0842","Type":"ContainerStarted","Data":"be10650cdc5bde0f4a4fe456d284e435181637804d8ba8cbd0f1bfdfcad72f79"} Oct 30 14:09:14.635628 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:14.635236 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" event={"ID":"00fdf663-9bc9-4349-a565-6a8dc01c0842","Type":"ContainerStarted","Data":"84639024ad5ef879eb97ecd486b3c3eda580e47e54b45acd81d71781efb54a0a"} Oct 30 14:09:14.681110 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:14.681064 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" podStartSLOduration=1.681052363 podStartE2EDuration="1.681052363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:09:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:09:14.679460426 +0000 UTC m=+1224.423594471" watchObservedRunningTime="2026-10-30 14:09:14.681052363 +0000 UTC m=+1224.425186453" Oct 30 14:09:15.580426 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:15.580394 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" Oct 30 14:09:15.660205 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:15.660177 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:17.857035 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:17.856984 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:19.718393 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:19.718357 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:21.601736 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:21.601683 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:23.377883 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:23.377851 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:23.580468 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:23.580427 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" Oct 30 14:09:23.593514 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:23.593487 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" Oct 30 14:09:23.718149 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:23.718064 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" Oct 30 14:09:25.198458 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:25.198413 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:25.215154 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:25.215118 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc"] Oct 30 14:09:25.220075 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:25.220046 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc" Oct 30 14:09:25.258179 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:25.258070 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc"] Oct 30 14:09:25.264119 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:25.264081 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a3afaea-c040-4642-8632-6b5a0dca807c-credential-socket\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc\" (UID: \"6a3afaea-c040-4642-8632-6b5a0dca807c\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc" Oct 30 14:09:25.264266 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:25.264193 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a3afaea-c040-4642-8632-6b5a0dca807c-istio-token\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc\" (UID: \"6a3afaea-c040-4642-8632-6b5a0dca807c\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc" Oct 30 14:09:25.264325 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:25.264263 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8qth\" (UniqueName: \"kubernetes.io/projected/6a3afaea-c040-4642-8632-6b5a0dca807c-kube-api-access-f8qth\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc\" (UID: \"6a3afaea-c040-4642-8632-6b5a0dca807c\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc" Oct 30 14:09:25.264397 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:25.264374 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a3afaea-c040-4642-8632-6b5a0dca807c-istio-envoy\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc\" (UID: \"6a3afaea-c040-4642-8632-6b5a0dca807c\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc" Oct 30 14:09:25.264481 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:25.264405 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a3afaea-c040-4642-8632-6b5a0dca807c-workload-socket\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc\" (UID: \"6a3afaea-c040-4642-8632-6b5a0dca807c\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc" Oct 30 14:09:25.264481 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:25.264435 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a3afaea-c040-4642-8632-6b5a0dca807c-istio-data\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc\" (UID: \"6a3afaea-c040-4642-8632-6b5a0dca807c\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc" Oct 30 14:09:25.264481 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:25.264457 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a3afaea-c040-4642-8632-6b5a0dca807c-istio-podinfo\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc\" (UID: \"6a3afaea-c040-4642-8632-6b5a0dca807c\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc" Oct 30 14:09:25.264593 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:25.264560 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a3afaea-c040-4642-8632-6b5a0dca807c-istiod-ca-cert\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc\" (UID: \"6a3afaea-c040-4642-8632-6b5a0dca807c\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc" Oct 30 14:09:25.264626 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:25.264596 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a3afaea-c040-4642-8632-6b5a0dca807c-workload-certs\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc\" (UID: \"6a3afaea-c040-4642-8632-6b5a0dca807c\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc" Oct 30 14:09:25.365694 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:25.365653 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a3afaea-c040-4642-8632-6b5a0dca807c-istiod-ca-cert\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc\" (UID: \"6a3afaea-c040-4642-8632-6b5a0dca807c\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc" Oct 30 14:09:25.365855 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:25.365703 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a3afaea-c040-4642-8632-6b5a0dca807c-workload-certs\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc\" (UID: \"6a3afaea-c040-4642-8632-6b5a0dca807c\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc" Oct 30 14:09:25.365855 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:25.365782 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a3afaea-c040-4642-8632-6b5a0dca807c-credential-socket\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc\" (UID: \"6a3afaea-c040-4642-8632-6b5a0dca807c\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc" Oct 30 14:09:25.365855 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:25.365840 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a3afaea-c040-4642-8632-6b5a0dca807c-istio-token\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc\" (UID: \"6a3afaea-c040-4642-8632-6b5a0dca807c\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc" Oct 30 14:09:25.366005 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:25.365861 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8qth\" (UniqueName: \"kubernetes.io/projected/6a3afaea-c040-4642-8632-6b5a0dca807c-kube-api-access-f8qth\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc\" (UID: \"6a3afaea-c040-4642-8632-6b5a0dca807c\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc" Oct 30 14:09:25.366005 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:25.365922 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a3afaea-c040-4642-8632-6b5a0dca807c-istio-envoy\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc\" (UID: \"6a3afaea-c040-4642-8632-6b5a0dca807c\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc" Oct 30 14:09:25.366005 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:25.365940 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a3afaea-c040-4642-8632-6b5a0dca807c-workload-socket\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc\" (UID: \"6a3afaea-c040-4642-8632-6b5a0dca807c\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc" Oct 30 14:09:25.366005 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:25.365967 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a3afaea-c040-4642-8632-6b5a0dca807c-istio-data\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc\" (UID: \"6a3afaea-c040-4642-8632-6b5a0dca807c\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc" Oct 30 14:09:25.366005 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:25.365991 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a3afaea-c040-4642-8632-6b5a0dca807c-istio-podinfo\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc\" (UID: \"6a3afaea-c040-4642-8632-6b5a0dca807c\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc" Oct 30 14:09:25.366288 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:25.366255 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a3afaea-c040-4642-8632-6b5a0dca807c-workload-certs\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc\" (UID: \"6a3afaea-c040-4642-8632-6b5a0dca807c\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc" Oct 30 14:09:25.366359 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:25.366325 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a3afaea-c040-4642-8632-6b5a0dca807c-workload-socket\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc\" (UID: \"6a3afaea-c040-4642-8632-6b5a0dca807c\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc" Oct 30 14:09:25.366478 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:25.366452 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a3afaea-c040-4642-8632-6b5a0dca807c-istio-data\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc\" (UID: \"6a3afaea-c040-4642-8632-6b5a0dca807c\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc" Oct 30 14:09:25.366587 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:25.366486 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a3afaea-c040-4642-8632-6b5a0dca807c-credential-socket\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc\" (UID: \"6a3afaea-c040-4642-8632-6b5a0dca807c\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc" Oct 30 14:09:25.366587 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:25.366492 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a3afaea-c040-4642-8632-6b5a0dca807c-istiod-ca-cert\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc\" (UID: \"6a3afaea-c040-4642-8632-6b5a0dca807c\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc" Oct 30 14:09:25.368224 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:25.368199 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a3afaea-c040-4642-8632-6b5a0dca807c-istio-envoy\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc\" (UID: \"6a3afaea-c040-4642-8632-6b5a0dca807c\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc" Oct 30 14:09:25.368609 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:25.368586 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a3afaea-c040-4642-8632-6b5a0dca807c-istio-podinfo\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc\" (UID: \"6a3afaea-c040-4642-8632-6b5a0dca807c\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc" Oct 30 14:09:25.391617 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:25.391590 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a3afaea-c040-4642-8632-6b5a0dca807c-istio-token\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc\" (UID: \"6a3afaea-c040-4642-8632-6b5a0dca807c\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc" Oct 30 14:09:25.392010 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:25.391986 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8qth\" (UniqueName: \"kubernetes.io/projected/6a3afaea-c040-4642-8632-6b5a0dca807c-kube-api-access-f8qth\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc\" (UID: \"6a3afaea-c040-4642-8632-6b5a0dca807c\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc" Oct 30 14:09:25.557709 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:25.557687 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc" Oct 30 14:09:25.884854 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:25.884823 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc"] Oct 30 14:09:25.885958 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:09:25.885929 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a3afaea_c040_4642_8632_6b5a0dca807c.slice/crio-63981d4c4da5f27462823e1737f5e998e243f3c1c2217f902ac61e01082f75c0 WatchSource:0}: Error finding container 63981d4c4da5f27462823e1737f5e998e243f3c1c2217f902ac61e01082f75c0: Status 404 returned error can't find the container with id 63981d4c4da5f27462823e1737f5e998e243f3c1c2217f902ac61e01082f75c0 Oct 30 14:09:25.889693 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:25.889631 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:09:25.889814 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:25.889795 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:09:25.889899 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:25.889877 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:09:26.725056 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:26.725022 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc" event={"ID":"6a3afaea-c040-4642-8632-6b5a0dca807c","Type":"ContainerStarted","Data":"8b46911b829d0492fd3e4bdf74b8302dfd5918d5a4ba8a5dacadd15aa77c02a1"} Oct 30 14:09:26.725496 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:26.725077 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc" event={"ID":"6a3afaea-c040-4642-8632-6b5a0dca807c","Type":"ContainerStarted","Data":"63981d4c4da5f27462823e1737f5e998e243f3c1c2217f902ac61e01082f75c0"} Oct 30 14:09:27.415078 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:27.415041 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:27.558702 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:27.558670 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc" Oct 30 14:09:27.560556 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:27.560525 4982 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 14:09:27.560732 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:27.560582 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc" podUID="6a3afaea-c040-4642-8632-6b5a0dca807c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 14:09:28.059976 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:09:28.059940 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:09:28.560663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:28.558125 4982 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 14:09:28.560663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:28.558188 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc" podUID="6a3afaea-c040-4642-8632-6b5a0dca807c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 14:09:29.558611 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:29.558156 4982 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 14:09:29.558611 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:29.558221 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc" podUID="6a3afaea-c040-4642-8632-6b5a0dca807c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 14:09:29.955008 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:29.954916 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:30.558873 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:30.558838 4982 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 14:09:30.559318 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:30.558897 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc" podUID="6a3afaea-c040-4642-8632-6b5a0dca807c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 14:09:31.558867 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:31.558827 4982 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 14:09:31.559031 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:31.558896 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc" podUID="6a3afaea-c040-4642-8632-6b5a0dca807c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 14:09:31.837218 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:31.837123 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:32.558702 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:32.558664 4982 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 14:09:32.558873 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:32.558731 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc" podUID="6a3afaea-c040-4642-8632-6b5a0dca807c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 14:09:33.833791 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:33.833750 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:34.599816 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:34.599780 4982 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:09:34.600126 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:34.600091 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc" podUID="6a3afaea-c040-4642-8632-6b5a0dca807c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:09:34.759851 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:34.759808 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc" Oct 30 14:09:34.760696 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:34.760671 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc" Oct 30 14:09:34.762128 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:34.762106 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc" Oct 30 14:09:34.822713 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:34.822663 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc" podStartSLOduration=9.8226337 podStartE2EDuration="9.8226337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:09:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:09:26.80307174 +0000 UTC m=+1236.547205784" watchObservedRunningTime="2026-10-30 14:09:34.8226337 +0000 UTC m=+1244.566767743" Oct 30 14:09:35.632101 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:35.632061 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:37.532796 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:37.532764 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:37.675938 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:37.675899 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:39.434147 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:39.433476 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:41.613606 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:41.613547 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:42.445831 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:42.445797 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:43.526089 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:43.526048 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:43.657601 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:43.657559 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-00926c/distinct-w3-00926c-kserve-774df5b58-cv8vt"] Oct 30 14:09:43.662864 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:43.662840 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-00926c/distinct-w3-00926c-kserve-774df5b58-cv8vt" Oct 30 14:09:43.682177 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:43.682152 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-00926c/distinct-w3-00926c-kserve-774df5b58-cv8vt"] Oct 30 14:09:43.691818 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:43.691792 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-00926c\"/\"distinct-w3-00926c-kserve-self-signed-certs\"" Oct 30 14:09:43.734728 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:43.734695 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:43.770070 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:43.770044 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec-dshm\") pod \"distinct-w3-00926c-kserve-774df5b58-cv8vt\" (UID: \"b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec\") " pod="e2e-models-e2e-worker-w3-00926c/distinct-w3-00926c-kserve-774df5b58-cv8vt" Oct 30 14:09:43.770215 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:43.770074 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec-tls-certs\") pod \"distinct-w3-00926c-kserve-774df5b58-cv8vt\" (UID: \"b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec\") " pod="e2e-models-e2e-worker-w3-00926c/distinct-w3-00926c-kserve-774df5b58-cv8vt" Oct 30 14:09:43.770215 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:43.770096 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec-home\") pod \"distinct-w3-00926c-kserve-774df5b58-cv8vt\" (UID: \"b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec\") " pod="e2e-models-e2e-worker-w3-00926c/distinct-w3-00926c-kserve-774df5b58-cv8vt" Oct 30 14:09:43.770215 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:43.770126 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlc8w\" (UniqueName: \"kubernetes.io/projected/b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec-kube-api-access-mlc8w\") pod \"distinct-w3-00926c-kserve-774df5b58-cv8vt\" (UID: \"b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec\") " pod="e2e-models-e2e-worker-w3-00926c/distinct-w3-00926c-kserve-774df5b58-cv8vt" Oct 30 14:09:43.770373 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:43.770343 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec-model-cache\") pod \"distinct-w3-00926c-kserve-774df5b58-cv8vt\" (UID: \"b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec\") " pod="e2e-models-e2e-worker-w3-00926c/distinct-w3-00926c-kserve-774df5b58-cv8vt" Oct 30 14:09:43.770543 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:43.770526 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec-tmp-dir\") pod \"distinct-w3-00926c-kserve-774df5b58-cv8vt\" (UID: \"b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec\") " pod="e2e-models-e2e-worker-w3-00926c/distinct-w3-00926c-kserve-774df5b58-cv8vt" Oct 30 14:09:43.872601 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:43.872546 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec-dshm\") pod \"distinct-w3-00926c-kserve-774df5b58-cv8vt\" (UID: \"b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec\") " pod="e2e-models-e2e-worker-w3-00926c/distinct-w3-00926c-kserve-774df5b58-cv8vt" Oct 30 14:09:43.872601 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:43.872603 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec-tls-certs\") pod \"distinct-w3-00926c-kserve-774df5b58-cv8vt\" (UID: \"b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec\") " pod="e2e-models-e2e-worker-w3-00926c/distinct-w3-00926c-kserve-774df5b58-cv8vt" Oct 30 14:09:43.872895 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:43.872631 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec-home\") pod \"distinct-w3-00926c-kserve-774df5b58-cv8vt\" (UID: \"b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec\") " pod="e2e-models-e2e-worker-w3-00926c/distinct-w3-00926c-kserve-774df5b58-cv8vt" Oct 30 14:09:43.872895 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:43.872698 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlc8w\" (UniqueName: \"kubernetes.io/projected/b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec-kube-api-access-mlc8w\") pod \"distinct-w3-00926c-kserve-774df5b58-cv8vt\" (UID: \"b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec\") " pod="e2e-models-e2e-worker-w3-00926c/distinct-w3-00926c-kserve-774df5b58-cv8vt" Oct 30 14:09:43.872895 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:43.872803 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec-model-cache\") pod \"distinct-w3-00926c-kserve-774df5b58-cv8vt\" (UID: \"b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec\") " pod="e2e-models-e2e-worker-w3-00926c/distinct-w3-00926c-kserve-774df5b58-cv8vt" Oct 30 14:09:43.873062 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:43.872946 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec-tmp-dir\") pod \"distinct-w3-00926c-kserve-774df5b58-cv8vt\" (UID: \"b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec\") " pod="e2e-models-e2e-worker-w3-00926c/distinct-w3-00926c-kserve-774df5b58-cv8vt" Oct 30 14:09:43.873443 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:43.873413 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec-tmp-dir\") pod \"distinct-w3-00926c-kserve-774df5b58-cv8vt\" (UID: \"b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec\") " pod="e2e-models-e2e-worker-w3-00926c/distinct-w3-00926c-kserve-774df5b58-cv8vt" Oct 30 14:09:43.875370 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:43.875347 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec-model-cache\") pod \"distinct-w3-00926c-kserve-774df5b58-cv8vt\" (UID: \"b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec\") " pod="e2e-models-e2e-worker-w3-00926c/distinct-w3-00926c-kserve-774df5b58-cv8vt" Oct 30 14:09:43.875759 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:43.875706 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec-home\") pod \"distinct-w3-00926c-kserve-774df5b58-cv8vt\" (UID: \"b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec\") " pod="e2e-models-e2e-worker-w3-00926c/distinct-w3-00926c-kserve-774df5b58-cv8vt" Oct 30 14:09:43.896673 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:43.896627 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec-dshm\") pod \"distinct-w3-00926c-kserve-774df5b58-cv8vt\" (UID: \"b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec\") " pod="e2e-models-e2e-worker-w3-00926c/distinct-w3-00926c-kserve-774df5b58-cv8vt" Oct 30 14:09:43.903790 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:43.903754 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec-tls-certs\") pod \"distinct-w3-00926c-kserve-774df5b58-cv8vt\" (UID: \"b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec\") " pod="e2e-models-e2e-worker-w3-00926c/distinct-w3-00926c-kserve-774df5b58-cv8vt" Oct 30 14:09:43.907156 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:43.907128 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlc8w\" (UniqueName: \"kubernetes.io/projected/b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec-kube-api-access-mlc8w\") pod \"distinct-w3-00926c-kserve-774df5b58-cv8vt\" (UID: \"b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec\") " pod="e2e-models-e2e-worker-w3-00926c/distinct-w3-00926c-kserve-774df5b58-cv8vt" Oct 30 14:09:43.977979 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:43.977946 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-00926c/distinct-w3-00926c-kserve-774df5b58-cv8vt" Oct 30 14:09:44.341595 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:44.341560 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-00926c/distinct-w3-00926c-kserve-774df5b58-cv8vt"] Oct 30 14:09:45.198473 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:45.198439 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-00926c/distinct-w3-00926c-kserve-774df5b58-cv8vt" event={"ID":"b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec","Type":"ContainerStarted","Data":"3c238d8bd8a85494179991ea3c8193ed2423c27b8d0f7690b704e77f3a692a97"} Oct 30 14:09:45.198980 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:45.198481 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-00926c/distinct-w3-00926c-kserve-774df5b58-cv8vt" event={"ID":"b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec","Type":"ContainerStarted","Data":"9733c7995f6c37d9e6ab96af4488a2d03212585f36e5e431bc422634ef2c7c95"} Oct 30 14:09:45.243544 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:45.243494 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-00926c/distinct-w3-00926c-kserve-774df5b58-cv8vt" podStartSLOduration=2.243481252 podStartE2EDuration="2.243481252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:09:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:09:45.242979087 +0000 UTC m=+1254.987113130" watchObservedRunningTime="2026-10-30 14:09:45.243481252 +0000 UTC m=+1254.987615293" Oct 30 14:09:45.295630 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:45.295595 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:45.979069 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:45.979038 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-00926c/distinct-w3-00926c-kserve-774df5b58-cv8vt" Oct 30 14:09:47.089089 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:47.089053 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:49.129707 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:49.129635 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:50.892742 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:50.892706 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:52.898086 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:52.898048 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:53.979673 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:53.979429 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-00926c/distinct-w3-00926c-kserve-774df5b58-cv8vt" Oct 30 14:09:54.576874 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:54.576806 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" podUID="d65dee9b-4f8b-4f09-b00b-0793d732d3ab" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:09:54.650039 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:54.649979 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" podUID="cc3a3a28-d723-459a-93d0-ff115eac479a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:09:54.728664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:54.715878 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" podUID="ab822fe6-4349-4051-b2fd-c6e5ceb50d08" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:09:54.778665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:54.762322 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" podUID="15e7b7f9-2570-410a-a398-4c07c2e465ea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded" Oct 30 14:09:54.866736 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:54.834204 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8" podUID="f3093abe-9b1e-492a-8837-4bd42ea39d2d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:09:54.965071 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:54.965030 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-00926c/distinct-w3-00926c-kserve-774df5b58-cv8vt" Oct 30 14:09:55.007665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:55.002129 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" podUID="4fa94d86-57ce-4d23-95d5-4efa965f6864" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 14:09:55.062663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:55.061768 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" podUID="946696b9-91bd-437a-9f23-c16b73a5a9a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 14:09:55.584660 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:55.584529 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" podUID="d65dee9b-4f8b-4f09-b00b-0793d732d3ab" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:09:55.609666 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:55.584919 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" Oct 30 14:09:55.651664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:55.640011 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-00926c/distinct-w3-00926c-kserve-774df5b58-cv8vt" Oct 30 14:09:55.661663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:55.652803 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" podUID="cc3a3a28-d723-459a-93d0-ff115eac479a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:09:55.661663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:55.653436 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" Oct 30 14:09:55.753726 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:55.741836 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" podUID="00fdf663-9bc9-4349-a565-6a8dc01c0842" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 14:09:55.775661 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:55.772037 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" podUID="15e7b7f9-2570-410a-a398-4c07c2e465ea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:09:55.785099 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:55.785074 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" Oct 30 14:09:55.877748 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:55.869046 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8" podUID="f3093abe-9b1e-492a-8837-4bd42ea39d2d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:09:55.881671 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:55.878743 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8" Oct 30 14:09:56.170390 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:56.169752 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" podUID="cbd754bd-771d-4007-8ac0-7e413dc24915" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:09:56.250816 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:56.250202 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8" Oct 30 14:09:56.268828 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:56.268632 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:09:56.438297 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:56.438217 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" Oct 30 14:09:56.472909 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:56.472879 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" Oct 30 14:09:56.579666 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:56.574477 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" Oct 30 14:09:58.012516 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:09:58.012482 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:09:58.447043 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:09:58.447013 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:00.202185 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:00.202148 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:00.550833 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:00.550743 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:02.133443 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:02.133408 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:03.973573 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:03.973517 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:05.838171 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:05.838128 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:07.689210 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:07.689154 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:09.612215 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:09.612161 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:09.779338 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:09.779299 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw"] Oct 30 14:10:09.785749 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:09.785704 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw" Oct 30 14:10:09.813134 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:09.813093 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf"] Oct 30 14:10:09.823984 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:09.823948 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf" Oct 30 14:10:09.859695 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:09.858205 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28-istio-podinfo\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw\" (UID: \"2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw" Oct 30 14:10:09.860082 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:09.860048 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28-workload-certs\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw\" (UID: \"2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw" Oct 30 14:10:09.860252 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:09.860228 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28-istio-data\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw\" (UID: \"2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw" Oct 30 14:10:09.860389 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:09.860371 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28-istiod-ca-cert\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw\" (UID: \"2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw" Oct 30 14:10:09.860530 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:09.860515 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28-istio-token\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw\" (UID: \"2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw" Oct 30 14:10:09.860714 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:09.860693 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28-workload-socket\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw\" (UID: \"2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw" Oct 30 14:10:09.860911 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:09.860890 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j94cr\" (UniqueName: \"kubernetes.io/projected/2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28-kube-api-access-j94cr\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw\" (UID: \"2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw" Oct 30 14:10:09.861061 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:09.861043 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28-credential-socket\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw\" (UID: \"2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw" Oct 30 14:10:09.861184 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:09.861166 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28-istio-envoy\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw\" (UID: \"2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw" Oct 30 14:10:09.869160 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:09.869084 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw"] Oct 30 14:10:09.872480 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:09.872449 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf"] Oct 30 14:10:09.962396 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:09.962335 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j94cr\" (UniqueName: \"kubernetes.io/projected/2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28-kube-api-access-j94cr\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw\" (UID: \"2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw" Oct 30 14:10:09.962396 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:09.962401 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b616a96d-9d08-4171-89a5-d84cd2039e4a-istio-podinfo\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf\" (UID: \"b616a96d-9d08-4171-89a5-d84cd2039e4a\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf" Oct 30 14:10:09.962671 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:09.962498 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28-credential-socket\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw\" (UID: \"2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw" Oct 30 14:10:09.962671 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:09.962529 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28-istio-envoy\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw\" (UID: \"2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw" Oct 30 14:10:09.962671 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:09.962559 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b616a96d-9d08-4171-89a5-d84cd2039e4a-istio-data\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf\" (UID: \"b616a96d-9d08-4171-89a5-d84cd2039e4a\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf" Oct 30 14:10:09.962671 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:09.962597 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b616a96d-9d08-4171-89a5-d84cd2039e4a-credential-socket\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf\" (UID: \"b616a96d-9d08-4171-89a5-d84cd2039e4a\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf" Oct 30 14:10:09.962671 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:09.962634 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b616a96d-9d08-4171-89a5-d84cd2039e4a-workload-certs\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf\" (UID: \"b616a96d-9d08-4171-89a5-d84cd2039e4a\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf" Oct 30 14:10:09.962955 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:09.962677 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b616a96d-9d08-4171-89a5-d84cd2039e4a-workload-socket\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf\" (UID: \"b616a96d-9d08-4171-89a5-d84cd2039e4a\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf" Oct 30 14:10:09.962955 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:09.962800 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b616a96d-9d08-4171-89a5-d84cd2039e4a-istiod-ca-cert\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf\" (UID: \"b616a96d-9d08-4171-89a5-d84cd2039e4a\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf" Oct 30 14:10:09.962955 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:09.962862 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b616a96d-9d08-4171-89a5-d84cd2039e4a-istio-token\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf\" (UID: \"b616a96d-9d08-4171-89a5-d84cd2039e4a\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf" Oct 30 14:10:09.962955 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:09.962923 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28-credential-socket\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw\" (UID: \"2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw" Oct 30 14:10:09.963138 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:09.962973 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28-istio-podinfo\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw\" (UID: \"2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw" Oct 30 14:10:09.963138 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:09.963008 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b616a96d-9d08-4171-89a5-d84cd2039e4a-istio-envoy\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf\" (UID: \"b616a96d-9d08-4171-89a5-d84cd2039e4a\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf" Oct 30 14:10:09.963138 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:09.963118 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxl6w\" (UniqueName: \"kubernetes.io/projected/b616a96d-9d08-4171-89a5-d84cd2039e4a-kube-api-access-dxl6w\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf\" (UID: \"b616a96d-9d08-4171-89a5-d84cd2039e4a\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf" Oct 30 14:10:09.963245 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:09.963158 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28-workload-certs\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw\" (UID: \"2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw" Oct 30 14:10:09.963245 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:09.963182 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28-istio-data\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw\" (UID: \"2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw" Oct 30 14:10:09.963245 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:09.963209 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28-istiod-ca-cert\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw\" (UID: \"2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw" Oct 30 14:10:09.963346 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:09.963256 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28-istio-token\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw\" (UID: \"2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw" Oct 30 14:10:09.963346 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:09.963304 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28-workload-socket\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw\" (UID: \"2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw" Oct 30 14:10:09.963765 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:09.963741 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28-istio-data\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw\" (UID: \"2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw" Oct 30 14:10:09.963898 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:09.963792 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28-workload-socket\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw\" (UID: \"2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw" Oct 30 14:10:09.964189 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:09.964164 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28-istiod-ca-cert\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw\" (UID: \"2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw" Oct 30 14:10:09.964346 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:09.964312 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28-workload-certs\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw\" (UID: \"2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw" Oct 30 14:10:09.965146 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:09.965121 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28-istio-envoy\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw\" (UID: \"2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw" Oct 30 14:10:09.965602 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:09.965585 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28-istio-podinfo\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw\" (UID: \"2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw" Oct 30 14:10:09.990616 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:09.990575 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28-istio-token\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw\" (UID: \"2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw" Oct 30 14:10:09.990844 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:09.990820 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j94cr\" (UniqueName: \"kubernetes.io/projected/2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28-kube-api-access-j94cr\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw\" (UID: \"2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw" Oct 30 14:10:10.064444 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:10.064403 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b616a96d-9d08-4171-89a5-d84cd2039e4a-istio-envoy\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf\" (UID: \"b616a96d-9d08-4171-89a5-d84cd2039e4a\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf" Oct 30 14:10:10.064687 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:10.064532 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxl6w\" (UniqueName: \"kubernetes.io/projected/b616a96d-9d08-4171-89a5-d84cd2039e4a-kube-api-access-dxl6w\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf\" (UID: \"b616a96d-9d08-4171-89a5-d84cd2039e4a\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf" Oct 30 14:10:10.064687 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:10.064658 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b616a96d-9d08-4171-89a5-d84cd2039e4a-istio-podinfo\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf\" (UID: \"b616a96d-9d08-4171-89a5-d84cd2039e4a\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf" Oct 30 14:10:10.064834 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:10.064706 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b616a96d-9d08-4171-89a5-d84cd2039e4a-istio-data\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf\" (UID: \"b616a96d-9d08-4171-89a5-d84cd2039e4a\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf" Oct 30 14:10:10.064834 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:10.064742 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b616a96d-9d08-4171-89a5-d84cd2039e4a-credential-socket\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf\" (UID: \"b616a96d-9d08-4171-89a5-d84cd2039e4a\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf" Oct 30 14:10:10.064834 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:10.064775 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b616a96d-9d08-4171-89a5-d84cd2039e4a-workload-certs\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf\" (UID: \"b616a96d-9d08-4171-89a5-d84cd2039e4a\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf" Oct 30 14:10:10.064834 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:10.064806 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b616a96d-9d08-4171-89a5-d84cd2039e4a-workload-socket\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf\" (UID: \"b616a96d-9d08-4171-89a5-d84cd2039e4a\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf" Oct 30 14:10:10.065034 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:10.064854 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b616a96d-9d08-4171-89a5-d84cd2039e4a-istiod-ca-cert\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf\" (UID: \"b616a96d-9d08-4171-89a5-d84cd2039e4a\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf" Oct 30 14:10:10.065034 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:10.064891 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b616a96d-9d08-4171-89a5-d84cd2039e4a-istio-token\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf\" (UID: \"b616a96d-9d08-4171-89a5-d84cd2039e4a\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf" Oct 30 14:10:10.065184 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:10.065149 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b616a96d-9d08-4171-89a5-d84cd2039e4a-istio-data\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf\" (UID: \"b616a96d-9d08-4171-89a5-d84cd2039e4a\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf" Oct 30 14:10:10.065289 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:10.065267 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b616a96d-9d08-4171-89a5-d84cd2039e4a-workload-certs\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf\" (UID: \"b616a96d-9d08-4171-89a5-d84cd2039e4a\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf" Oct 30 14:10:10.065396 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:10.065371 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b616a96d-9d08-4171-89a5-d84cd2039e4a-workload-socket\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf\" (UID: \"b616a96d-9d08-4171-89a5-d84cd2039e4a\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf" Oct 30 14:10:10.065482 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:10.065460 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b616a96d-9d08-4171-89a5-d84cd2039e4a-credential-socket\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf\" (UID: \"b616a96d-9d08-4171-89a5-d84cd2039e4a\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf" Oct 30 14:10:10.066010 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:10.065988 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b616a96d-9d08-4171-89a5-d84cd2039e4a-istiod-ca-cert\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf\" (UID: \"b616a96d-9d08-4171-89a5-d84cd2039e4a\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf" Oct 30 14:10:10.067136 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:10.067115 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b616a96d-9d08-4171-89a5-d84cd2039e4a-istio-envoy\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf\" (UID: \"b616a96d-9d08-4171-89a5-d84cd2039e4a\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf" Oct 30 14:10:10.067275 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:10.067254 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b616a96d-9d08-4171-89a5-d84cd2039e4a-istio-podinfo\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf\" (UID: \"b616a96d-9d08-4171-89a5-d84cd2039e4a\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf" Oct 30 14:10:10.091016 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:10.090928 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b616a96d-9d08-4171-89a5-d84cd2039e4a-istio-token\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf\" (UID: \"b616a96d-9d08-4171-89a5-d84cd2039e4a\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf" Oct 30 14:10:10.091184 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:10.091070 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxl6w\" (UniqueName: \"kubernetes.io/projected/b616a96d-9d08-4171-89a5-d84cd2039e4a-kube-api-access-dxl6w\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf\" (UID: \"b616a96d-9d08-4171-89a5-d84cd2039e4a\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf" Oct 30 14:10:10.165895 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:10.165792 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw" Oct 30 14:10:10.169471 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:10.169437 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf" Oct 30 14:10:10.525975 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:10:10.525937 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f8d2c5b_3fc9_4dbd_9c17_e39101f1cf28.slice/crio-0d1f36a956c23e7b13f7364daf3ada1d7148c12da09c49fedf4f3c614512ff90 WatchSource:0}: Error finding container 0d1f36a956c23e7b13f7364daf3ada1d7148c12da09c49fedf4f3c614512ff90: Status 404 returned error can't find the container with id 0d1f36a956c23e7b13f7364daf3ada1d7148c12da09c49fedf4f3c614512ff90 Oct 30 14:10:10.527211 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:10.527185 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw"] Oct 30 14:10:10.528897 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:10.528862 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:10:10.528966 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:10.528950 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:10:10.529076 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:10.528996 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:10:10.532027 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:10.531999 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf"] Oct 30 14:10:10.532222 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:10:10.532200 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb616a96d_9d08_4171_89a5_d84cd2039e4a.slice/crio-efbd281cefff27db5e04d6462d1e8a0bdd21d30e7e80c25ecc6739f8a4f60bd2 WatchSource:0}: Error finding container efbd281cefff27db5e04d6462d1e8a0bdd21d30e7e80c25ecc6739f8a4f60bd2: Status 404 returned error can't find the container with id efbd281cefff27db5e04d6462d1e8a0bdd21d30e7e80c25ecc6739f8a4f60bd2 Oct 30 14:10:10.540697 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:10.540139 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:10:10.540697 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:10.540242 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:10:10.540697 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:10.540287 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:10:10.542945 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:10.542918 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc"] Oct 30 14:10:10.550178 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:10.549843 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc" Oct 30 14:10:10.577795 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:10.574111 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc"] Oct 30 14:10:10.589788 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:10.589751 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw" event={"ID":"2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28","Type":"ContainerStarted","Data":"0d1f36a956c23e7b13f7364daf3ada1d7148c12da09c49fedf4f3c614512ff90"} Oct 30 14:10:10.591402 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:10.591371 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf" event={"ID":"b616a96d-9d08-4171-89a5-d84cd2039e4a","Type":"ContainerStarted","Data":"efbd281cefff27db5e04d6462d1e8a0bdd21d30e7e80c25ecc6739f8a4f60bd2"} Oct 30 14:10:10.675807 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:10.675766 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed4b142-fae5-4489-818c-8b53e3fea272-workload-socket\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc\" (UID: \"1ed4b142-fae5-4489-818c-8b53e3fea272\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc" Oct 30 14:10:10.676258 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:10.675940 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ed4b142-fae5-4489-818c-8b53e3fea272-istio-podinfo\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc\" (UID: \"1ed4b142-fae5-4489-818c-8b53e3fea272\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc" Oct 30 14:10:10.676258 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:10.676013 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed4b142-fae5-4489-818c-8b53e3fea272-credential-socket\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc\" (UID: \"1ed4b142-fae5-4489-818c-8b53e3fea272\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc" Oct 30 14:10:10.676258 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:10.676120 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw6hf\" (UniqueName: \"kubernetes.io/projected/1ed4b142-fae5-4489-818c-8b53e3fea272-kube-api-access-pw6hf\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc\" (UID: \"1ed4b142-fae5-4489-818c-8b53e3fea272\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc" Oct 30 14:10:10.676258 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:10.676152 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ed4b142-fae5-4489-818c-8b53e3fea272-istiod-ca-cert\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc\" (UID: \"1ed4b142-fae5-4489-818c-8b53e3fea272\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc" Oct 30 14:10:10.676489 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:10.676275 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ed4b142-fae5-4489-818c-8b53e3fea272-workload-certs\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc\" (UID: \"1ed4b142-fae5-4489-818c-8b53e3fea272\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc" Oct 30 14:10:10.676489 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:10.676321 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ed4b142-fae5-4489-818c-8b53e3fea272-istio-data\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc\" (UID: \"1ed4b142-fae5-4489-818c-8b53e3fea272\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc" Oct 30 14:10:10.676489 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:10.676431 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ed4b142-fae5-4489-818c-8b53e3fea272-istio-envoy\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc\" (UID: \"1ed4b142-fae5-4489-818c-8b53e3fea272\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc" Oct 30 14:10:10.676489 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:10.676467 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ed4b142-fae5-4489-818c-8b53e3fea272-istio-token\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc\" (UID: \"1ed4b142-fae5-4489-818c-8b53e3fea272\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc" Oct 30 14:10:10.778482 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:10.778062 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ed4b142-fae5-4489-818c-8b53e3fea272-workload-certs\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc\" (UID: \"1ed4b142-fae5-4489-818c-8b53e3fea272\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc" Oct 30 14:10:10.778482 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:10.778234 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ed4b142-fae5-4489-818c-8b53e3fea272-istio-data\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc\" (UID: \"1ed4b142-fae5-4489-818c-8b53e3fea272\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc" Oct 30 14:10:10.778482 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:10.778313 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ed4b142-fae5-4489-818c-8b53e3fea272-istio-envoy\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc\" (UID: \"1ed4b142-fae5-4489-818c-8b53e3fea272\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc" Oct 30 14:10:10.778482 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:10.778347 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ed4b142-fae5-4489-818c-8b53e3fea272-istio-token\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc\" (UID: \"1ed4b142-fae5-4489-818c-8b53e3fea272\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc" Oct 30 14:10:10.778482 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:10.778474 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed4b142-fae5-4489-818c-8b53e3fea272-workload-socket\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc\" (UID: \"1ed4b142-fae5-4489-818c-8b53e3fea272\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc" Oct 30 14:10:10.778898 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:10.778541 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ed4b142-fae5-4489-818c-8b53e3fea272-istio-podinfo\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc\" (UID: \"1ed4b142-fae5-4489-818c-8b53e3fea272\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc" Oct 30 14:10:10.778898 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:10.778573 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ed4b142-fae5-4489-818c-8b53e3fea272-workload-certs\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc\" (UID: \"1ed4b142-fae5-4489-818c-8b53e3fea272\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc" Oct 30 14:10:10.778898 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:10.778587 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ed4b142-fae5-4489-818c-8b53e3fea272-istio-data\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc\" (UID: \"1ed4b142-fae5-4489-818c-8b53e3fea272\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc" Oct 30 14:10:10.778898 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:10.778595 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed4b142-fae5-4489-818c-8b53e3fea272-credential-socket\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc\" (UID: \"1ed4b142-fae5-4489-818c-8b53e3fea272\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc" Oct 30 14:10:10.778898 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:10.778720 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw6hf\" (UniqueName: \"kubernetes.io/projected/1ed4b142-fae5-4489-818c-8b53e3fea272-kube-api-access-pw6hf\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc\" (UID: \"1ed4b142-fae5-4489-818c-8b53e3fea272\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc" Oct 30 14:10:10.778898 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:10.778753 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ed4b142-fae5-4489-818c-8b53e3fea272-istiod-ca-cert\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc\" (UID: \"1ed4b142-fae5-4489-818c-8b53e3fea272\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc" Oct 30 14:10:10.778898 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:10.778889 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed4b142-fae5-4489-818c-8b53e3fea272-workload-socket\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc\" (UID: \"1ed4b142-fae5-4489-818c-8b53e3fea272\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc" Oct 30 14:10:10.779251 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:10.779232 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed4b142-fae5-4489-818c-8b53e3fea272-credential-socket\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc\" (UID: \"1ed4b142-fae5-4489-818c-8b53e3fea272\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc" Oct 30 14:10:10.779625 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:10.779599 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ed4b142-fae5-4489-818c-8b53e3fea272-istiod-ca-cert\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc\" (UID: \"1ed4b142-fae5-4489-818c-8b53e3fea272\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc" Oct 30 14:10:10.782223 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:10.782188 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ed4b142-fae5-4489-818c-8b53e3fea272-istio-podinfo\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc\" (UID: \"1ed4b142-fae5-4489-818c-8b53e3fea272\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc" Oct 30 14:10:10.782372 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:10.782351 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ed4b142-fae5-4489-818c-8b53e3fea272-istio-envoy\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc\" (UID: \"1ed4b142-fae5-4489-818c-8b53e3fea272\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc" Oct 30 14:10:10.807239 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:10.807201 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ed4b142-fae5-4489-818c-8b53e3fea272-istio-token\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc\" (UID: \"1ed4b142-fae5-4489-818c-8b53e3fea272\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc" Oct 30 14:10:10.807536 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:10.807509 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw6hf\" (UniqueName: \"kubernetes.io/projected/1ed4b142-fae5-4489-818c-8b53e3fea272-kube-api-access-pw6hf\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc\" (UID: \"1ed4b142-fae5-4489-818c-8b53e3fea272\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc" Oct 30 14:10:10.886998 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:10.886953 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc" Oct 30 14:10:11.226710 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:11.226672 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc"] Oct 30 14:10:11.227853 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:10:11.227816 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ed4b142_fae5_4489_818c_8b53e3fea272.slice/crio-9bc8f4cbc4ff6e507bfa49314a66d05b9d46a7034600604662ea9bbb0c84d5bf WatchSource:0}: Error finding container 9bc8f4cbc4ff6e507bfa49314a66d05b9d46a7034600604662ea9bbb0c84d5bf: Status 404 returned error can't find the container with id 9bc8f4cbc4ff6e507bfa49314a66d05b9d46a7034600604662ea9bbb0c84d5bf Oct 30 14:10:11.231132 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:11.231077 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:10:11.231252 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:11.231187 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:10:11.231252 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:11.231241 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:10:11.617668 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:11.617243 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:11.642264 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:11.642211 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw" event={"ID":"2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28","Type":"ContainerStarted","Data":"9e508c66aa266c2278f1dd5abb2387d3e3f4901a55d519df8f3ffceabfdd225b"} Oct 30 14:10:11.647934 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:11.647874 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf" event={"ID":"b616a96d-9d08-4171-89a5-d84cd2039e4a","Type":"ContainerStarted","Data":"733e0862c3083fff16ee42062bb8acc9c5aeed7936283a0f45e45e08f3f49653"} Oct 30 14:10:11.654788 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:11.654702 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc" event={"ID":"1ed4b142-fae5-4489-818c-8b53e3fea272","Type":"ContainerStarted","Data":"7bae0c3d4a401a4cb6b00e460fbb02f8e318ccac576fe114d38cd7f58d21a233"} Oct 30 14:10:11.654788 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:11.654754 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc" event={"ID":"1ed4b142-fae5-4489-818c-8b53e3fea272","Type":"ContainerStarted","Data":"9bc8f4cbc4ff6e507bfa49314a66d05b9d46a7034600604662ea9bbb0c84d5bf"} Oct 30 14:10:11.723667 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:11.721359 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw" podStartSLOduration=2.721337404 podStartE2EDuration="2.721337404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:10:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:10:11.696251417 +0000 UTC m=+1281.440385461" watchObservedRunningTime="2026-10-30 14:10:11.721337404 +0000 UTC m=+1281.465471448" Oct 30 14:10:11.753460 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:11.753404 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc" podStartSLOduration=1.753388288 podStartE2EDuration="1.753388288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:10:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:10:11.749976381 +0000 UTC m=+1281.494110425" watchObservedRunningTime="2026-10-30 14:10:11.753388288 +0000 UTC m=+1281.497522329" Oct 30 14:10:11.831611 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:11.830690 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf" podStartSLOduration=2.83066416 podStartE2EDuration="2.83066416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:10:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:10:11.818546874 +0000 UTC m=+1281.562680919" watchObservedRunningTime="2026-10-30 14:10:11.83066416 +0000 UTC m=+1281.574798203" Oct 30 14:10:11.903731 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:11.887744 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc" Oct 30 14:10:12.166952 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:12.166858 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw" Oct 30 14:10:12.168665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:12.168623 4982 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 14:10:12.168755 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:12.168693 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw" podUID="2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 14:10:12.169793 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:12.169769 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf" Oct 30 14:10:12.171118 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:12.171098 4982 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 14:10:12.171187 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:12.171149 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf" podUID="b616a96d-9d08-4171-89a5-d84cd2039e4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 14:10:12.946671 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:12.930961 4982 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 30 14:10:12.946671 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:12.931034 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc" podUID="1ed4b142-fae5-4489-818c-8b53e3fea272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 30 14:10:13.169026 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:13.168985 4982 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 14:10:13.169306 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:13.169276 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw" podUID="2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 14:10:13.170124 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:13.170039 4982 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 14:10:13.170308 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:13.170286 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf" podUID="b616a96d-9d08-4171-89a5-d84cd2039e4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 14:10:13.896662 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:13.895957 4982 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 30 14:10:13.896662 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:13.896116 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc" podUID="1ed4b142-fae5-4489-818c-8b53e3fea272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 30 14:10:13.933665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:13.932757 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" podUID="4fa94d86-57ce-4d23-95d5-4efa965f6864" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:14.167066 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:14.166976 4982 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 14:10:14.168330 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:14.168286 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw" podUID="2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 14:10:14.178796 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:14.178757 4982 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 14:10:14.178979 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:14.178959 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf" podUID="b616a96d-9d08-4171-89a5-d84cd2039e4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 14:10:14.369675 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:14.368968 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:14.583753 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:14.582753 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" podUID="d65dee9b-4f8b-4f09-b00b-0793d732d3ab" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:14.717663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:14.707222 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" podUID="ab822fe6-4349-4051-b2fd-c6e5ceb50d08" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:14.946721 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:14.921676 4982 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 30 14:10:15.029195 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:15.029144 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc" podUID="1ed4b142-fae5-4489-818c-8b53e3fea272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 30 14:10:15.029425 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:15.021728 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" podUID="946696b9-91bd-437a-9f23-c16b73a5a9a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:15.029810 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:14.961817 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" podUID="4fa94d86-57ce-4d23-95d5-4efa965f6864" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:15.034256 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:15.034227 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" Oct 30 14:10:15.241598 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:15.241203 4982 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 14:10:15.241598 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:15.241269 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf" podUID="b616a96d-9d08-4171-89a5-d84cd2039e4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 14:10:15.241598 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:15.241347 4982 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 14:10:15.241598 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:15.241370 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw" podUID="2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 14:10:15.509581 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:15.503555 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" podUID="15e7b7f9-2570-410a-a398-4c07c2e465ea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:15.509581 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:15.505274 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" podUID="cc3a3a28-d723-459a-93d0-ff115eac479a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded" Oct 30 14:10:15.750785 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:15.739903 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" podUID="ab822fe6-4349-4051-b2fd-c6e5ceb50d08" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:15.750785 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:15.743086 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" Oct 30 14:10:15.966587 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:15.966550 4982 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 30 14:10:16.036555 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:15.966841 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc" podUID="1ed4b142-fae5-4489-818c-8b53e3fea272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 30 14:10:16.037245 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:16.037209 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" podUID="4fa94d86-57ce-4d23-95d5-4efa965f6864" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 14:10:16.079437 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:16.079403 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" Oct 30 14:10:16.224049 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:16.223968 4982 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 14:10:16.309081 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:16.269439 4982 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 14:10:16.309081 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:16.303625 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf" podUID="b616a96d-9d08-4171-89a5-d84cd2039e4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 14:10:16.309674 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:16.302808 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw" podUID="2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 14:10:16.341673 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:16.341339 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" Oct 30 14:10:16.376022 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:16.375746 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8" podUID="f3093abe-9b1e-492a-8837-4bd42ea39d2d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:16.510262 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:16.510141 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" podUID="cc3a3a28-d723-459a-93d0-ff115eac479a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:16.627464 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:16.627425 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" Oct 30 14:10:16.649665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:16.646332 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" podUID="d65dee9b-4f8b-4f09-b00b-0793d732d3ab" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded" Oct 30 14:10:16.665665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:16.656372 4982 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" podUID="d65dee9b-4f8b-4f09-b00b-0793d732d3ab" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:16.922878 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:16.922823 4982 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 30 14:10:16.939671 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:16.931729 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc" podUID="1ed4b142-fae5-4489-818c-8b53e3fea272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 30 14:10:16.963809 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:16.926605 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" podUID="00fdf663-9bc9-4349-a565-6a8dc01c0842" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 14:10:17.160659 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:17.132949 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" podUID="ab822fe6-4349-4051-b2fd-c6e5ceb50d08" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded" Oct 30 14:10:17.227772 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:17.215486 4982 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 14:10:17.227772 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:17.215555 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw" podUID="2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 14:10:17.330875 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:17.330818 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" podUID="cbd754bd-771d-4007-8ac0-7e413dc24915" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:17.376784 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:17.376736 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8" podUID="f3093abe-9b1e-492a-8837-4bd42ea39d2d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:17.412674 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:17.409978 4982 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 14:10:17.412674 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:17.410055 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf" podUID="b616a96d-9d08-4171-89a5-d84cd2039e4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 14:10:17.412674 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:17.410834 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8" Oct 30 14:10:17.539158 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:17.539033 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" podUID="cc3a3a28-d723-459a-93d0-ff115eac479a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded" Oct 30 14:10:17.620939 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:17.620879 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" podUID="15e7b7f9-2570-410a-a398-4c07c2e465ea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:17.664676 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:17.662852 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" podUID="d65dee9b-4f8b-4f09-b00b-0793d732d3ab" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:17.664676 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:17.663221 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" Oct 30 14:10:17.678414 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:17.678365 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:17.974850 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:17.974815 4982 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 30 14:10:18.109662 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:18.029164 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" podUID="00fdf663-9bc9-4349-a565-6a8dc01c0842" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:18.109662 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:18.029693 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" podUID="946696b9-91bd-437a-9f23-c16b73a5a9a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:18.109662 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:18.082489 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" Oct 30 14:10:18.112056 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:18.029732 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/distinct-w3-00926c-kserve-774df5b58-cv8vt" podUID="b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:18.112056 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:18.111789 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" podUID="4fa94d86-57ce-4d23-95d5-4efa965f6864" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:18.192417 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:18.192384 4982 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 14:10:18.192831 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:18.192684 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw" podUID="2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 14:10:18.198953 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:18.198910 4982 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 14:10:18.199171 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:18.199134 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf" podUID="b616a96d-9d08-4171-89a5-d84cd2039e4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 14:10:18.284819 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:18.284722 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" podUID="ab822fe6-4349-4051-b2fd-c6e5ceb50d08" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:18.285390 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:18.285358 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" Oct 30 14:10:18.299667 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:18.285544 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc" podUID="1ed4b142-fae5-4489-818c-8b53e3fea272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 30 14:10:18.353154 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:18.353104 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" podUID="cbd754bd-771d-4007-8ac0-7e413dc24915" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:18.354131 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:18.354105 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" Oct 30 14:10:18.416766 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:18.416716 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8" podUID="f3093abe-9b1e-492a-8837-4bd42ea39d2d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:18.604860 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:18.591165 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" podUID="cc3a3a28-d723-459a-93d0-ff115eac479a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:18.635662 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:18.624776 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" podUID="15e7b7f9-2570-410a-a398-4c07c2e465ea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:18.689667 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:18.665627 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" podUID="d65dee9b-4f8b-4f09-b00b-0793d732d3ab" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:18.751661 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:18.701231 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" Oct 30 14:10:18.814690 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:18.760827 4982 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" podUID="cc3a3a28-d723-459a-93d0-ff115eac479a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": context deadline exceeded" Oct 30 14:10:18.946834 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:18.788184 4982 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:10:18.946834 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:18.941052 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8" podUID="00eefe61-57f5-4d1c-b9a5-8dd54db2922e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:18.946834 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:18.843148 4982 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" podUID="946696b9-91bd-437a-9f23-c16b73a5a9a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:18.946834 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:18.808086 4982 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8" podUID="f3093abe-9b1e-492a-8837-4bd42ea39d2d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:18.946834 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:18.890324 4982 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 30 14:10:18.946834 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:18.941149 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc" podUID="1ed4b142-fae5-4489-818c-8b53e3fea272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 30 14:10:19.188795 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:19.188734 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/distinct-w3-00926c-kserve-774df5b58-cv8vt" podUID="b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 30 14:10:19.189402 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:19.189367 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" podUID="00fdf663-9bc9-4349-a565-6a8dc01c0842" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 14:10:19.203713 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:19.190003 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" podUID="4fa94d86-57ce-4d23-95d5-4efa965f6864" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 14:10:19.430871 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:19.190083 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-00926c/distinct-w3-00926c-kserve-774df5b58-cv8vt" Oct 30 14:10:19.430871 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:19.418576 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" Oct 30 14:10:19.430871 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:19.234785 4982 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 14:10:19.430871 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:19.419173 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf" podUID="b616a96d-9d08-4171-89a5-d84cd2039e4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 14:10:19.430871 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:19.234834 4982 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 14:10:19.430871 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:19.419229 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw" podUID="2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 14:10:19.430871 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:19.357419 4982 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:10:19.430871 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:19.419269 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk" podUID="99cbf845-7193-4e4d-8315-b5b0614cfd33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:19.430871 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:19.301081 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" podUID="ab822fe6-4349-4051-b2fd-c6e5ceb50d08" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:19.430871 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:19.357471 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" podUID="cbd754bd-771d-4007-8ac0-7e413dc24915" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 14:10:19.436207 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:19.434621 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8" podUID="f3093abe-9b1e-492a-8837-4bd42ea39d2d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:19.592771 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:19.592726 4982 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" podUID="ab822fe6-4349-4051-b2fd-c6e5ceb50d08" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": context deadline exceeded" Oct 30 14:10:19.593089 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:19.593059 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" podUID="cc3a3a28-d723-459a-93d0-ff115eac479a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded" Oct 30 14:10:19.597296 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:19.597271 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" Oct 30 14:10:19.647208 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:19.641214 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" podUID="15e7b7f9-2570-410a-a398-4c07c2e465ea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:19.730706 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:19.728730 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" podUID="d65dee9b-4f8b-4f09-b00b-0793d732d3ab" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:19.801665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:19.799007 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" Oct 30 14:10:19.906741 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:19.906515 4982 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 30 14:10:19.906741 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:19.906578 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc" podUID="1ed4b142-fae5-4489-818c-8b53e3fea272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 30 14:10:19.977550 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:19.945965 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" Oct 30 14:10:20.037693 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:20.000258 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" podUID="946696b9-91bd-437a-9f23-c16b73a5a9a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:20.174777 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:20.174590 4982 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 14:10:20.175067 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:20.175024 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf" podUID="b616a96d-9d08-4171-89a5-d84cd2039e4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 14:10:20.177766 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:20.177737 4982 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 14:10:20.177925 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:20.177899 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw" podUID="2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 14:10:20.200693 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:20.189747 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/distinct-w3-00926c-kserve-774df5b58-cv8vt" podUID="b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 30 14:10:20.439467 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:20.439344 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8" podUID="f3093abe-9b1e-492a-8837-4bd42ea39d2d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:20.441397 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:20.441347 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" podUID="cbd754bd-771d-4007-8ac0-7e413dc24915" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 14:10:20.444188 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:20.444147 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" podUID="4fa94d86-57ce-4d23-95d5-4efa965f6864" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:20.461932 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:20.461842 4982 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" podUID="4fa94d86-57ce-4d23-95d5-4efa965f6864" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": context deadline exceeded" Oct 30 14:10:20.632848 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:20.600768 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" podUID="cc3a3a28-d723-459a-93d0-ff115eac479a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded" Oct 30 14:10:20.706124 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:20.706020 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" podUID="15e7b7f9-2570-410a-a398-4c07c2e465ea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:20.912661 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:20.911294 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-00926c/distinct-w3-00926c-kserve-774df5b58-cv8vt" Oct 30 14:10:20.912661 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:20.912094 4982 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 30 14:10:20.912661 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:20.912144 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc" podUID="1ed4b142-fae5-4489-818c-8b53e3fea272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 30 14:10:21.013006 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:21.012906 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" podUID="946696b9-91bd-437a-9f23-c16b73a5a9a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:21.013614 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:21.013588 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" Oct 30 14:10:21.188559 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:21.188523 4982 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 14:10:21.188848 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:21.188815 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf" podUID="b616a96d-9d08-4171-89a5-d84cd2039e4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 14:10:21.190411 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:21.190384 4982 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 14:10:21.190581 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:21.190557 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw" podUID="2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 14:10:21.420671 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:21.418144 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" Oct 30 14:10:21.457663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:21.448723 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" podUID="4fa94d86-57ce-4d23-95d5-4efa965f6864" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:21.457663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:21.449896 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" podUID="cbd754bd-771d-4007-8ac0-7e413dc24915" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:21.457663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:21.449894 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8" podUID="f3093abe-9b1e-492a-8837-4bd42ea39d2d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:21.626429 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:21.625736 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" podUID="cc3a3a28-d723-459a-93d0-ff115eac479a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:21.728550 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:21.728098 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" Oct 30 14:10:21.737099 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:21.733039 4982 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" podUID="cbd754bd-771d-4007-8ac0-7e413dc24915" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": context deadline exceeded" Oct 30 14:10:21.741463 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:21.741420 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" podUID="d65dee9b-4f8b-4f09-b00b-0793d732d3ab" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:21.900670 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:21.891145 4982 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 30 14:10:21.900670 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:21.891210 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc" podUID="1ed4b142-fae5-4489-818c-8b53e3fea272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 30 14:10:21.957486 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:21.957363 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:22.054775 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:22.013813 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" podUID="946696b9-91bd-437a-9f23-c16b73a5a9a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:22.161680 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:22.122616 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" Oct 30 14:10:22.455670 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:22.449711 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" podUID="4fa94d86-57ce-4d23-95d5-4efa965f6864" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:22.455670 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:22.452989 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8" Oct 30 14:10:22.458859 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:22.458832 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" Oct 30 14:10:22.472664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:22.469540 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" Oct 30 14:10:22.581160 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:22.581095 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw" Oct 30 14:10:22.581731 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:22.581706 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw" Oct 30 14:10:22.585249 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:22.585226 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw" Oct 30 14:10:23.140039 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:23.139995 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc" Oct 30 14:10:23.140489 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:23.140429 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc" Oct 30 14:10:23.142298 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:23.142275 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc" Oct 30 14:10:23.268918 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:23.268879 4982 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:10:23.269156 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:23.269125 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf" podUID="b616a96d-9d08-4171-89a5-d84cd2039e4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:23.904166 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:23.904127 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf" Oct 30 14:10:23.904721 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:23.904452 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf" Oct 30 14:10:23.905452 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:23.905435 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf" Oct 30 14:10:23.988577 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:23.988548 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:24.810274 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:24.810240 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld"] Oct 30 14:10:24.816036 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:24.816011 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld" Oct 30 14:10:24.849737 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:24.849709 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld"] Oct 30 14:10:24.872024 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:24.871989 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5716f397-1e25-4fd2-b1e3-2b289d1f67c9-istio-token\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld\" (UID: \"5716f397-1e25-4fd2-b1e3-2b289d1f67c9\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld" Oct 30 14:10:24.872160 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:24.872047 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5716f397-1e25-4fd2-b1e3-2b289d1f67c9-istiod-ca-cert\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld\" (UID: \"5716f397-1e25-4fd2-b1e3-2b289d1f67c9\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld" Oct 30 14:10:24.872160 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:24.872146 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5716f397-1e25-4fd2-b1e3-2b289d1f67c9-istio-podinfo\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld\" (UID: \"5716f397-1e25-4fd2-b1e3-2b289d1f67c9\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld" Oct 30 14:10:24.872287 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:24.872212 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5716f397-1e25-4fd2-b1e3-2b289d1f67c9-credential-socket\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld\" (UID: \"5716f397-1e25-4fd2-b1e3-2b289d1f67c9\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld" Oct 30 14:10:24.872287 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:24.872251 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5716f397-1e25-4fd2-b1e3-2b289d1f67c9-workload-certs\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld\" (UID: \"5716f397-1e25-4fd2-b1e3-2b289d1f67c9\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld" Oct 30 14:10:24.872287 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:24.872270 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5716f397-1e25-4fd2-b1e3-2b289d1f67c9-istio-data\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld\" (UID: \"5716f397-1e25-4fd2-b1e3-2b289d1f67c9\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld" Oct 30 14:10:24.872458 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:24.872405 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5716f397-1e25-4fd2-b1e3-2b289d1f67c9-workload-socket\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld\" (UID: \"5716f397-1e25-4fd2-b1e3-2b289d1f67c9\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld" Oct 30 14:10:24.872458 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:24.872440 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5716f397-1e25-4fd2-b1e3-2b289d1f67c9-istio-envoy\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld\" (UID: \"5716f397-1e25-4fd2-b1e3-2b289d1f67c9\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld" Oct 30 14:10:24.872563 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:24.872494 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkbw4\" (UniqueName: \"kubernetes.io/projected/5716f397-1e25-4fd2-b1e3-2b289d1f67c9-kube-api-access-vkbw4\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld\" (UID: \"5716f397-1e25-4fd2-b1e3-2b289d1f67c9\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld" Oct 30 14:10:24.973586 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:24.973539 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5716f397-1e25-4fd2-b1e3-2b289d1f67c9-istio-envoy\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld\" (UID: \"5716f397-1e25-4fd2-b1e3-2b289d1f67c9\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld" Oct 30 14:10:24.974106 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:24.973604 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkbw4\" (UniqueName: \"kubernetes.io/projected/5716f397-1e25-4fd2-b1e3-2b289d1f67c9-kube-api-access-vkbw4\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld\" (UID: \"5716f397-1e25-4fd2-b1e3-2b289d1f67c9\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld" Oct 30 14:10:24.974106 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:24.973748 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5716f397-1e25-4fd2-b1e3-2b289d1f67c9-istio-token\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld\" (UID: \"5716f397-1e25-4fd2-b1e3-2b289d1f67c9\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld" Oct 30 14:10:24.974106 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:24.973794 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5716f397-1e25-4fd2-b1e3-2b289d1f67c9-istiod-ca-cert\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld\" (UID: \"5716f397-1e25-4fd2-b1e3-2b289d1f67c9\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld" Oct 30 14:10:24.974106 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:24.973861 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5716f397-1e25-4fd2-b1e3-2b289d1f67c9-istio-podinfo\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld\" (UID: \"5716f397-1e25-4fd2-b1e3-2b289d1f67c9\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld" Oct 30 14:10:24.974106 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:24.973934 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5716f397-1e25-4fd2-b1e3-2b289d1f67c9-credential-socket\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld\" (UID: \"5716f397-1e25-4fd2-b1e3-2b289d1f67c9\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld" Oct 30 14:10:24.974106 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:24.973964 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5716f397-1e25-4fd2-b1e3-2b289d1f67c9-workload-certs\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld\" (UID: \"5716f397-1e25-4fd2-b1e3-2b289d1f67c9\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld" Oct 30 14:10:24.974106 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:24.974040 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5716f397-1e25-4fd2-b1e3-2b289d1f67c9-istio-data\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld\" (UID: \"5716f397-1e25-4fd2-b1e3-2b289d1f67c9\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld" Oct 30 14:10:24.974898 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:24.974168 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5716f397-1e25-4fd2-b1e3-2b289d1f67c9-workload-socket\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld\" (UID: \"5716f397-1e25-4fd2-b1e3-2b289d1f67c9\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld" Oct 30 14:10:24.974898 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:24.974288 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5716f397-1e25-4fd2-b1e3-2b289d1f67c9-credential-socket\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld\" (UID: \"5716f397-1e25-4fd2-b1e3-2b289d1f67c9\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld" Oct 30 14:10:24.974898 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:24.974314 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5716f397-1e25-4fd2-b1e3-2b289d1f67c9-workload-certs\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld\" (UID: \"5716f397-1e25-4fd2-b1e3-2b289d1f67c9\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld" Oct 30 14:10:24.974898 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:24.974399 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5716f397-1e25-4fd2-b1e3-2b289d1f67c9-istio-data\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld\" (UID: \"5716f397-1e25-4fd2-b1e3-2b289d1f67c9\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld" Oct 30 14:10:24.974898 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:24.974545 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5716f397-1e25-4fd2-b1e3-2b289d1f67c9-workload-socket\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld\" (UID: \"5716f397-1e25-4fd2-b1e3-2b289d1f67c9\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld" Oct 30 14:10:24.974898 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:24.974729 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5716f397-1e25-4fd2-b1e3-2b289d1f67c9-istiod-ca-cert\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld\" (UID: \"5716f397-1e25-4fd2-b1e3-2b289d1f67c9\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld" Oct 30 14:10:24.976564 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:24.976534 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5716f397-1e25-4fd2-b1e3-2b289d1f67c9-istio-podinfo\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld\" (UID: \"5716f397-1e25-4fd2-b1e3-2b289d1f67c9\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld" Oct 30 14:10:24.976564 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:24.976550 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5716f397-1e25-4fd2-b1e3-2b289d1f67c9-istio-envoy\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld\" (UID: \"5716f397-1e25-4fd2-b1e3-2b289d1f67c9\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld" Oct 30 14:10:24.999890 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:24.999863 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkbw4\" (UniqueName: \"kubernetes.io/projected/5716f397-1e25-4fd2-b1e3-2b289d1f67c9-kube-api-access-vkbw4\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld\" (UID: \"5716f397-1e25-4fd2-b1e3-2b289d1f67c9\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld" Oct 30 14:10:25.000337 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:25.000305 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5716f397-1e25-4fd2-b1e3-2b289d1f67c9-istio-token\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld\" (UID: \"5716f397-1e25-4fd2-b1e3-2b289d1f67c9\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld" Oct 30 14:10:25.129759 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:25.129669 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld" Oct 30 14:10:25.465298 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:10:25.465249 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5716f397_1e25_4fd2_b1e3_2b289d1f67c9.slice/crio-336c8f46f8dd36bad6249d725611b8be8840fa645d2f9d882f79826d7d04a484 WatchSource:0}: Error finding container 336c8f46f8dd36bad6249d725611b8be8840fa645d2f9d882f79826d7d04a484: Status 404 returned error can't find the container with id 336c8f46f8dd36bad6249d725611b8be8840fa645d2f9d882f79826d7d04a484 Oct 30 14:10:25.465716 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:25.465692 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld"] Oct 30 14:10:25.468000 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:25.467967 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:10:25.468122 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:25.468033 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:10:25.468122 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:25.468061 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:10:25.810255 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:25.810215 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:26.416666 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:26.413701 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld" event={"ID":"5716f397-1e25-4fd2-b1e3-2b289d1f67c9","Type":"ContainerStarted","Data":"cc7962755abca63077edcc967775e41df63354e8f23e03df114fc9fd38ce88e7"} Oct 30 14:10:26.416666 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:26.413788 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld" event={"ID":"5716f397-1e25-4fd2-b1e3-2b289d1f67c9","Type":"ContainerStarted","Data":"336c8f46f8dd36bad6249d725611b8be8840fa645d2f9d882f79826d7d04a484"} Oct 30 14:10:26.520661 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:26.519758 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld" podStartSLOduration=2.519727217 podStartE2EDuration="2.519727217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:10:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:10:26.464145694 +0000 UTC m=+1296.208279736" watchObservedRunningTime="2026-10-30 14:10:26.519727217 +0000 UTC m=+1296.263861260" Oct 30 14:10:26.831173 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:26.831127 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:27.138713 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:27.137980 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld" Oct 30 14:10:27.138713 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:27.138207 4982 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 14:10:27.138713 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:27.138256 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld" podUID="5716f397-1e25-4fd2-b1e3-2b289d1f67c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 14:10:27.640990 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:27.640956 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:28.114907 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:10:28.114447 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:10:28.130143 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:28.130093 4982 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 14:10:28.130327 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:28.130182 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld" podUID="5716f397-1e25-4fd2-b1e3-2b289d1f67c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 14:10:29.130942 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:29.130901 4982 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 14:10:29.131459 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:29.130973 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld" podUID="5716f397-1e25-4fd2-b1e3-2b289d1f67c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 14:10:29.729828 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:29.729784 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:30.130695 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:30.130652 4982 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 14:10:30.130898 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:30.130724 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld" podUID="5716f397-1e25-4fd2-b1e3-2b289d1f67c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 14:10:31.130281 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:31.130250 4982 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 14:10:31.130901 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:31.130857 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld" podUID="5716f397-1e25-4fd2-b1e3-2b289d1f67c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 14:10:31.532169 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:31.532078 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:32.130135 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:32.130098 4982 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 14:10:32.130296 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:32.130179 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld" podUID="5716f397-1e25-4fd2-b1e3-2b289d1f67c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 14:10:33.134498 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:33.134420 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld" Oct 30 14:10:33.134915 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:33.134892 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld" Oct 30 14:10:33.135632 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:33.135611 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld" Oct 30 14:10:33.504935 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:33.504831 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:35.341829 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:35.341793 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:37.788151 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:37.787747 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" podUID="d65dee9b-4f8b-4f09-b00b-0793d732d3ab" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:37.817924 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:37.817883 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:39.698130 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:39.698067 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:41.461030 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:41.460993 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:42.241237 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:42.241202 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-00926c/distinct-2-w3-00926c-kserve-746b64d6b5-z66nw"] Oct 30 14:10:42.245570 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:42.245547 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-00926c/distinct-2-w3-00926c-kserve-746b64d6b5-z66nw" Oct 30 14:10:42.273759 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:42.273734 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-00926c/distinct-2-w3-00926c-kserve-746b64d6b5-z66nw"] Oct 30 14:10:42.278467 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:42.278349 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-00926c\"/\"distinct-2-w3-00926c-kserve-self-signed-certs\"" Oct 30 14:10:42.338761 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:42.338736 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:42.355160 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:42.355130 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91f37a37-70fe-4627-b2dc-77a9668d9a5f-dshm\") pod \"distinct-2-w3-00926c-kserve-746b64d6b5-z66nw\" (UID: \"91f37a37-70fe-4627-b2dc-77a9668d9a5f\") " pod="e2e-models-e2e-worker-w3-00926c/distinct-2-w3-00926c-kserve-746b64d6b5-z66nw" Oct 30 14:10:42.355307 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:42.355197 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvkrj\" (UniqueName: \"kubernetes.io/projected/91f37a37-70fe-4627-b2dc-77a9668d9a5f-kube-api-access-gvkrj\") pod \"distinct-2-w3-00926c-kserve-746b64d6b5-z66nw\" (UID: \"91f37a37-70fe-4627-b2dc-77a9668d9a5f\") " pod="e2e-models-e2e-worker-w3-00926c/distinct-2-w3-00926c-kserve-746b64d6b5-z66nw" Oct 30 14:10:42.355387 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:42.355329 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91f37a37-70fe-4627-b2dc-77a9668d9a5f-tls-certs\") pod \"distinct-2-w3-00926c-kserve-746b64d6b5-z66nw\" (UID: \"91f37a37-70fe-4627-b2dc-77a9668d9a5f\") " pod="e2e-models-e2e-worker-w3-00926c/distinct-2-w3-00926c-kserve-746b64d6b5-z66nw" Oct 30 14:10:42.355425 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:42.355400 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91f37a37-70fe-4627-b2dc-77a9668d9a5f-tmp-dir\") pod \"distinct-2-w3-00926c-kserve-746b64d6b5-z66nw\" (UID: \"91f37a37-70fe-4627-b2dc-77a9668d9a5f\") " pod="e2e-models-e2e-worker-w3-00926c/distinct-2-w3-00926c-kserve-746b64d6b5-z66nw" Oct 30 14:10:42.355562 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:42.355543 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91f37a37-70fe-4627-b2dc-77a9668d9a5f-home\") pod \"distinct-2-w3-00926c-kserve-746b64d6b5-z66nw\" (UID: \"91f37a37-70fe-4627-b2dc-77a9668d9a5f\") " pod="e2e-models-e2e-worker-w3-00926c/distinct-2-w3-00926c-kserve-746b64d6b5-z66nw" Oct 30 14:10:42.355823 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:42.355801 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91f37a37-70fe-4627-b2dc-77a9668d9a5f-model-cache\") pod \"distinct-2-w3-00926c-kserve-746b64d6b5-z66nw\" (UID: \"91f37a37-70fe-4627-b2dc-77a9668d9a5f\") " pod="e2e-models-e2e-worker-w3-00926c/distinct-2-w3-00926c-kserve-746b64d6b5-z66nw" Oct 30 14:10:42.463724 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:42.463678 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91f37a37-70fe-4627-b2dc-77a9668d9a5f-home\") pod \"distinct-2-w3-00926c-kserve-746b64d6b5-z66nw\" (UID: \"91f37a37-70fe-4627-b2dc-77a9668d9a5f\") " pod="e2e-models-e2e-worker-w3-00926c/distinct-2-w3-00926c-kserve-746b64d6b5-z66nw" Oct 30 14:10:42.515081 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:42.514993 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91f37a37-70fe-4627-b2dc-77a9668d9a5f-model-cache\") pod \"distinct-2-w3-00926c-kserve-746b64d6b5-z66nw\" (UID: \"91f37a37-70fe-4627-b2dc-77a9668d9a5f\") " pod="e2e-models-e2e-worker-w3-00926c/distinct-2-w3-00926c-kserve-746b64d6b5-z66nw" Oct 30 14:10:42.515371 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:42.515347 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91f37a37-70fe-4627-b2dc-77a9668d9a5f-dshm\") pod \"distinct-2-w3-00926c-kserve-746b64d6b5-z66nw\" (UID: \"91f37a37-70fe-4627-b2dc-77a9668d9a5f\") " pod="e2e-models-e2e-worker-w3-00926c/distinct-2-w3-00926c-kserve-746b64d6b5-z66nw" Oct 30 14:10:42.515557 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:42.515540 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvkrj\" (UniqueName: \"kubernetes.io/projected/91f37a37-70fe-4627-b2dc-77a9668d9a5f-kube-api-access-gvkrj\") pod \"distinct-2-w3-00926c-kserve-746b64d6b5-z66nw\" (UID: \"91f37a37-70fe-4627-b2dc-77a9668d9a5f\") " pod="e2e-models-e2e-worker-w3-00926c/distinct-2-w3-00926c-kserve-746b64d6b5-z66nw" Oct 30 14:10:42.515782 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:42.515763 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91f37a37-70fe-4627-b2dc-77a9668d9a5f-tls-certs\") pod \"distinct-2-w3-00926c-kserve-746b64d6b5-z66nw\" (UID: \"91f37a37-70fe-4627-b2dc-77a9668d9a5f\") " pod="e2e-models-e2e-worker-w3-00926c/distinct-2-w3-00926c-kserve-746b64d6b5-z66nw" Oct 30 14:10:42.515925 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:42.515909 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91f37a37-70fe-4627-b2dc-77a9668d9a5f-tmp-dir\") pod \"distinct-2-w3-00926c-kserve-746b64d6b5-z66nw\" (UID: \"91f37a37-70fe-4627-b2dc-77a9668d9a5f\") " pod="e2e-models-e2e-worker-w3-00926c/distinct-2-w3-00926c-kserve-746b64d6b5-z66nw" Oct 30 14:10:42.516232 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:42.464710 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91f37a37-70fe-4627-b2dc-77a9668d9a5f-home\") pod \"distinct-2-w3-00926c-kserve-746b64d6b5-z66nw\" (UID: \"91f37a37-70fe-4627-b2dc-77a9668d9a5f\") " pod="e2e-models-e2e-worker-w3-00926c/distinct-2-w3-00926c-kserve-746b64d6b5-z66nw" Oct 30 14:10:42.527435 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:42.527312 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91f37a37-70fe-4627-b2dc-77a9668d9a5f-model-cache\") pod \"distinct-2-w3-00926c-kserve-746b64d6b5-z66nw\" (UID: \"91f37a37-70fe-4627-b2dc-77a9668d9a5f\") " pod="e2e-models-e2e-worker-w3-00926c/distinct-2-w3-00926c-kserve-746b64d6b5-z66nw" Oct 30 14:10:42.529634 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:42.529610 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91f37a37-70fe-4627-b2dc-77a9668d9a5f-tmp-dir\") pod \"distinct-2-w3-00926c-kserve-746b64d6b5-z66nw\" (UID: \"91f37a37-70fe-4627-b2dc-77a9668d9a5f\") " pod="e2e-models-e2e-worker-w3-00926c/distinct-2-w3-00926c-kserve-746b64d6b5-z66nw" Oct 30 14:10:42.563203 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:42.555913 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91f37a37-70fe-4627-b2dc-77a9668d9a5f-dshm\") pod \"distinct-2-w3-00926c-kserve-746b64d6b5-z66nw\" (UID: \"91f37a37-70fe-4627-b2dc-77a9668d9a5f\") " pod="e2e-models-e2e-worker-w3-00926c/distinct-2-w3-00926c-kserve-746b64d6b5-z66nw" Oct 30 14:10:42.564457 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:42.564422 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvkrj\" (UniqueName: \"kubernetes.io/projected/91f37a37-70fe-4627-b2dc-77a9668d9a5f-kube-api-access-gvkrj\") pod \"distinct-2-w3-00926c-kserve-746b64d6b5-z66nw\" (UID: \"91f37a37-70fe-4627-b2dc-77a9668d9a5f\") " pod="e2e-models-e2e-worker-w3-00926c/distinct-2-w3-00926c-kserve-746b64d6b5-z66nw" Oct 30 14:10:42.583577 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:42.583538 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91f37a37-70fe-4627-b2dc-77a9668d9a5f-tls-certs\") pod \"distinct-2-w3-00926c-kserve-746b64d6b5-z66nw\" (UID: \"91f37a37-70fe-4627-b2dc-77a9668d9a5f\") " pod="e2e-models-e2e-worker-w3-00926c/distinct-2-w3-00926c-kserve-746b64d6b5-z66nw" Oct 30 14:10:42.584602 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:42.584577 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-00926c/distinct-2-w3-00926c-kserve-746b64d6b5-z66nw" Oct 30 14:10:42.607970 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:42.607935 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:43.005741 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:10:43.005708 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91f37a37_70fe_4627_b2dc_77a9668d9a5f.slice/crio-757b53c8e277384284fdd144d0dc6273f72ed86b0848f8d329da9af58ad860b7 WatchSource:0}: Error finding container 757b53c8e277384284fdd144d0dc6273f72ed86b0848f8d329da9af58ad860b7: Status 404 returned error can't find the container with id 757b53c8e277384284fdd144d0dc6273f72ed86b0848f8d329da9af58ad860b7 Oct 30 14:10:43.005872 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:43.005778 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-00926c/distinct-2-w3-00926c-kserve-746b64d6b5-z66nw"] Oct 30 14:10:43.349097 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:43.349058 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:43.647388 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:43.647310 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-00926c/distinct-2-w3-00926c-kserve-746b64d6b5-z66nw" event={"ID":"91f37a37-70fe-4627-b2dc-77a9668d9a5f","Type":"ContainerStarted","Data":"a26dd6fe249791f6e7b2984c8ac9c5790db86bd4a9af61780d45f8e9edb452ed"} Oct 30 14:10:43.647388 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:43.647360 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-00926c/distinct-2-w3-00926c-kserve-746b64d6b5-z66nw" event={"ID":"91f37a37-70fe-4627-b2dc-77a9668d9a5f","Type":"ContainerStarted","Data":"757b53c8e277384284fdd144d0dc6273f72ed86b0848f8d329da9af58ad860b7"} Oct 30 14:10:43.692099 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:43.692050 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-00926c/distinct-2-w3-00926c-kserve-746b64d6b5-z66nw" podStartSLOduration=1.692036855 podStartE2EDuration="1.692036855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:10:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:10:43.691420348 +0000 UTC m=+1313.435554390" watchObservedRunningTime="2026-10-30 14:10:43.692036855 +0000 UTC m=+1313.436170897" Oct 30 14:10:44.585057 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:44.585017 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-00926c/distinct-2-w3-00926c-kserve-746b64d6b5-z66nw" Oct 30 14:10:45.133066 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:45.133027 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:46.952910 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:46.952871 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:48.874593 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:48.874556 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:50.700581 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:50.700548 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:52.543476 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:52.543422 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:52.585695 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:52.585661 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-00926c/distinct-2-w3-00926c-kserve-746b64d6b5-z66nw" Oct 30 14:10:52.602321 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:52.602287 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-00926c/distinct-2-w3-00926c-kserve-746b64d6b5-z66nw" Oct 30 14:10:52.718916 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:52.718884 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-00926c/distinct-2-w3-00926c-kserve-746b64d6b5-z66nw" Oct 30 14:10:54.369462 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:54.369426 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:55.684101 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:55.684062 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp"] Oct 30 14:10:55.689834 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:55.689809 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp" Oct 30 14:10:55.727417 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:55.727382 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp"] Oct 30 14:10:55.831183 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:55.831135 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3ffe590-102f-420e-b108-3d6ea542c0b4-istio-token\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp\" (UID: \"e3ffe590-102f-420e-b108-3d6ea542c0b4\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp" Oct 30 14:10:55.831183 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:55.831187 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3ffe590-102f-420e-b108-3d6ea542c0b4-istio-envoy\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp\" (UID: \"e3ffe590-102f-420e-b108-3d6ea542c0b4\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp" Oct 30 14:10:55.831551 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:55.831202 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ffe590-102f-420e-b108-3d6ea542c0b4-credential-socket\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp\" (UID: \"e3ffe590-102f-420e-b108-3d6ea542c0b4\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp" Oct 30 14:10:55.831551 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:55.831299 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3ffe590-102f-420e-b108-3d6ea542c0b4-istiod-ca-cert\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp\" (UID: \"e3ffe590-102f-420e-b108-3d6ea542c0b4\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp" Oct 30 14:10:55.831551 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:55.831443 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3ffe590-102f-420e-b108-3d6ea542c0b4-workload-certs\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp\" (UID: \"e3ffe590-102f-420e-b108-3d6ea542c0b4\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp" Oct 30 14:10:55.831551 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:55.831494 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ffe590-102f-420e-b108-3d6ea542c0b4-workload-socket\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp\" (UID: \"e3ffe590-102f-420e-b108-3d6ea542c0b4\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp" Oct 30 14:10:55.831551 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:55.831522 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3ffe590-102f-420e-b108-3d6ea542c0b4-istio-data\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp\" (UID: \"e3ffe590-102f-420e-b108-3d6ea542c0b4\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp" Oct 30 14:10:55.831766 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:55.831596 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3ffe590-102f-420e-b108-3d6ea542c0b4-istio-podinfo\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp\" (UID: \"e3ffe590-102f-420e-b108-3d6ea542c0b4\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp" Oct 30 14:10:55.831766 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:55.831757 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljpq5\" (UniqueName: \"kubernetes.io/projected/e3ffe590-102f-420e-b108-3d6ea542c0b4-kube-api-access-ljpq5\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp\" (UID: \"e3ffe590-102f-420e-b108-3d6ea542c0b4\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp" Oct 30 14:10:55.933207 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:55.933165 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3ffe590-102f-420e-b108-3d6ea542c0b4-istio-token\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp\" (UID: \"e3ffe590-102f-420e-b108-3d6ea542c0b4\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp" Oct 30 14:10:55.933207 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:55.933214 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3ffe590-102f-420e-b108-3d6ea542c0b4-istio-envoy\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp\" (UID: \"e3ffe590-102f-420e-b108-3d6ea542c0b4\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp" Oct 30 14:10:55.933443 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:55.933231 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ffe590-102f-420e-b108-3d6ea542c0b4-credential-socket\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp\" (UID: \"e3ffe590-102f-420e-b108-3d6ea542c0b4\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp" Oct 30 14:10:55.933443 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:55.933341 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3ffe590-102f-420e-b108-3d6ea542c0b4-istiod-ca-cert\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp\" (UID: \"e3ffe590-102f-420e-b108-3d6ea542c0b4\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp" Oct 30 14:10:55.933443 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:55.933380 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3ffe590-102f-420e-b108-3d6ea542c0b4-workload-certs\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp\" (UID: \"e3ffe590-102f-420e-b108-3d6ea542c0b4\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp" Oct 30 14:10:55.933443 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:55.933432 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ffe590-102f-420e-b108-3d6ea542c0b4-workload-socket\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp\" (UID: \"e3ffe590-102f-420e-b108-3d6ea542c0b4\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp" Oct 30 14:10:55.933677 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:55.933458 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3ffe590-102f-420e-b108-3d6ea542c0b4-istio-data\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp\" (UID: \"e3ffe590-102f-420e-b108-3d6ea542c0b4\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp" Oct 30 14:10:55.933677 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:55.933484 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3ffe590-102f-420e-b108-3d6ea542c0b4-istio-podinfo\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp\" (UID: \"e3ffe590-102f-420e-b108-3d6ea542c0b4\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp" Oct 30 14:10:55.933677 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:55.933631 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ffe590-102f-420e-b108-3d6ea542c0b4-credential-socket\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp\" (UID: \"e3ffe590-102f-420e-b108-3d6ea542c0b4\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp" Oct 30 14:10:55.933849 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:55.933716 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljpq5\" (UniqueName: \"kubernetes.io/projected/e3ffe590-102f-420e-b108-3d6ea542c0b4-kube-api-access-ljpq5\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp\" (UID: \"e3ffe590-102f-420e-b108-3d6ea542c0b4\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp" Oct 30 14:10:55.933849 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:55.933820 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3ffe590-102f-420e-b108-3d6ea542c0b4-workload-certs\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp\" (UID: \"e3ffe590-102f-420e-b108-3d6ea542c0b4\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp" Oct 30 14:10:55.934176 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:55.934086 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3ffe590-102f-420e-b108-3d6ea542c0b4-istiod-ca-cert\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp\" (UID: \"e3ffe590-102f-420e-b108-3d6ea542c0b4\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp" Oct 30 14:10:55.934176 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:55.934144 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3ffe590-102f-420e-b108-3d6ea542c0b4-istio-data\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp\" (UID: \"e3ffe590-102f-420e-b108-3d6ea542c0b4\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp" Oct 30 14:10:55.934369 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:55.934229 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ffe590-102f-420e-b108-3d6ea542c0b4-workload-socket\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp\" (UID: \"e3ffe590-102f-420e-b108-3d6ea542c0b4\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp" Oct 30 14:10:55.935854 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:55.935830 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3ffe590-102f-420e-b108-3d6ea542c0b4-istio-envoy\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp\" (UID: \"e3ffe590-102f-420e-b108-3d6ea542c0b4\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp" Oct 30 14:10:55.935980 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:55.935963 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3ffe590-102f-420e-b108-3d6ea542c0b4-istio-podinfo\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp\" (UID: \"e3ffe590-102f-420e-b108-3d6ea542c0b4\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp" Oct 30 14:10:55.959132 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:55.959097 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3ffe590-102f-420e-b108-3d6ea542c0b4-istio-token\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp\" (UID: \"e3ffe590-102f-420e-b108-3d6ea542c0b4\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp" Oct 30 14:10:55.959300 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:55.959276 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljpq5\" (UniqueName: \"kubernetes.io/projected/e3ffe590-102f-420e-b108-3d6ea542c0b4-kube-api-access-ljpq5\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp\" (UID: \"e3ffe590-102f-420e-b108-3d6ea542c0b4\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp" Oct 30 14:10:56.007309 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:56.007270 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp" Oct 30 14:10:56.174399 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:56.174314 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:56.336407 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:10:56.336376 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3ffe590_102f_420e_b108_3d6ea542c0b4.slice/crio-e1dfee73ad96fb4293cf143176a1b9292f43a26cf52b51853863e13387cdb3f9 WatchSource:0}: Error finding container e1dfee73ad96fb4293cf143176a1b9292f43a26cf52b51853863e13387cdb3f9: Status 404 returned error can't find the container with id e1dfee73ad96fb4293cf143176a1b9292f43a26cf52b51853863e13387cdb3f9 Oct 30 14:10:56.338786 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:56.338757 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp"] Oct 30 14:10:56.339396 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:56.339362 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:10:56.339473 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:56.339429 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:10:56.339473 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:56.339459 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:10:56.742761 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:56.742671 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp" event={"ID":"e3ffe590-102f-420e-b108-3d6ea542c0b4","Type":"ContainerStarted","Data":"ebe47a4c527369313c27c3c37b1355d5191b33bd6ee24038ed99f1e40402f0cd"} Oct 30 14:10:56.742761 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:56.742742 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp" event={"ID":"e3ffe590-102f-420e-b108-3d6ea542c0b4","Type":"ContainerStarted","Data":"e1dfee73ad96fb4293cf143176a1b9292f43a26cf52b51853863e13387cdb3f9"} Oct 30 14:10:56.795476 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:56.795407 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp" podStartSLOduration=1.7953869390000001 podStartE2EDuration="1.795386939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:10:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:10:56.792900655 +0000 UTC m=+1326.537034702" watchObservedRunningTime="2026-10-30 14:10:56.795386939 +0000 UTC m=+1326.539520982" Oct 30 14:10:56.878084 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:56.878039 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:57.009716 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:57.008705 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp" Oct 30 14:10:57.010774 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:57.010439 4982 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 14:10:57.010774 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:57.010502 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp" podUID="e3ffe590-102f-420e-b108-3d6ea542c0b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 14:10:58.009092 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:58.009054 4982 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 14:10:58.009689 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:58.009634 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp" podUID="e3ffe590-102f-420e-b108-3d6ea542c0b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 14:10:58.043677 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:10:58.042842 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:10:58.376589 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:58.376554 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:10:59.010792 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:59.010748 4982 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 14:10:59.011285 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:10:59.010831 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp" podUID="e3ffe590-102f-420e-b108-3d6ea542c0b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 14:11:00.008131 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:00.008086 4982 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 14:11:00.008281 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:00.008161 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp" podUID="e3ffe590-102f-420e-b108-3d6ea542c0b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 14:11:00.364981 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:00.364935 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:01.051670 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:01.049742 4982 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 14:11:01.051670 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:01.049800 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp" podUID="e3ffe590-102f-420e-b108-3d6ea542c0b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 14:11:01.597664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:01.596292 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" podUID="cc3a3a28-d723-459a-93d0-ff115eac479a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:01.731665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:01.730729 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" podUID="d65dee9b-4f8b-4f09-b00b-0793d732d3ab" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:01.919722 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:01.917782 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/distinct-w3-00926c-kserve-774df5b58-cv8vt" podUID="b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:02.009089 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:02.009048 4982 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 14:11:02.009386 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:02.009355 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp" podUID="e3ffe590-102f-420e-b108-3d6ea542c0b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 14:11:02.050770 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:02.050360 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" podUID="946696b9-91bd-437a-9f23-c16b73a5a9a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:02.202995 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:02.201759 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" podUID="00fdf663-9bc9-4349-a565-6a8dc01c0842" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:02.434665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:02.421709 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" podUID="15e7b7f9-2570-410a-a398-4c07c2e465ea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:02.438557 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:02.437245 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" podUID="ab822fe6-4349-4051-b2fd-c6e5ceb50d08" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded" Oct 30 14:11:02.460790 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:02.451545 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" podUID="4fa94d86-57ce-4d23-95d5-4efa965f6864" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:02.464668 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:02.464091 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8" podUID="f3093abe-9b1e-492a-8837-4bd42ea39d2d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:02.504951 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:02.503263 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" podUID="cbd754bd-771d-4007-8ac0-7e413dc24915" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:02.600666 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:02.599745 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" podUID="cc3a3a28-d723-459a-93d0-ff115eac479a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:02.600666 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:02.600665 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" Oct 30 14:11:02.856237 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:02.707741 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/distinct-2-w3-00926c-kserve-746b64d6b5-z66nw" podUID="91f37a37-70fe-4627-b2dc-77a9668d9a5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:02.856237 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:02.738731 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" podUID="d65dee9b-4f8b-4f09-b00b-0793d732d3ab" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:02.856237 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:02.739508 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" Oct 30 14:11:02.949666 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:02.937177 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/distinct-w3-00926c-kserve-774df5b58-cv8vt" podUID="b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:02.949666 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:02.937930 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-00926c/distinct-w3-00926c-kserve-774df5b58-cv8vt" Oct 30 14:11:03.019733 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:03.017594 4982 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 14:11:03.019733 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:03.017677 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp" podUID="e3ffe590-102f-420e-b108-3d6ea542c0b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 14:11:03.066569 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:03.065166 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" podUID="946696b9-91bd-437a-9f23-c16b73a5a9a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 14:11:03.066569 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:03.066506 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" Oct 30 14:11:03.076879 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:03.068334 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" Oct 30 14:11:03.191957 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:03.190084 4982 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:11:03.191957 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:03.190149 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml" podUID="a6687b0d-4a6b-4fe8-9c62-47057042db79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:03.226561 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:03.216756 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" podUID="00fdf663-9bc9-4349-a565-6a8dc01c0842" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:03.226561 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:03.218022 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" Oct 30 14:11:03.466589 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:03.466474 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8" podUID="f3093abe-9b1e-492a-8837-4bd42ea39d2d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:03.503664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:03.482886 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" podUID="15e7b7f9-2570-410a-a398-4c07c2e465ea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded" Oct 30 14:11:03.529697 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:03.507877 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8" Oct 30 14:11:03.529697 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:03.508182 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" Oct 30 14:11:03.619491 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:03.568923 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" podUID="ab822fe6-4349-4051-b2fd-c6e5ceb50d08" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded" Oct 30 14:11:03.654687 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:03.601331 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" podUID="4fa94d86-57ce-4d23-95d5-4efa965f6864" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:03.654687 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:03.620086 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" podUID="cbd754bd-771d-4007-8ac0-7e413dc24915" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:03.654687 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:03.648607 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" podUID="cc3a3a28-d723-459a-93d0-ff115eac479a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:03.654687 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:03.648909 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" Oct 30 14:11:03.654687 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:03.649349 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" Oct 30 14:11:03.654687 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:03.650529 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" Oct 30 14:11:03.660955 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:03.620623 4982 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-00926c/distinct-2-w3-00926c-kserve-746b64d6b5-z66nw" podUID="91f37a37-70fe-4627-b2dc-77a9668d9a5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": context deadline exceeded" Oct 30 14:11:03.722720 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:03.715846 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/distinct-2-w3-00926c-kserve-746b64d6b5-z66nw" podUID="91f37a37-70fe-4627-b2dc-77a9668d9a5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:03.771994 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:03.759752 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-00926c/distinct-2-w3-00926c-kserve-746b64d6b5-z66nw" Oct 30 14:11:03.973720 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:03.943236 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/distinct-w3-00926c-kserve-774df5b58-cv8vt" podUID="b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:03.999664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:03.957627 4982 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:11:03.999664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:03.992183 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4" podUID="b14bb682-071a-4c22-ad2d-cab7434a4192" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:04.128664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:04.071707 4982 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:11:04.128664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:04.071780 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls" podUID="2a20886d-6388-4460-ae47-12a5a28e0eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:04.128664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:04.116130 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" podUID="d65dee9b-4f8b-4f09-b00b-0793d732d3ab" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:04.157666 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:04.133181 4982 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 14:11:04.172041 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:04.171999 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" podUID="946696b9-91bd-437a-9f23-c16b73a5a9a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 14:11:04.191771 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:04.191713 4982 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" podUID="15e7b7f9-2570-410a-a398-4c07c2e465ea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": context deadline exceeded" Oct 30 14:11:04.214658 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:04.211037 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp" podUID="e3ffe590-102f-420e-b108-3d6ea542c0b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 14:11:04.214896 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:04.211520 4982 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:11:04.215027 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:04.215004 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84" podUID="e0991b98-e891-4582-a9ce-88846fcaad92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:04.215133 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:04.211543 4982 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:11:04.215234 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:04.215215 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk" podUID="e8fa477f-0e45-42fa-9941-514ea90d1094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:04.220715 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:04.220684 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" podUID="00fdf663-9bc9-4349-a565-6a8dc01c0842" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:04.256203 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:04.256112 4982 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:11:04.256501 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:04.256468 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld" podUID="5716f397-1e25-4fd2-b1e3-2b289d1f67c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:04.418662 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:04.418316 4982 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:11:04.418662 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:04.418397 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk" podUID="99cbf845-7193-4e4d-8315-b5b0614cfd33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:04.484203 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:04.484151 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8" podUID="f3093abe-9b1e-492a-8837-4bd42ea39d2d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:04.508125 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:04.508027 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" podUID="15e7b7f9-2570-410a-a398-4c07c2e465ea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:04.643661 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:04.639768 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" podUID="cbd754bd-771d-4007-8ac0-7e413dc24915" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:04.659754 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:04.659715 4982 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:11:04.660021 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:04.659988 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w" podUID="7db57691-2261-4a03-abe3-a0fa7bb45429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:04.660193 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:04.660174 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" podUID="ab822fe6-4349-4051-b2fd-c6e5ceb50d08" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded" Oct 30 14:11:04.661061 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:04.661029 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" podUID="cc3a3a28-d723-459a-93d0-ff115eac479a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:04.661871 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:04.661846 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" podUID="4fa94d86-57ce-4d23-95d5-4efa965f6864" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:04.748675 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:04.747749 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/distinct-2-w3-00926c-kserve-746b64d6b5-z66nw" podUID="91f37a37-70fe-4627-b2dc-77a9668d9a5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 30 14:11:04.748867 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:04.748709 4982 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" podUID="00fdf663-9bc9-4349-a565-6a8dc01c0842" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:04.998187 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:04.998140 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/distinct-w3-00926c-kserve-774df5b58-cv8vt" podUID="b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:05.018701 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:05.008948 4982 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-00926c/distinct-w3-00926c-kserve-774df5b58-cv8vt" podUID="b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": context deadline exceeded" Oct 30 14:11:05.021268 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:05.021237 4982 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 14:11:05.021576 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:05.021549 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp" podUID="e3ffe590-102f-420e-b108-3d6ea542c0b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 14:11:05.117766 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:05.117724 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" podUID="d65dee9b-4f8b-4f09-b00b-0793d732d3ab" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded" Oct 30 14:11:05.122668 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:05.118455 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" Oct 30 14:11:05.221879 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:05.221820 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" podUID="946696b9-91bd-437a-9f23-c16b73a5a9a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 14:11:05.222284 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:05.222250 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" podUID="00fdf663-9bc9-4349-a565-6a8dc01c0842" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:05.488762 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:05.488717 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8" podUID="f3093abe-9b1e-492a-8837-4bd42ea39d2d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Oct 30 14:11:05.669090 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:05.547746 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" podUID="15e7b7f9-2570-410a-a398-4c07c2e465ea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:05.670348 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:05.640028 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" podUID="cbd754bd-771d-4007-8ac0-7e413dc24915" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:05.694664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:05.661123 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" podUID="ab822fe6-4349-4051-b2fd-c6e5ceb50d08" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:05.694664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:05.668717 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" podUID="4fa94d86-57ce-4d23-95d5-4efa965f6864" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:05.694664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:05.668724 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" podUID="cc3a3a28-d723-459a-93d0-ff115eac479a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:05.761741 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:05.760546 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/distinct-2-w3-00926c-kserve-746b64d6b5-z66nw" podUID="91f37a37-70fe-4627-b2dc-77a9668d9a5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 30 14:11:05.844376 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:05.844331 4982 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:11:05.844668 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:05.844619 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc" podUID="6a3afaea-c040-4642-8632-6b5a0dca807c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:06.001659 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:06.000488 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/distinct-w3-00926c-kserve-774df5b58-cv8vt" podUID="b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 30 14:11:06.020842 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:06.020753 4982 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 14:11:06.021115 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:06.021081 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp" podUID="e3ffe590-102f-420e-b108-3d6ea542c0b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 14:11:06.124785 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:06.124733 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" podUID="d65dee9b-4f8b-4f09-b00b-0793d732d3ab" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded" Oct 30 14:11:06.225621 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:06.223733 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" podUID="00fdf663-9bc9-4349-a565-6a8dc01c0842" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:06.225621 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:06.224455 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" podUID="946696b9-91bd-437a-9f23-c16b73a5a9a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 14:11:06.642897 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:06.642853 4982 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" podUID="d65dee9b-4f8b-4f09-b00b-0793d732d3ab" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:06.651309 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:06.651271 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:06.669087 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:06.668873 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8" podUID="f3093abe-9b1e-492a-8837-4bd42ea39d2d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Oct 30 14:11:06.674039 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:06.673704 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" podUID="15e7b7f9-2570-410a-a398-4c07c2e465ea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded" Oct 30 14:11:06.688470 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:06.687724 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" podUID="cbd754bd-771d-4007-8ac0-7e413dc24915" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:06.696675 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:06.688723 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" podUID="4fa94d86-57ce-4d23-95d5-4efa965f6864" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 14:11:06.696675 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:06.689015 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" podUID="ab822fe6-4349-4051-b2fd-c6e5ceb50d08" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded" Oct 30 14:11:06.696675 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:06.689277 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" podUID="cc3a3a28-d723-459a-93d0-ff115eac479a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded" Oct 30 14:11:06.773673 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:06.770437 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/distinct-2-w3-00926c-kserve-746b64d6b5-z66nw" podUID="91f37a37-70fe-4627-b2dc-77a9668d9a5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:07.018614 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:07.018509 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/distinct-w3-00926c-kserve-774df5b58-cv8vt" podUID="b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 30 14:11:07.028127 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:07.028094 4982 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 14:11:07.028434 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:07.028381 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp" podUID="e3ffe590-102f-420e-b108-3d6ea542c0b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 14:11:07.161827 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:07.161768 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" podUID="d65dee9b-4f8b-4f09-b00b-0793d732d3ab" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded" Oct 30 14:11:07.349949 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:07.319271 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" podUID="946696b9-91bd-437a-9f23-c16b73a5a9a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:07.350954 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:07.319305 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" podUID="00fdf663-9bc9-4349-a565-6a8dc01c0842" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:07.396680 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:07.379341 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" Oct 30 14:11:07.519879 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:07.519846 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" Oct 30 14:11:07.521119 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:07.521095 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8" Oct 30 14:11:07.531103 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:07.531081 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" Oct 30 14:11:07.556441 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:07.552877 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" Oct 30 14:11:07.574282 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:07.570824 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" Oct 30 14:11:07.574282 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:07.571435 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" Oct 30 14:11:07.574282 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:07.571685 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-00926c/distinct-2-w3-00926c-kserve-746b64d6b5-z66nw" Oct 30 14:11:07.588452 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:07.588429 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-00926c/distinct-w3-00926c-kserve-774df5b58-cv8vt" Oct 30 14:11:07.596229 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:07.594832 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" Oct 30 14:11:07.601762 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:07.601709 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" Oct 30 14:11:08.012979 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:08.012886 4982 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 14:11:08.013506 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:08.013474 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp" podUID="e3ffe590-102f-420e-b108-3d6ea542c0b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 14:11:08.536358 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:08.536324 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:09.175489 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:09.175461 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp" Oct 30 14:11:09.175975 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:09.175947 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp" Oct 30 14:11:09.176795 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:09.176775 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp" Oct 30 14:11:10.527062 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:10.526996 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:12.322388 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:12.322352 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:14.153773 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:14.153737 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:16.059708 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:16.059673 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:17.881580 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:17.881546 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:19.704796 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:19.704759 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:21.638570 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:21.638539 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:23.456907 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:23.456819 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:25.236701 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:25.236668 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:27.084351 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:27.084307 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:28.022827 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:11:28.022788 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:11:29.583776 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:29.583722 4982 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" podUID="ab822fe6-4349-4051-b2fd-c6e5ceb50d08" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:29.631203 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:29.631154 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" podUID="946696b9-91bd-437a-9f23-c16b73a5a9a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:29.676665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:29.670970 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" podUID="cc3a3a28-d723-459a-93d0-ff115eac479a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded" Oct 30 14:11:29.676941 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:29.676909 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" podUID="15e7b7f9-2570-410a-a398-4c07c2e465ea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded" Oct 30 14:11:29.689007 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:29.688970 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" podUID="4fa94d86-57ce-4d23-95d5-4efa965f6864" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:29.705666 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:29.700800 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" podUID="ab822fe6-4349-4051-b2fd-c6e5ceb50d08" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:30.634814 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:30.634768 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" podUID="946696b9-91bd-437a-9f23-c16b73a5a9a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:30.636893 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:30.636861 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" Oct 30 14:11:30.671812 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:30.671767 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" podUID="cc3a3a28-d723-459a-93d0-ff115eac479a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:30.673633 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:30.673610 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" Oct 30 14:11:30.683399 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:30.681726 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" podUID="15e7b7f9-2570-410a-a398-4c07c2e465ea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:30.683399 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:30.682112 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" Oct 30 14:11:30.695672 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:30.691720 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" podUID="4fa94d86-57ce-4d23-95d5-4efa965f6864" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 14:11:30.695672 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:30.692112 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" Oct 30 14:11:30.731664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:30.714912 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" podUID="ab822fe6-4349-4051-b2fd-c6e5ceb50d08" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:30.734091 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:30.734062 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" Oct 30 14:11:31.178666 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:30.827185 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/distinct-2-w3-00926c-kserve-746b64d6b5-z66nw" podUID="91f37a37-70fe-4627-b2dc-77a9668d9a5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:31.178666 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:31.008766 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/distinct-w3-00926c-kserve-774df5b58-cv8vt" podUID="b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 30 14:11:31.178666 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:31.151468 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" podUID="d65dee9b-4f8b-4f09-b00b-0793d732d3ab" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:31.225376 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:31.225332 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" podUID="00fdf663-9bc9-4349-a565-6a8dc01c0842" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:31.389364 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:31.389316 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" podUID="cbd754bd-771d-4007-8ac0-7e413dc24915" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:31.564915 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:31.563871 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8" podUID="f3093abe-9b1e-492a-8837-4bd42ea39d2d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:31.650001 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:31.649953 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" podUID="946696b9-91bd-437a-9f23-c16b73a5a9a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:31.686034 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:31.676751 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" podUID="cc3a3a28-d723-459a-93d0-ff115eac479a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded" Oct 30 14:11:31.686688 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:31.686633 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" podUID="15e7b7f9-2570-410a-a398-4c07c2e465ea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:31.719122 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:31.719080 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" podUID="4fa94d86-57ce-4d23-95d5-4efa965f6864" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 14:11:31.732959 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:31.732918 4982 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" podUID="cbd754bd-771d-4007-8ac0-7e413dc24915" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:31.736663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:31.733212 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" podUID="ab822fe6-4349-4051-b2fd-c6e5ceb50d08" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded" Oct 30 14:11:32.163664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:32.152751 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/distinct-2-w3-00926c-kserve-746b64d6b5-z66nw" podUID="91f37a37-70fe-4627-b2dc-77a9668d9a5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:32.163664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:32.155010 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-00926c/distinct-2-w3-00926c-kserve-746b64d6b5-z66nw" Oct 30 14:11:32.163664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:32.156179 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" podUID="d65dee9b-4f8b-4f09-b00b-0793d732d3ab" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:32.167562 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:32.167524 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/distinct-w3-00926c-kserve-774df5b58-cv8vt" podUID="b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 30 14:11:32.168934 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:32.168912 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" Oct 30 14:11:32.182060 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:32.182034 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-00926c/distinct-w3-00926c-kserve-774df5b58-cv8vt" Oct 30 14:11:32.404898 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:32.308221 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" podUID="00fdf663-9bc9-4349-a565-6a8dc01c0842" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 14:11:32.404898 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:32.319054 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" Oct 30 14:11:32.422363 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:32.415793 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" podUID="cbd754bd-771d-4007-8ac0-7e413dc24915" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:32.579670 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:32.577667 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8" podUID="f3093abe-9b1e-492a-8837-4bd42ea39d2d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:32.683348 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:32.683253 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" podUID="cc3a3a28-d723-459a-93d0-ff115eac479a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:32.821663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:32.684021 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" podUID="946696b9-91bd-437a-9f23-c16b73a5a9a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 14:11:32.821663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:32.713462 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" podUID="15e7b7f9-2570-410a-a398-4c07c2e465ea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded" Oct 30 14:11:32.821663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:32.740719 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" podUID="ab822fe6-4349-4051-b2fd-c6e5ceb50d08" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:32.822004 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:32.741162 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" podUID="4fa94d86-57ce-4d23-95d5-4efa965f6864" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:32.822065 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:32.785733 4982 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:11:32.822122 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:32.822064 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4" podUID="de285ba3-daf9-4154-98b0-95b642020499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:33.014017 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:32.992487 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" Oct 30 14:11:33.014314 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:33.014293 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8" Oct 30 14:11:33.142389 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:33.142341 4982 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:11:33.142702 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:33.142671 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml" podUID="a6687b0d-4a6b-4fe8-9c62-47057042db79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:33.174899 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:33.174857 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/distinct-w3-00926c-kserve-774df5b58-cv8vt" podUID="b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:33.175615 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:33.175583 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" podUID="d65dee9b-4f8b-4f09-b00b-0793d732d3ab" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:33.176120 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:33.176097 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/distinct-2-w3-00926c-kserve-746b64d6b5-z66nw" podUID="91f37a37-70fe-4627-b2dc-77a9668d9a5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:33.555704 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:33.555659 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" podUID="00fdf663-9bc9-4349-a565-6a8dc01c0842" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 14:11:33.623664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:33.616406 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" podUID="cbd754bd-771d-4007-8ac0-7e413dc24915" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 14:11:33.650593 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:33.650546 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:33.678952 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:33.678904 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8" podUID="f3093abe-9b1e-492a-8837-4bd42ea39d2d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Oct 30 14:11:33.779671 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:33.694879 4982 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-00926c/distinct-2-w3-00926c-kserve-746b64d6b5-z66nw" podUID="91f37a37-70fe-4627-b2dc-77a9668d9a5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": context deadline exceeded" Oct 30 14:11:33.780969 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:33.758957 4982 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:11:33.808738 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:33.807714 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8" podUID="00eefe61-57f5-4d1c-b9a5-8dd54db2922e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:33.808738 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:33.783562 4982 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:11:33.808738 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:33.807812 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp" podUID="2ae22b83-f2b9-47c9-b51b-f70a43f699b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:33.832807 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:33.832742 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" podUID="cc3a3a28-d723-459a-93d0-ff115eac479a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:33.850517 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:33.850470 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" podUID="4fa94d86-57ce-4d23-95d5-4efa965f6864" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:33.851126 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:33.851093 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" podUID="946696b9-91bd-437a-9f23-c16b73a5a9a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:33.851587 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:33.851561 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" podUID="15e7b7f9-2570-410a-a398-4c07c2e465ea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:33.852020 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:33.851995 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" podUID="ab822fe6-4349-4051-b2fd-c6e5ceb50d08" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded" Oct 30 14:11:34.012673 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:34.010043 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" Oct 30 14:11:34.022292 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:34.022241 4982 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:11:34.022564 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:34.022529 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls" podUID="2a20886d-6388-4460-ae47-12a5a28e0eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:34.099905 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:34.099820 4982 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" podUID="15e7b7f9-2570-410a-a398-4c07c2e465ea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:34.151828 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:34.151779 4982 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:11:34.152109 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:34.152066 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84" podUID="e0991b98-e891-4582-a9ce-88846fcaad92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:34.153080 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:34.153052 4982 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:11:34.156776 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:34.156710 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk" podUID="e8fa477f-0e45-42fa-9941-514ea90d1094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:34.187048 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:34.186846 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/distinct-w3-00926c-kserve-774df5b58-cv8vt" podUID="b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:34.225236 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:34.225194 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8" Oct 30 14:11:34.226945 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:34.226912 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/distinct-2-w3-00926c-kserve-746b64d6b5-z66nw" podUID="91f37a37-70fe-4627-b2dc-77a9668d9a5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:34.555694 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:34.555586 4982 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:11:34.637681 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:34.578912 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk" podUID="99cbf845-7193-4e4d-8315-b5b0614cfd33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:34.637681 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:34.588704 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" podUID="00fdf663-9bc9-4349-a565-6a8dc01c0842" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 14:11:34.655246 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:34.638297 4982 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:11:34.655246 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:34.638356 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w" podUID="7db57691-2261-4a03-abe3-a0fa7bb45429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:34.655246 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:34.649211 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" podUID="cbd754bd-771d-4007-8ac0-7e413dc24915" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:34.723695 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:34.712751 4982 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" podUID="00fdf663-9bc9-4349-a565-6a8dc01c0842" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:34.880664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:34.873350 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" podUID="ab822fe6-4349-4051-b2fd-c6e5ceb50d08" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:34.880664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:34.874498 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" podUID="cc3a3a28-d723-459a-93d0-ff115eac479a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded" Oct 30 14:11:34.880664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:34.875490 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" podUID="4fa94d86-57ce-4d23-95d5-4efa965f6864" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 14:11:34.880664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:34.876467 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" podUID="946696b9-91bd-437a-9f23-c16b73a5a9a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 14:11:34.880664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:34.877389 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" podUID="15e7b7f9-2570-410a-a398-4c07c2e465ea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded" Oct 30 14:11:34.988015 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:34.985445 4982 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-00926c/distinct-w3-00926c-kserve-774df5b58-cv8vt" podUID="b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:35.190824 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:35.190726 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/distinct-w3-00926c-kserve-774df5b58-cv8vt" podUID="b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:35.230683 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:35.230629 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/distinct-2-w3-00926c-kserve-746b64d6b5-z66nw" podUID="91f37a37-70fe-4627-b2dc-77a9668d9a5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 30 14:11:35.342849 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:35.341693 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" Oct 30 14:11:35.343959 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:35.343128 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" Oct 30 14:11:35.379685 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:35.364152 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" Oct 30 14:11:35.595066 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:35.595034 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" Oct 30 14:11:35.604782 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:35.604757 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-00926c/distinct-w3-00926c-kserve-774df5b58-cv8vt" Oct 30 14:11:35.616263 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:35.616241 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-00926c/distinct-2-w3-00926c-kserve-746b64d6b5-z66nw" Oct 30 14:11:35.649772 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:35.649745 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" Oct 30 14:11:35.658659 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:35.650772 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" podUID="cbd754bd-771d-4007-8ac0-7e413dc24915" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:35.670061 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:35.670042 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" Oct 30 14:11:35.676545 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:35.676527 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" Oct 30 14:11:36.351208 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:36.351167 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:36.548456 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:36.548422 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:37.955265 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:37.955225 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:38.103066 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:38.103024 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:38.263670 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:38.263575 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:40.084194 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:40.084166 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:42.072271 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:42.072234 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:42.716073 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:42.716041 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:43.890056 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:43.890020 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:45.819972 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:45.819930 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:47.658622 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:47.658580 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:49.668316 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:49.668256 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:51.462789 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:51.462741 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:53.243505 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:53.243449 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:55.269781 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:55.269741 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:57.548712 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:57.548626 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" podUID="00fdf663-9bc9-4349-a565-6a8dc01c0842" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:57.618272 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:57.618216 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/distinct-2-w3-00926c-kserve-746b64d6b5-z66nw" podUID="91f37a37-70fe-4627-b2dc-77a9668d9a5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:57.681958 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:57.681890 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" podUID="cc3a3a28-d723-459a-93d0-ff115eac479a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:57.716893 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:57.716077 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" podUID="15e7b7f9-2570-410a-a398-4c07c2e465ea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:57.717720 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:57.717686 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" podUID="cbd754bd-771d-4007-8ac0-7e413dc24915" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:57.744426 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:57.744349 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" podUID="ab822fe6-4349-4051-b2fd-c6e5ceb50d08" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:57.744825 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:57.744760 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" podUID="4fa94d86-57ce-4d23-95d5-4efa965f6864" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 14:11:57.775381 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:57.775335 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8" podUID="f3093abe-9b1e-492a-8837-4bd42ea39d2d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Oct 30 14:11:57.837866 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:57.837802 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" podUID="946696b9-91bd-437a-9f23-c16b73a5a9a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:58.013804 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:58.013749 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" podUID="d65dee9b-4f8b-4f09-b00b-0793d732d3ab" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:58.505963 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:11:58.505914 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:11:58.605795 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:58.556486 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" podUID="00fdf663-9bc9-4349-a565-6a8dc01c0842" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:58.624545 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:58.618764 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" Oct 30 14:11:58.624545 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:58.620100 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/distinct-2-w3-00926c-kserve-746b64d6b5-z66nw" podUID="91f37a37-70fe-4627-b2dc-77a9668d9a5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:58.624545 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:58.622602 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-00926c/distinct-2-w3-00926c-kserve-746b64d6b5-z66nw" Oct 30 14:11:58.711690 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:58.705296 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" podUID="cc3a3a28-d723-459a-93d0-ff115eac479a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded" Oct 30 14:11:58.711690 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:58.705768 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" Oct 30 14:11:58.732828 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:58.732766 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" podUID="cbd754bd-771d-4007-8ac0-7e413dc24915" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:58.739671 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:58.733232 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" Oct 30 14:11:58.768793 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:58.765701 4982 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" podUID="cc3a3a28-d723-459a-93d0-ff115eac479a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": context deadline exceeded" Oct 30 14:11:58.768793 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:58.765806 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" podUID="15e7b7f9-2570-410a-a398-4c07c2e465ea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:58.768793 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:58.766098 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" Oct 30 14:11:58.768793 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:58.766534 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" podUID="4fa94d86-57ce-4d23-95d5-4efa965f6864" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 14:11:58.768793 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:58.766785 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" Oct 30 14:11:58.768793 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:58.768056 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" podUID="ab822fe6-4349-4051-b2fd-c6e5ceb50d08" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded" Oct 30 14:11:58.768793 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:58.768359 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" Oct 30 14:11:58.788403 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:58.788360 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8" podUID="f3093abe-9b1e-492a-8837-4bd42ea39d2d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:58.789046 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:58.789018 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8" Oct 30 14:11:58.852415 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:58.852371 4982 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8" podUID="f3093abe-9b1e-492a-8837-4bd42ea39d2d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:58.853844 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:58.853813 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" podUID="946696b9-91bd-437a-9f23-c16b73a5a9a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 14:11:58.854420 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:58.854399 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" Oct 30 14:11:58.874125 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:58.874072 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:11:59.123684 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:59.107628 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" podUID="d65dee9b-4f8b-4f09-b00b-0793d732d3ab" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded" Oct 30 14:11:59.156399 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:59.156367 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" Oct 30 14:11:59.241451 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:59.241388 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/distinct-w3-00926c-kserve-774df5b58-cv8vt" podUID="b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:59.494748 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:59.493469 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" Oct 30 14:11:59.618625 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:59.596852 4982 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" podUID="ab822fe6-4349-4051-b2fd-c6e5ceb50d08" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:59.621856 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:59.621811 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/distinct-2-w3-00926c-kserve-746b64d6b5-z66nw" podUID="91f37a37-70fe-4627-b2dc-77a9668d9a5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:59.761371 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:59.761283 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-00926c/distinct-2-w3-00926c-kserve-746b64d6b5-z66nw" Oct 30 14:11:59.832139 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:59.832083 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" podUID="cbd754bd-771d-4007-8ac0-7e413dc24915" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 14:12:00.104583 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:59.877478 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" podUID="ab822fe6-4349-4051-b2fd-c6e5ceb50d08" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded" Oct 30 14:12:00.105110 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:59.888961 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" podUID="cc3a3a28-d723-459a-93d0-ff115eac479a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded" Oct 30 14:12:00.107727 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:59.925796 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" podUID="946696b9-91bd-437a-9f23-c16b73a5a9a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 14:12:00.108760 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:11:59.959471 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8" podUID="f3093abe-9b1e-492a-8837-4bd42ea39d2d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 14:12:00.126383 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:00.003719 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" Oct 30 14:12:00.126830 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:00.126802 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" Oct 30 14:12:00.127675 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:00.032979 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" podUID="4fa94d86-57ce-4d23-95d5-4efa965f6864" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 14:12:00.669780 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:00.669746 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8" Oct 30 14:12:00.670377 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:00.670358 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" Oct 30 14:12:00.670523 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:00.670510 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" Oct 30 14:12:00.670674 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:00.670660 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" Oct 30 14:12:00.670843 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:00.670830 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" Oct 30 14:12:00.670969 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:00.670957 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" Oct 30 14:12:00.963279 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:00.963148 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:02.769018 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:02.768980 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:04.734073 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:04.733951 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:06.654654 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:06.654611 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:08.463254 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:08.463218 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:09.781926 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:09.781886 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:09.957386 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:09.957347 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:10.421518 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:10.421480 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:11.990605 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:11.990570 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:12.146551 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:12.146514 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:12.332676 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:12.332621 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:12.486618 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:12.486579 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:12.731312 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:12.731202 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:12.897414 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:12.897380 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:13.053060 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:13.052920 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:13.232979 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:13.232944 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:13.824397 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:13.824363 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:14.536474 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:14.536437 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:14.795064 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:14.794931 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:16.575752 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:16.575718 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:16.886056 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:16.885953 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:17.054477 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:17.054428 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:17.305756 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:17.305676 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:17.465575 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:17.465540 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:18.175810 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:18.175762 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:18.327759 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:18.327721 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:18.942419 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:18.942380 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:20.082412 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:20.082372 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:20.222076 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:20.222041 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ea3e7b1d-6fb74b859f-kxc7w"] Oct 30 14:12:20.227377 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:20.227352 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ea3e7b1d-6fb74b859f-kxc7w" Oct 30 14:12:20.233601 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:20.233575 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:20.246013 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:20.245986 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ea3e7b1d-6fb74b859f-kxc7w"] Oct 30 14:12:20.254603 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:20.254579 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-ea3e7b1d-dockercfg-jnqhl\"" Oct 30 14:12:20.276365 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:20.276345 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-ea3e7b1d\"" Oct 30 14:12:20.286695 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:20.286666 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd6nt\" (UniqueName: \"kubernetes.io/projected/3029292f-1aa0-49ce-bda4-a033a001e53e-kube-api-access-dd6nt\") pod \"payload-pre-processing-e2e-derive-ea3e7b1d-6fb74b859f-kxc7w\" (UID: \"3029292f-1aa0-49ce-bda4-a033a001e53e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ea3e7b1d-6fb74b859f-kxc7w" Oct 30 14:12:20.287094 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:20.287069 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3029292f-1aa0-49ce-bda4-a033a001e53e-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-ea3e7b1d-6fb74b859f-kxc7w\" (UID: \"3029292f-1aa0-49ce-bda4-a033a001e53e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ea3e7b1d-6fb74b859f-kxc7w" Oct 30 14:12:20.389098 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:20.389028 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd6nt\" (UniqueName: \"kubernetes.io/projected/3029292f-1aa0-49ce-bda4-a033a001e53e-kube-api-access-dd6nt\") pod \"payload-pre-processing-e2e-derive-ea3e7b1d-6fb74b859f-kxc7w\" (UID: \"3029292f-1aa0-49ce-bda4-a033a001e53e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ea3e7b1d-6fb74b859f-kxc7w" Oct 30 14:12:20.389220 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:20.389171 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3029292f-1aa0-49ce-bda4-a033a001e53e-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-ea3e7b1d-6fb74b859f-kxc7w\" (UID: \"3029292f-1aa0-49ce-bda4-a033a001e53e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ea3e7b1d-6fb74b859f-kxc7w" Oct 30 14:12:20.390214 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:20.390187 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3029292f-1aa0-49ce-bda4-a033a001e53e-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-ea3e7b1d-6fb74b859f-kxc7w\" (UID: \"3029292f-1aa0-49ce-bda4-a033a001e53e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ea3e7b1d-6fb74b859f-kxc7w" Oct 30 14:12:20.423939 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:20.423909 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd6nt\" (UniqueName: \"kubernetes.io/projected/3029292f-1aa0-49ce-bda4-a033a001e53e-kube-api-access-dd6nt\") pod \"payload-pre-processing-e2e-derive-ea3e7b1d-6fb74b859f-kxc7w\" (UID: \"3029292f-1aa0-49ce-bda4-a033a001e53e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ea3e7b1d-6fb74b859f-kxc7w" Oct 30 14:12:20.532266 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:20.532169 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:20.545999 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:20.545965 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ea3e7b1d-6fb74b859f-kxc7w" Oct 30 14:12:20.888537 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:20.888506 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ea3e7b1d-6fb74b859f-kxc7w"] Oct 30 14:12:20.889068 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:12:20.889037 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3029292f_1aa0_49ce_bda4_a033a001e53e.slice/crio-074a1cac2da214a20f8bf8afcf911ef1059c2e44c1a0d77194466aab56a9821e WatchSource:0}: Error finding container 074a1cac2da214a20f8bf8afcf911ef1059c2e44c1a0d77194466aab56a9821e: Status 404 returned error can't find the container with id 074a1cac2da214a20f8bf8afcf911ef1059c2e44c1a0d77194466aab56a9821e Oct 30 14:12:20.891591 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:20.891567 4982 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 14:12:20.998772 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:20.998669 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:21.579118 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:21.579042 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:21.742122 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:21.742084 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:21.833975 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:21.833898 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ea3e7b1d-6fb74b859f-kxc7w" event={"ID":"3029292f-1aa0-49ce-bda4-a033a001e53e","Type":"ContainerStarted","Data":"074a1cac2da214a20f8bf8afcf911ef1059c2e44c1a0d77194466aab56a9821e"} Oct 30 14:12:23.497665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:23.497502 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" podUID="00fdf663-9bc9-4349-a565-6a8dc01c0842" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:12:23.575665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:23.575204 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" podUID="15e7b7f9-2570-410a-a398-4c07c2e465ea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:12:23.582670 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:23.579372 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" podUID="4fa94d86-57ce-4d23-95d5-4efa965f6864" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:12:23.614712 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:23.611924 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:23.710672 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:23.710455 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" podUID="946696b9-91bd-437a-9f23-c16b73a5a9a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 14:12:23.977662 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:23.976835 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" podUID="d65dee9b-4f8b-4f09-b00b-0793d732d3ab" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:12:24.115675 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:24.112844 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8" podUID="f3093abe-9b1e-492a-8837-4bd42ea39d2d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:12:24.115675 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:24.113154 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" podUID="cbd754bd-771d-4007-8ac0-7e413dc24915" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 14:12:24.179674 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:24.179157 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/distinct-w3-00926c-kserve-774df5b58-cv8vt" podUID="b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:12:24.727319 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:24.724028 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" podUID="946696b9-91bd-437a-9f23-c16b73a5a9a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 14:12:24.727319 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:24.724732 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" Oct 30 14:12:24.919786 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:24.919363 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" Oct 30 14:12:25.742378 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:25.742343 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:25.958043 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:25.957945 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ea3e7b1d-6fb74b859f-kxc7w" event={"ID":"3029292f-1aa0-49ce-bda4-a033a001e53e","Type":"ContainerStarted","Data":"6e3609571012334c0245ebd1748bf8959fd0cbbd5150a3f6a7a4f6a1444d7e80"} Oct 30 14:12:25.958586 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:25.958567 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-ea3e7b1d-6fb74b859f-kxc7w" Oct 30 14:12:26.006339 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:26.006282 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-ea3e7b1d-6fb74b859f-kxc7w" podStartSLOduration=1.239807881 podStartE2EDuration="6.006267264s" totalImagesPullingTime="4.766459383s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:12:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:12:26.002559896 +0000 UTC m=+1415.746693951" watchObservedRunningTime="2026-10-30 14:12:26.006267264 +0000 UTC m=+1415.750401328" Oct 30 14:12:27.557540 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:27.557504 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:27.700174 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:27.700140 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:27.990998 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:27.990898 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:28.014364 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:12:28.014329 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:12:28.407388 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:28.407353 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-25917498-openshift-default-77dc9bb847-smvqn"] Oct 30 14:12:28.412541 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:28.412515 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-25917498-openshift-default-77dc9bb847-smvqn" Oct 30 14:12:28.435397 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:28.435367 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-25917498-openshift-default-dockercfg-rfpxz\"" Oct 30 14:12:28.452367 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:28.452344 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-25917498-openshift-default-77dc9bb847-smvqn"] Oct 30 14:12:28.497928 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:28.497895 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8104c709-90b5-4c71-9540-a766b2add736-istio-podinfo\") pod \"e2e-ait-25917498-openshift-default-77dc9bb847-smvqn\" (UID: \"8104c709-90b5-4c71-9540-a766b2add736\") " pod="openshift-ingress/e2e-ait-25917498-openshift-default-77dc9bb847-smvqn" Oct 30 14:12:28.497928 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:28.497937 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhtfn\" (UniqueName: \"kubernetes.io/projected/8104c709-90b5-4c71-9540-a766b2add736-kube-api-access-jhtfn\") pod \"e2e-ait-25917498-openshift-default-77dc9bb847-smvqn\" (UID: \"8104c709-90b5-4c71-9540-a766b2add736\") " pod="openshift-ingress/e2e-ait-25917498-openshift-default-77dc9bb847-smvqn" Oct 30 14:12:28.498166 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:28.497999 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8104c709-90b5-4c71-9540-a766b2add736-istio-envoy\") pod \"e2e-ait-25917498-openshift-default-77dc9bb847-smvqn\" (UID: \"8104c709-90b5-4c71-9540-a766b2add736\") " pod="openshift-ingress/e2e-ait-25917498-openshift-default-77dc9bb847-smvqn" Oct 30 14:12:28.498166 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:28.498085 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8104c709-90b5-4c71-9540-a766b2add736-istio-token\") pod \"e2e-ait-25917498-openshift-default-77dc9bb847-smvqn\" (UID: \"8104c709-90b5-4c71-9540-a766b2add736\") " pod="openshift-ingress/e2e-ait-25917498-openshift-default-77dc9bb847-smvqn" Oct 30 14:12:28.498166 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:28.498143 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8104c709-90b5-4c71-9540-a766b2add736-istiod-ca-cert\") pod \"e2e-ait-25917498-openshift-default-77dc9bb847-smvqn\" (UID: \"8104c709-90b5-4c71-9540-a766b2add736\") " pod="openshift-ingress/e2e-ait-25917498-openshift-default-77dc9bb847-smvqn" Oct 30 14:12:28.498324 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:28.498193 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8104c709-90b5-4c71-9540-a766b2add736-workload-certs\") pod \"e2e-ait-25917498-openshift-default-77dc9bb847-smvqn\" (UID: \"8104c709-90b5-4c71-9540-a766b2add736\") " pod="openshift-ingress/e2e-ait-25917498-openshift-default-77dc9bb847-smvqn" Oct 30 14:12:28.498324 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:28.498228 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8104c709-90b5-4c71-9540-a766b2add736-istio-data\") pod \"e2e-ait-25917498-openshift-default-77dc9bb847-smvqn\" (UID: \"8104c709-90b5-4c71-9540-a766b2add736\") " pod="openshift-ingress/e2e-ait-25917498-openshift-default-77dc9bb847-smvqn" Oct 30 14:12:28.498324 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:28.498282 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8104c709-90b5-4c71-9540-a766b2add736-workload-socket\") pod \"e2e-ait-25917498-openshift-default-77dc9bb847-smvqn\" (UID: \"8104c709-90b5-4c71-9540-a766b2add736\") " pod="openshift-ingress/e2e-ait-25917498-openshift-default-77dc9bb847-smvqn" Oct 30 14:12:28.498474 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:28.498336 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8104c709-90b5-4c71-9540-a766b2add736-credential-socket\") pod \"e2e-ait-25917498-openshift-default-77dc9bb847-smvqn\" (UID: \"8104c709-90b5-4c71-9540-a766b2add736\") " pod="openshift-ingress/e2e-ait-25917498-openshift-default-77dc9bb847-smvqn" Oct 30 14:12:28.509080 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:28.509027 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:28.600792 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:28.600753 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8104c709-90b5-4c71-9540-a766b2add736-istio-podinfo\") pod \"e2e-ait-25917498-openshift-default-77dc9bb847-smvqn\" (UID: \"8104c709-90b5-4c71-9540-a766b2add736\") " pod="openshift-ingress/e2e-ait-25917498-openshift-default-77dc9bb847-smvqn" Oct 30 14:12:28.613370 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:28.613337 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhtfn\" (UniqueName: \"kubernetes.io/projected/8104c709-90b5-4c71-9540-a766b2add736-kube-api-access-jhtfn\") pod \"e2e-ait-25917498-openshift-default-77dc9bb847-smvqn\" (UID: \"8104c709-90b5-4c71-9540-a766b2add736\") " pod="openshift-ingress/e2e-ait-25917498-openshift-default-77dc9bb847-smvqn" Oct 30 14:12:28.613615 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:28.613597 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8104c709-90b5-4c71-9540-a766b2add736-istio-envoy\") pod \"e2e-ait-25917498-openshift-default-77dc9bb847-smvqn\" (UID: \"8104c709-90b5-4c71-9540-a766b2add736\") " pod="openshift-ingress/e2e-ait-25917498-openshift-default-77dc9bb847-smvqn" Oct 30 14:12:28.613871 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:28.613850 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8104c709-90b5-4c71-9540-a766b2add736-istio-token\") pod \"e2e-ait-25917498-openshift-default-77dc9bb847-smvqn\" (UID: \"8104c709-90b5-4c71-9540-a766b2add736\") " pod="openshift-ingress/e2e-ait-25917498-openshift-default-77dc9bb847-smvqn" Oct 30 14:12:28.614049 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:28.614034 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8104c709-90b5-4c71-9540-a766b2add736-istiod-ca-cert\") pod \"e2e-ait-25917498-openshift-default-77dc9bb847-smvqn\" (UID: \"8104c709-90b5-4c71-9540-a766b2add736\") " pod="openshift-ingress/e2e-ait-25917498-openshift-default-77dc9bb847-smvqn" Oct 30 14:12:28.614190 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:28.614173 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8104c709-90b5-4c71-9540-a766b2add736-workload-certs\") pod \"e2e-ait-25917498-openshift-default-77dc9bb847-smvqn\" (UID: \"8104c709-90b5-4c71-9540-a766b2add736\") " pod="openshift-ingress/e2e-ait-25917498-openshift-default-77dc9bb847-smvqn" Oct 30 14:12:28.614310 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:28.614291 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8104c709-90b5-4c71-9540-a766b2add736-istio-data\") pod \"e2e-ait-25917498-openshift-default-77dc9bb847-smvqn\" (UID: \"8104c709-90b5-4c71-9540-a766b2add736\") " pod="openshift-ingress/e2e-ait-25917498-openshift-default-77dc9bb847-smvqn" Oct 30 14:12:28.614452 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:28.614436 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8104c709-90b5-4c71-9540-a766b2add736-workload-socket\") pod \"e2e-ait-25917498-openshift-default-77dc9bb847-smvqn\" (UID: \"8104c709-90b5-4c71-9540-a766b2add736\") " pod="openshift-ingress/e2e-ait-25917498-openshift-default-77dc9bb847-smvqn" Oct 30 14:12:28.632792 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:28.617826 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8104c709-90b5-4c71-9540-a766b2add736-workload-certs\") pod \"e2e-ait-25917498-openshift-default-77dc9bb847-smvqn\" (UID: \"8104c709-90b5-4c71-9540-a766b2add736\") " pod="openshift-ingress/e2e-ait-25917498-openshift-default-77dc9bb847-smvqn" Oct 30 14:12:28.633010 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:28.618474 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8104c709-90b5-4c71-9540-a766b2add736-istiod-ca-cert\") pod \"e2e-ait-25917498-openshift-default-77dc9bb847-smvqn\" (UID: \"8104c709-90b5-4c71-9540-a766b2add736\") " pod="openshift-ingress/e2e-ait-25917498-openshift-default-77dc9bb847-smvqn" Oct 30 14:12:28.633131 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:28.618710 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8104c709-90b5-4c71-9540-a766b2add736-istio-data\") pod \"e2e-ait-25917498-openshift-default-77dc9bb847-smvqn\" (UID: \"8104c709-90b5-4c71-9540-a766b2add736\") " pod="openshift-ingress/e2e-ait-25917498-openshift-default-77dc9bb847-smvqn" Oct 30 14:12:28.633245 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:28.618929 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8104c709-90b5-4c71-9540-a766b2add736-workload-socket\") pod \"e2e-ait-25917498-openshift-default-77dc9bb847-smvqn\" (UID: \"8104c709-90b5-4c71-9540-a766b2add736\") " pod="openshift-ingress/e2e-ait-25917498-openshift-default-77dc9bb847-smvqn" Oct 30 14:12:28.633357 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:28.632480 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8104c709-90b5-4c71-9540-a766b2add736-istio-podinfo\") pod \"e2e-ait-25917498-openshift-default-77dc9bb847-smvqn\" (UID: \"8104c709-90b5-4c71-9540-a766b2add736\") " pod="openshift-ingress/e2e-ait-25917498-openshift-default-77dc9bb847-smvqn" Oct 30 14:12:28.633507 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:28.633487 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8104c709-90b5-4c71-9540-a766b2add736-credential-socket\") pod \"e2e-ait-25917498-openshift-default-77dc9bb847-smvqn\" (UID: \"8104c709-90b5-4c71-9540-a766b2add736\") " pod="openshift-ingress/e2e-ait-25917498-openshift-default-77dc9bb847-smvqn" Oct 30 14:12:28.640713 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:28.640683 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8104c709-90b5-4c71-9540-a766b2add736-credential-socket\") pod \"e2e-ait-25917498-openshift-default-77dc9bb847-smvqn\" (UID: \"8104c709-90b5-4c71-9540-a766b2add736\") " pod="openshift-ingress/e2e-ait-25917498-openshift-default-77dc9bb847-smvqn" Oct 30 14:12:28.656988 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:28.656704 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhtfn\" (UniqueName: \"kubernetes.io/projected/8104c709-90b5-4c71-9540-a766b2add736-kube-api-access-jhtfn\") pod \"e2e-ait-25917498-openshift-default-77dc9bb847-smvqn\" (UID: \"8104c709-90b5-4c71-9540-a766b2add736\") " pod="openshift-ingress/e2e-ait-25917498-openshift-default-77dc9bb847-smvqn" Oct 30 14:12:28.656988 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:28.656939 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8104c709-90b5-4c71-9540-a766b2add736-istio-envoy\") pod \"e2e-ait-25917498-openshift-default-77dc9bb847-smvqn\" (UID: \"8104c709-90b5-4c71-9540-a766b2add736\") " pod="openshift-ingress/e2e-ait-25917498-openshift-default-77dc9bb847-smvqn" Oct 30 14:12:28.658701 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:28.658580 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:28.661405 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:28.661384 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8104c709-90b5-4c71-9540-a766b2add736-istio-token\") pod \"e2e-ait-25917498-openshift-default-77dc9bb847-smvqn\" (UID: \"8104c709-90b5-4c71-9540-a766b2add736\") " pod="openshift-ingress/e2e-ait-25917498-openshift-default-77dc9bb847-smvqn" Oct 30 14:12:28.732947 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:28.732909 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-25917498-openshift-default-77dc9bb847-smvqn" Oct 30 14:12:29.132829 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:29.132801 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-25917498-openshift-default-77dc9bb847-smvqn"] Oct 30 14:12:29.133460 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:12:29.133434 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8104c709_90b5_4c71_9540_a766b2add736.slice/crio-d0bb23cfedcb7891fbc6fb93fbf134f11b4c2d7231baab311a24f04fdb2b40eb WatchSource:0}: Error finding container d0bb23cfedcb7891fbc6fb93fbf134f11b4c2d7231baab311a24f04fdb2b40eb: Status 404 returned error can't find the container with id d0bb23cfedcb7891fbc6fb93fbf134f11b4c2d7231baab311a24f04fdb2b40eb Oct 30 14:12:29.141984 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:29.141948 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:12:29.142090 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:29.142039 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:12:29.142090 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:29.142084 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:12:29.427457 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:29.427378 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:30.023286 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:30.023250 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:30.047624 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:30.047586 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-25917498-openshift-default-77dc9bb847-smvqn" event={"ID":"8104c709-90b5-4c71-9540-a766b2add736","Type":"ContainerStarted","Data":"e5f5e55f4a55842ed1cc2b3e9fa9f75aa58b37706b78d0b3ae43c10f6900c918"} Oct 30 14:12:30.047789 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:30.047665 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-25917498-openshift-default-77dc9bb847-smvqn" event={"ID":"8104c709-90b5-4c71-9540-a766b2add736","Type":"ContainerStarted","Data":"d0bb23cfedcb7891fbc6fb93fbf134f11b4c2d7231baab311a24f04fdb2b40eb"} Oct 30 14:12:30.131119 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:30.131064 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-25917498-openshift-default-77dc9bb847-smvqn" podStartSLOduration=2.13105075 podStartE2EDuration="2.13105075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:12:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:12:30.113054876 +0000 UTC m=+1419.857188920" watchObservedRunningTime="2026-10-30 14:12:30.13105075 +0000 UTC m=+1419.875184791" Oct 30 14:12:30.174003 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:30.173974 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:30.326522 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:30.326373 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:30.734397 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:30.734313 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-25917498-openshift-default-77dc9bb847-smvqn" Oct 30 14:12:30.739612 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:30.739586 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-25917498-openshift-default-77dc9bb847-smvqn" Oct 30 14:12:31.053475 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:31.053397 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-25917498-openshift-default-77dc9bb847-smvqn" Oct 30 14:12:31.054321 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:31.054303 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-25917498-openshift-default-77dc9bb847-smvqn" Oct 30 14:12:31.374769 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:31.374726 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:31.542544 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:31.542507 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:31.804167 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:31.804026 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:31.976821 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:31.976779 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:32.148908 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:32.148856 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:33.705623 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:33.705589 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:33.772592 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:33.768371 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ea3e7b1d-66757797d5-54tg9"] Oct 30 14:12:33.816912 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:33.816875 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ea3e7b1d-66757797d5-54tg9"] Oct 30 14:12:33.817223 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:33.817200 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea3e7b1d-66757797d5-54tg9" Oct 30 14:12:33.851488 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:33.851457 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:33.880520 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:33.880487 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-ea3e7b1d-serving-cert\"" Oct 30 14:12:33.927590 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:33.927559 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa0a51b1-d2bd-4321-acfe-ddb5ccd955ae-maas-api-tls\") pod \"maas-api-e2e-derive-ea3e7b1d-66757797d5-54tg9\" (UID: \"aa0a51b1-d2bd-4321-acfe-ddb5ccd955ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea3e7b1d-66757797d5-54tg9" Oct 30 14:12:33.927812 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:33.927748 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa0a51b1-d2bd-4321-acfe-ddb5ccd955ae-metrics-certs\") pod \"maas-api-e2e-derive-ea3e7b1d-66757797d5-54tg9\" (UID: \"aa0a51b1-d2bd-4321-acfe-ddb5ccd955ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea3e7b1d-66757797d5-54tg9" Oct 30 14:12:33.927881 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:33.927862 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5vvh\" (UniqueName: \"kubernetes.io/projected/aa0a51b1-d2bd-4321-acfe-ddb5ccd955ae-kube-api-access-x5vvh\") pod \"maas-api-e2e-derive-ea3e7b1d-66757797d5-54tg9\" (UID: \"aa0a51b1-d2bd-4321-acfe-ddb5ccd955ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea3e7b1d-66757797d5-54tg9" Oct 30 14:12:33.927936 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:33.927906 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa0a51b1-d2bd-4321-acfe-ddb5ccd955ae-openshift-service-ca\") pod \"maas-api-e2e-derive-ea3e7b1d-66757797d5-54tg9\" (UID: \"aa0a51b1-d2bd-4321-acfe-ddb5ccd955ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea3e7b1d-66757797d5-54tg9" Oct 30 14:12:33.996005 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:33.995919 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:34.029477 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:34.029443 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5vvh\" (UniqueName: \"kubernetes.io/projected/aa0a51b1-d2bd-4321-acfe-ddb5ccd955ae-kube-api-access-x5vvh\") pod \"maas-api-e2e-derive-ea3e7b1d-66757797d5-54tg9\" (UID: \"aa0a51b1-d2bd-4321-acfe-ddb5ccd955ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea3e7b1d-66757797d5-54tg9" Oct 30 14:12:34.029659 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:34.029532 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa0a51b1-d2bd-4321-acfe-ddb5ccd955ae-openshift-service-ca\") pod \"maas-api-e2e-derive-ea3e7b1d-66757797d5-54tg9\" (UID: \"aa0a51b1-d2bd-4321-acfe-ddb5ccd955ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea3e7b1d-66757797d5-54tg9" Oct 30 14:12:34.029745 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:34.029699 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa0a51b1-d2bd-4321-acfe-ddb5ccd955ae-maas-api-tls\") pod \"maas-api-e2e-derive-ea3e7b1d-66757797d5-54tg9\" (UID: \"aa0a51b1-d2bd-4321-acfe-ddb5ccd955ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea3e7b1d-66757797d5-54tg9" Oct 30 14:12:34.029871 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:34.029854 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa0a51b1-d2bd-4321-acfe-ddb5ccd955ae-metrics-certs\") pod \"maas-api-e2e-derive-ea3e7b1d-66757797d5-54tg9\" (UID: \"aa0a51b1-d2bd-4321-acfe-ddb5ccd955ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea3e7b1d-66757797d5-54tg9" Oct 30 14:12:34.030363 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:34.030329 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa0a51b1-d2bd-4321-acfe-ddb5ccd955ae-openshift-service-ca\") pod \"maas-api-e2e-derive-ea3e7b1d-66757797d5-54tg9\" (UID: \"aa0a51b1-d2bd-4321-acfe-ddb5ccd955ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea3e7b1d-66757797d5-54tg9" Oct 30 14:12:34.032172 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:34.032143 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa0a51b1-d2bd-4321-acfe-ddb5ccd955ae-maas-api-tls\") pod \"maas-api-e2e-derive-ea3e7b1d-66757797d5-54tg9\" (UID: \"aa0a51b1-d2bd-4321-acfe-ddb5ccd955ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea3e7b1d-66757797d5-54tg9" Oct 30 14:12:34.032274 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:34.032174 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa0a51b1-d2bd-4321-acfe-ddb5ccd955ae-metrics-certs\") pod \"maas-api-e2e-derive-ea3e7b1d-66757797d5-54tg9\" (UID: \"aa0a51b1-d2bd-4321-acfe-ddb5ccd955ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea3e7b1d-66757797d5-54tg9" Oct 30 14:12:34.056535 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:34.056507 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5vvh\" (UniqueName: \"kubernetes.io/projected/aa0a51b1-d2bd-4321-acfe-ddb5ccd955ae-kube-api-access-x5vvh\") pod \"maas-api-e2e-derive-ea3e7b1d-66757797d5-54tg9\" (UID: \"aa0a51b1-d2bd-4321-acfe-ddb5ccd955ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea3e7b1d-66757797d5-54tg9" Oct 30 14:12:34.145512 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:34.145470 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:34.145780 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:34.145752 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea3e7b1d-66757797d5-54tg9" Oct 30 14:12:34.476815 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:12:34.476777 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa0a51b1_d2bd_4321_acfe_ddb5ccd955ae.slice/crio-0568471c5045b375f0785b023bda120fdc4c8838003dfa6be25c47d44b613b8f WatchSource:0}: Error finding container 0568471c5045b375f0785b023bda120fdc4c8838003dfa6be25c47d44b613b8f: Status 404 returned error can't find the container with id 0568471c5045b375f0785b023bda120fdc4c8838003dfa6be25c47d44b613b8f Oct 30 14:12:34.479054 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:34.479011 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ea3e7b1d-66757797d5-54tg9"] Oct 30 14:12:35.086882 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:35.086848 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea3e7b1d-66757797d5-54tg9" event={"ID":"aa0a51b1-d2bd-4321-acfe-ddb5ccd955ae","Type":"ContainerStarted","Data":"ed7497714e2a9f1582e0bb2026279641c57413d640da35d42050a9ea9ce3b9d8"} Oct 30 14:12:35.087310 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:35.086909 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea3e7b1d-66757797d5-54tg9" event={"ID":"aa0a51b1-d2bd-4321-acfe-ddb5ccd955ae","Type":"ContainerStarted","Data":"0568471c5045b375f0785b023bda120fdc4c8838003dfa6be25c47d44b613b8f"} Oct 30 14:12:35.087310 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:35.086954 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea3e7b1d-66757797d5-54tg9" Oct 30 14:12:35.133743 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:35.133683 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea3e7b1d-66757797d5-54tg9" podStartSLOduration=1.684869328 podStartE2EDuration="2.13366545s" totalImagesPullingTime="448.796122ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:12:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:12:35.132960206 +0000 UTC m=+1424.877094247" watchObservedRunningTime="2026-10-30 14:12:35.13366545 +0000 UTC m=+1424.877799494" Oct 30 14:12:35.838817 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:35.838780 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:35.980895 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:35.980865 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:36.131271 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:36.131190 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:36.969509 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:36.969477 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-ea3e7b1d-6fb74b859f-kxc7w" Oct 30 14:12:37.738983 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:37.738882 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:38.976653 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:38.976604 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:39.126376 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:39.126337 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:39.301941 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:39.301867 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:39.846234 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:39.846199 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:40.006843 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:40.006807 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:41.104096 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:41.104064 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea3e7b1d-66757797d5-54tg9" Oct 30 14:12:41.741137 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:41.741096 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:42.871570 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:42.871532 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:43.620500 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:43.620463 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:44.171839 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:44.171800 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:45.060132 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:45.060079 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:45.658697 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:45.658611 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:47.218712 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:47.218633 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:47.488898 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:47.488808 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:48.204145 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:48.204105 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:49.373810 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:49.373755 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:51.690803 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:51.690715 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:52.050560 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:52.049759 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" podUID="d65dee9b-4f8b-4f09-b00b-0793d732d3ab" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 14:12:52.571671 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:52.568449 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" podUID="4fa94d86-57ce-4d23-95d5-4efa965f6864" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:12:53.050825 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:53.050718 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" podUID="d65dee9b-4f8b-4f09-b00b-0793d732d3ab" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:12:53.052038 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:53.052013 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" Oct 30 14:12:53.622664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:53.597797 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" podUID="cc3a3a28-d723-459a-93d0-ff115eac479a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 14:12:53.662670 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:53.634093 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" podUID="4fa94d86-57ce-4d23-95d5-4efa965f6864" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:12:53.662670 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:53.634520 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" Oct 30 14:12:53.700576 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:53.700531 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:53.732026 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:53.731989 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-25917498-7b65854ffc-57t5n"] Oct 30 14:12:53.811267 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:53.811224 4982 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-00926c/distinct-2-w3-00926c-kserve-746b64d6b5-z66nw" podUID="91f37a37-70fe-4627-b2dc-77a9668d9a5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 14:12:53.845671 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:53.832800 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/distinct-2-w3-00926c-kserve-746b64d6b5-z66nw" podUID="91f37a37-70fe-4627-b2dc-77a9668d9a5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 14:12:53.989730 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:53.920719 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:53.989730 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:53.930894 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" podUID="946696b9-91bd-437a-9f23-c16b73a5a9a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:12:53.990160 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:53.990131 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-25917498-7b65854ffc-57t5n"] Oct 30 14:12:53.990379 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:53.990364 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-25917498-7b65854ffc-57t5n" Oct 30 14:12:54.015665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:54.013772 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-25917498-serving-cert\"" Oct 30 14:12:54.042452 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:54.037799 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b12b40f-da8a-46d3-a34f-9a1bb0420656-maas-api-tls\") pod \"maas-api-e2e-ait-25917498-7b65854ffc-57t5n\" (UID: \"3b12b40f-da8a-46d3-a34f-9a1bb0420656\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-25917498-7b65854ffc-57t5n" Oct 30 14:12:54.042452 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:54.037881 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b12b40f-da8a-46d3-a34f-9a1bb0420656-metrics-certs\") pod \"maas-api-e2e-ait-25917498-7b65854ffc-57t5n\" (UID: \"3b12b40f-da8a-46d3-a34f-9a1bb0420656\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-25917498-7b65854ffc-57t5n" Oct 30 14:12:54.042452 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:54.037947 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wdfw\" (UniqueName: \"kubernetes.io/projected/3b12b40f-da8a-46d3-a34f-9a1bb0420656-kube-api-access-5wdfw\") pod \"maas-api-e2e-ait-25917498-7b65854ffc-57t5n\" (UID: \"3b12b40f-da8a-46d3-a34f-9a1bb0420656\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-25917498-7b65854ffc-57t5n" Oct 30 14:12:54.042452 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:54.038152 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b12b40f-da8a-46d3-a34f-9a1bb0420656-openshift-service-ca\") pod \"maas-api-e2e-ait-25917498-7b65854ffc-57t5n\" (UID: \"3b12b40f-da8a-46d3-a34f-9a1bb0420656\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-25917498-7b65854ffc-57t5n" Oct 30 14:12:54.051777 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:54.051730 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" podUID="d65dee9b-4f8b-4f09-b00b-0793d732d3ab" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:12:54.076979 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:54.076938 4982 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" podUID="15e7b7f9-2570-410a-a398-4c07c2e465ea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:12:54.121834 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:54.121788 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" podUID="cbd754bd-771d-4007-8ac0-7e413dc24915" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:12:54.123258 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:54.123208 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" podUID="ab822fe6-4349-4051-b2fd-c6e5ceb50d08" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded" Oct 30 14:12:54.148995 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:54.123475 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8" podUID="f3093abe-9b1e-492a-8837-4bd42ea39d2d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Oct 30 14:12:54.149470 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:54.147722 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b12b40f-da8a-46d3-a34f-9a1bb0420656-openshift-service-ca\") pod \"maas-api-e2e-ait-25917498-7b65854ffc-57t5n\" (UID: \"3b12b40f-da8a-46d3-a34f-9a1bb0420656\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-25917498-7b65854ffc-57t5n" Oct 30 14:12:54.149794 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:54.149769 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b12b40f-da8a-46d3-a34f-9a1bb0420656-maas-api-tls\") pod \"maas-api-e2e-ait-25917498-7b65854ffc-57t5n\" (UID: \"3b12b40f-da8a-46d3-a34f-9a1bb0420656\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-25917498-7b65854ffc-57t5n" Oct 30 14:12:54.150174 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:54.150138 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b12b40f-da8a-46d3-a34f-9a1bb0420656-metrics-certs\") pod \"maas-api-e2e-ait-25917498-7b65854ffc-57t5n\" (UID: \"3b12b40f-da8a-46d3-a34f-9a1bb0420656\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-25917498-7b65854ffc-57t5n" Oct 30 14:12:54.150399 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:54.150376 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wdfw\" (UniqueName: \"kubernetes.io/projected/3b12b40f-da8a-46d3-a34f-9a1bb0420656-kube-api-access-5wdfw\") pod \"maas-api-e2e-ait-25917498-7b65854ffc-57t5n\" (UID: \"3b12b40f-da8a-46d3-a34f-9a1bb0420656\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-25917498-7b65854ffc-57t5n" Oct 30 14:12:54.151024 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:54.148593 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b12b40f-da8a-46d3-a34f-9a1bb0420656-openshift-service-ca\") pod \"maas-api-e2e-ait-25917498-7b65854ffc-57t5n\" (UID: \"3b12b40f-da8a-46d3-a34f-9a1bb0420656\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-25917498-7b65854ffc-57t5n" Oct 30 14:12:54.154726 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:12:54.150052 4982 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-25917498-serving-cert: secret "maas-api-e2e-ait-25917498-serving-cert" not found Oct 30 14:12:54.154922 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:12:54.154909 4982 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b12b40f-da8a-46d3-a34f-9a1bb0420656-maas-api-tls podName:3b12b40f-da8a-46d3-a34f-9a1bb0420656 nodeName:}" failed. No retries permitted until 2026-10-30 14:12:54.654885143 +0000 UTC m=+1444.399019170 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3b12b40f-da8a-46d3-a34f-9a1bb0420656-maas-api-tls") pod "maas-api-e2e-ait-25917498-7b65854ffc-57t5n" (UID: "3b12b40f-da8a-46d3-a34f-9a1bb0420656") : secret "maas-api-e2e-ait-25917498-serving-cert" not found Oct 30 14:12:54.175326 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:54.175295 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b12b40f-da8a-46d3-a34f-9a1bb0420656-metrics-certs\") pod \"maas-api-e2e-ait-25917498-7b65854ffc-57t5n\" (UID: \"3b12b40f-da8a-46d3-a34f-9a1bb0420656\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-25917498-7b65854ffc-57t5n" Oct 30 14:12:54.187657 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:54.185696 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/distinct-w3-00926c-kserve-774df5b58-cv8vt" podUID="b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:12:54.309706 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:54.298274 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wdfw\" (UniqueName: \"kubernetes.io/projected/3b12b40f-da8a-46d3-a34f-9a1bb0420656-kube-api-access-5wdfw\") pod \"maas-api-e2e-ait-25917498-7b65854ffc-57t5n\" (UID: \"3b12b40f-da8a-46d3-a34f-9a1bb0420656\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-25917498-7b65854ffc-57t5n" Oct 30 14:12:54.531661 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:54.497870 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" podUID="00fdf663-9bc9-4349-a565-6a8dc01c0842" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:12:54.626245 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:54.567716 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" podUID="15e7b7f9-2570-410a-a398-4c07c2e465ea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded" Oct 30 14:12:54.627022 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:54.618847 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" podUID="cc3a3a28-d723-459a-93d0-ff115eac479a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded" Oct 30 14:12:54.681663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:54.661713 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" podUID="4fa94d86-57ce-4d23-95d5-4efa965f6864" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:12:54.681663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:54.661759 4982 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" podUID="00fdf663-9bc9-4349-a565-6a8dc01c0842" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": context deadline exceeded" Oct 30 14:12:54.765667 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:54.739804 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b12b40f-da8a-46d3-a34f-9a1bb0420656-maas-api-tls\") pod \"maas-api-e2e-ait-25917498-7b65854ffc-57t5n\" (UID: \"3b12b40f-da8a-46d3-a34f-9a1bb0420656\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-25917498-7b65854ffc-57t5n" Oct 30 14:12:54.809670 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:12:54.766195 4982 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-25917498-serving-cert: secret "maas-api-e2e-ait-25917498-serving-cert" not found Oct 30 14:12:54.809994 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:12:54.809974 4982 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b12b40f-da8a-46d3-a34f-9a1bb0420656-maas-api-tls podName:3b12b40f-da8a-46d3-a34f-9a1bb0420656 nodeName:}" failed. No retries permitted until 2026-10-30 14:12:55.809945792 +0000 UTC m=+1445.554079826 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3b12b40f-da8a-46d3-a34f-9a1bb0420656-maas-api-tls") pod "maas-api-e2e-ait-25917498-7b65854ffc-57t5n" (UID: "3b12b40f-da8a-46d3-a34f-9a1bb0420656") : secret "maas-api-e2e-ait-25917498-serving-cert" not found Oct 30 14:12:54.924718 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:54.891957 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/distinct-2-w3-00926c-kserve-746b64d6b5-z66nw" podUID="91f37a37-70fe-4627-b2dc-77a9668d9a5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 30 14:12:54.929709 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:54.929671 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" Oct 30 14:12:54.954795 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:54.954761 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-00926c/distinct-2-w3-00926c-kserve-746b64d6b5-z66nw" Oct 30 14:12:54.955691 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:54.941541 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" podUID="946696b9-91bd-437a-9f23-c16b73a5a9a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 14:12:54.957138 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:54.957113 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" Oct 30 14:12:55.013676 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:55.012940 4982 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-00926c/distinct-w3-00926c-kserve-774df5b58-cv8vt" podUID="b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": context deadline exceeded" Oct 30 14:12:55.092662 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:55.092332 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" podUID="d65dee9b-4f8b-4f09-b00b-0793d732d3ab" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:12:55.204717 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:55.202095 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8" podUID="f3093abe-9b1e-492a-8837-4bd42ea39d2d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Oct 30 14:12:55.204717 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:55.202523 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8" Oct 30 14:12:55.231044 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:55.208009 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" podUID="cbd754bd-771d-4007-8ac0-7e413dc24915" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:12:55.231044 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:55.208078 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" podUID="ab822fe6-4349-4051-b2fd-c6e5ceb50d08" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded" Oct 30 14:12:55.231044 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:55.208357 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" Oct 30 14:12:55.231044 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:55.208406 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" Oct 30 14:12:55.231044 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:55.213911 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" Oct 30 14:12:55.259886 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:55.259851 4982 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:12:55.260182 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:55.260147 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp" podUID="e3ffe590-102f-420e-b108-3d6ea542c0b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:12:55.330151 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:55.330120 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:55.367329 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:55.367297 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" Oct 30 14:12:55.404117 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:55.404072 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" Oct 30 14:12:55.662114 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:55.662071 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" podUID="15e7b7f9-2570-410a-a398-4c07c2e465ea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:12:55.684664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:55.677374 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" podUID="00fdf663-9bc9-4349-a565-6a8dc01c0842" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 14:12:55.709675 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:55.681790 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" Oct 30 14:12:55.709675 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:55.687113 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" Oct 30 14:12:55.709675 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:55.687157 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" Oct 30 14:12:55.709675 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:55.682694 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" podUID="cc3a3a28-d723-459a-93d0-ff115eac479a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:12:55.733110 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:55.733079 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" podUID="4fa94d86-57ce-4d23-95d5-4efa965f6864" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 14:12:55.841462 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:55.821568 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b12b40f-da8a-46d3-a34f-9a1bb0420656-maas-api-tls\") pod \"maas-api-e2e-ait-25917498-7b65854ffc-57t5n\" (UID: \"3b12b40f-da8a-46d3-a34f-9a1bb0420656\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-25917498-7b65854ffc-57t5n" Oct 30 14:12:55.842358 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:55.842331 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" Oct 30 14:12:55.842560 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:55.842547 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" Oct 30 14:12:55.884264 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:55.875302 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v"] Oct 30 14:12:55.884264 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:55.875511 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v" podUID="68c5e59c-004d-4751-bed1-f7943e0548fd" containerName="istio-proxy" containerID="cri-o://57dee7f0140bb9eb6c64184aa7957d4c5fa8b3dd914b28e44e360302195b4983" gracePeriod=30 Oct 30 14:12:55.912516 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:55.905114 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-00926c/distinct-2-w3-00926c-kserve-746b64d6b5-z66nw" Oct 30 14:12:55.953864 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:55.950031 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8" Oct 30 14:12:56.159179 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:56.159102 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b12b40f-da8a-46d3-a34f-9a1bb0420656-maas-api-tls\") pod \"maas-api-e2e-ait-25917498-7b65854ffc-57t5n\" (UID: \"3b12b40f-da8a-46d3-a34f-9a1bb0420656\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-25917498-7b65854ffc-57t5n" Oct 30 14:12:56.172005 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:56.171931 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" Oct 30 14:12:56.172628 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:56.172022 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-25917498-7b65854ffc-57t5n" Oct 30 14:12:56.263926 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:56.263036 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" Oct 30 14:12:56.489716 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:56.488379 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:56.873693 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:56.873386 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:12:57.239372 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:57.239345 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-25917498-7b65854ffc-57t5n"] Oct 30 14:12:57.239742 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:12:57.239475 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b12b40f_da8a_46d3_a34f_9a1bb0420656.slice/crio-8c2f0ca2c1ebf6625d602c75f530b189c5f26cf3ca1eb10500f0da30dc7c212a WatchSource:0}: Error finding container 8c2f0ca2c1ebf6625d602c75f530b189c5f26cf3ca1eb10500f0da30dc7c212a: Status 404 returned error can't find the container with id 8c2f0ca2c1ebf6625d602c75f530b189c5f26cf3ca1eb10500f0da30dc7c212a Oct 30 14:12:57.912247 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:57.912160 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-25917498-7b65854ffc-57t5n" event={"ID":"3b12b40f-da8a-46d3-a34f-9a1bb0420656","Type":"ContainerStarted","Data":"3cc7fde94e17aafe1319820a07bdd7990b30e1cb256ef7e0752048c75be9c49d"} Oct 30 14:12:57.912247 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:57.912230 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-25917498-7b65854ffc-57t5n" event={"ID":"3b12b40f-da8a-46d3-a34f-9a1bb0420656","Type":"ContainerStarted","Data":"8c2f0ca2c1ebf6625d602c75f530b189c5f26cf3ca1eb10500f0da30dc7c212a"} Oct 30 14:12:57.912498 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:57.912261 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-25917498-7b65854ffc-57t5n" Oct 30 14:12:57.957600 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:57.957536 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-25917498-7b65854ffc-57t5n" podStartSLOduration=4.62632665 podStartE2EDuration="4.957517022s" totalImagesPullingTime="331.190372ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:12:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:12:57.957031102 +0000 UTC m=+1447.701165146" watchObservedRunningTime="2026-10-30 14:12:57.957517022 +0000 UTC m=+1447.701651065" Oct 30 14:12:58.014098 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:12:58.014058 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:12:58.125306 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:12:58.125272 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:00.044234 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:00.044203 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:01.263963 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:01.263938 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v" Oct 30 14:13:01.339501 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:01.339462 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68c5e59c-004d-4751-bed1-f7943e0548fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68c5e59c-004d-4751-bed1-f7943e0548fd-istiod-ca-cert\") pod \"68c5e59c-004d-4751-bed1-f7943e0548fd\" (UID: \"68c5e59c-004d-4751-bed1-f7943e0548fd\") " Oct 30 14:13:01.339685 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:01.339535 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68c5e59c-004d-4751-bed1-f7943e0548fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68c5e59c-004d-4751-bed1-f7943e0548fd-workload-certs\") pod \"68c5e59c-004d-4751-bed1-f7943e0548fd\" (UID: \"68c5e59c-004d-4751-bed1-f7943e0548fd\") " Oct 30 14:13:01.340037 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:01.339629 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68c5e59c-004d-4751-bed1-f7943e0548fd-kube-api-access-hlp77\" (UniqueName: \"kubernetes.io/projected/68c5e59c-004d-4751-bed1-f7943e0548fd-kube-api-access-hlp77\") pod \"68c5e59c-004d-4751-bed1-f7943e0548fd\" (UID: \"68c5e59c-004d-4751-bed1-f7943e0548fd\") " Oct 30 14:13:01.340167 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:01.339861 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68c5e59c-004d-4751-bed1-f7943e0548fd-workload-certs" pod "68c5e59c-004d-4751-bed1-f7943e0548fd" (UID: "68c5e59c-004d-4751-bed1-f7943e0548fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:13:01.340167 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:01.339924 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68c5e59c-004d-4751-bed1-f7943e0548fd-istiod-ca-cert" pod "68c5e59c-004d-4751-bed1-f7943e0548fd" (UID: "68c5e59c-004d-4751-bed1-f7943e0548fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:13:01.340279 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:01.340248 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68c5e59c-004d-4751-bed1-f7943e0548fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68c5e59c-004d-4751-bed1-f7943e0548fd-istio-envoy\") pod \"68c5e59c-004d-4751-bed1-f7943e0548fd\" (UID: \"68c5e59c-004d-4751-bed1-f7943e0548fd\") " Oct 30 14:13:01.340399 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:01.340295 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68c5e59c-004d-4751-bed1-f7943e0548fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68c5e59c-004d-4751-bed1-f7943e0548fd-workload-socket\") pod \"68c5e59c-004d-4751-bed1-f7943e0548fd\" (UID: \"68c5e59c-004d-4751-bed1-f7943e0548fd\") " Oct 30 14:13:01.340399 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:01.340357 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68c5e59c-004d-4751-bed1-f7943e0548fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68c5e59c-004d-4751-bed1-f7943e0548fd-istio-data\") pod \"68c5e59c-004d-4751-bed1-f7943e0548fd\" (UID: \"68c5e59c-004d-4751-bed1-f7943e0548fd\") " Oct 30 14:13:01.340516 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:01.340395 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68c5e59c-004d-4751-bed1-f7943e0548fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68c5e59c-004d-4751-bed1-f7943e0548fd-credential-socket\") pod \"68c5e59c-004d-4751-bed1-f7943e0548fd\" (UID: \"68c5e59c-004d-4751-bed1-f7943e0548fd\") " Oct 30 14:13:01.340516 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:01.340465 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/68c5e59c-004d-4751-bed1-f7943e0548fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68c5e59c-004d-4751-bed1-f7943e0548fd-istio-podinfo\") pod \"68c5e59c-004d-4751-bed1-f7943e0548fd\" (UID: \"68c5e59c-004d-4751-bed1-f7943e0548fd\") " Oct 30 14:13:01.340805 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:01.340764 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68c5e59c-004d-4751-bed1-f7943e0548fd-istio-data" pod "68c5e59c-004d-4751-bed1-f7943e0548fd" (UID: "68c5e59c-004d-4751-bed1-f7943e0548fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:13:01.340907 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:01.340859 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68c5e59c-004d-4751-bed1-f7943e0548fd-istio-token\" (UniqueName: \"kubernetes.io/projected/68c5e59c-004d-4751-bed1-f7943e0548fd-istio-token\") pod \"68c5e59c-004d-4751-bed1-f7943e0548fd\" (UID: \"68c5e59c-004d-4751-bed1-f7943e0548fd\") " Oct 30 14:13:01.341072 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:01.341003 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68c5e59c-004d-4751-bed1-f7943e0548fd-credential-socket" pod "68c5e59c-004d-4751-bed1-f7943e0548fd" (UID: "68c5e59c-004d-4751-bed1-f7943e0548fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:13:01.341748 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:01.341244 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68c5e59c-004d-4751-bed1-f7943e0548fd-workload-socket" pod "68c5e59c-004d-4751-bed1-f7943e0548fd" (UID: "68c5e59c-004d-4751-bed1-f7943e0548fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:13:01.341748 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:01.341602 4982 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68c5e59c-004d-4751-bed1-f7943e0548fd-istiod-ca-cert\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:13:01.341748 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:01.341625 4982 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68c5e59c-004d-4751-bed1-f7943e0548fd-workload-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:13:01.341748 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:01.341654 4982 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68c5e59c-004d-4751-bed1-f7943e0548fd-workload-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:13:01.341748 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:01.341669 4982 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68c5e59c-004d-4751-bed1-f7943e0548fd-istio-data\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:13:01.341748 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:01.341681 4982 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68c5e59c-004d-4751-bed1-f7943e0548fd-credential-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:13:01.343262 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:01.343218 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68c5e59c-004d-4751-bed1-f7943e0548fd-kube-api-access-hlp77" pod "68c5e59c-004d-4751-bed1-f7943e0548fd" (UID: "68c5e59c-004d-4751-bed1-f7943e0548fd"). InnerVolumeSpecName "kube-api-access-hlp77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:13:01.343518 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:01.343483 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/68c5e59c-004d-4751-bed1-f7943e0548fd-istio-podinfo" pod "68c5e59c-004d-4751-bed1-f7943e0548fd" (UID: "68c5e59c-004d-4751-bed1-f7943e0548fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:13:01.343726 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:01.343704 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68c5e59c-004d-4751-bed1-f7943e0548fd-istio-token" pod "68c5e59c-004d-4751-bed1-f7943e0548fd" (UID: "68c5e59c-004d-4751-bed1-f7943e0548fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:13:01.356151 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:01.356127 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68c5e59c-004d-4751-bed1-f7943e0548fd-istio-envoy" pod "68c5e59c-004d-4751-bed1-f7943e0548fd" (UID: "68c5e59c-004d-4751-bed1-f7943e0548fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:13:01.443991 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:01.443951 4982 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68c5e59c-004d-4751-bed1-f7943e0548fd-istio-envoy\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:13:01.443991 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:01.443990 4982 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68c5e59c-004d-4751-bed1-f7943e0548fd-istio-podinfo\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:13:01.444312 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:01.444003 4982 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68c5e59c-004d-4751-bed1-f7943e0548fd-istio-token\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:13:01.444312 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:01.444016 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlp77\" (UniqueName: \"kubernetes.io/projected/68c5e59c-004d-4751-bed1-f7943e0548fd-kube-api-access-hlp77\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:13:01.945404 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:01.945373 4982 generic.go:356] "Generic (PLEG): container finished" podID="68c5e59c-004d-4751-bed1-f7943e0548fd" containerID="57dee7f0140bb9eb6c64184aa7957d4c5fa8b3dd914b28e44e360302195b4983" exitCode=0 Oct 30 14:13:01.945582 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:01.945419 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v" Oct 30 14:13:01.945582 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:01.945457 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v" event={"ID":"68c5e59c-004d-4751-bed1-f7943e0548fd","Type":"ContainerDied","Data":"57dee7f0140bb9eb6c64184aa7957d4c5fa8b3dd914b28e44e360302195b4983"} Oct 30 14:13:01.945582 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:01.945515 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v" event={"ID":"68c5e59c-004d-4751-bed1-f7943e0548fd","Type":"ContainerDied","Data":"6abce4cc1e54e59aaafc3953569dd3aa6354667791b4acf8535e572dca3a45aa"} Oct 30 14:13:01.945582 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:01.945541 4982 scope.go:122] "RemoveContainer" containerID="57dee7f0140bb9eb6c64184aa7957d4c5fa8b3dd914b28e44e360302195b4983" Oct 30 14:13:01.957674 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:01.957655 4982 scope.go:122] "RemoveContainer" containerID="57dee7f0140bb9eb6c64184aa7957d4c5fa8b3dd914b28e44e360302195b4983" Oct 30 14:13:01.957949 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:13:01.957932 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57dee7f0140bb9eb6c64184aa7957d4c5fa8b3dd914b28e44e360302195b4983\": container with ID starting with 57dee7f0140bb9eb6c64184aa7957d4c5fa8b3dd914b28e44e360302195b4983 not found: ID does not exist" containerID="57dee7f0140bb9eb6c64184aa7957d4c5fa8b3dd914b28e44e360302195b4983" Oct 30 14:13:01.958018 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:01.957956 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57dee7f0140bb9eb6c64184aa7957d4c5fa8b3dd914b28e44e360302195b4983"} err="failed to get container status \"57dee7f0140bb9eb6c64184aa7957d4c5fa8b3dd914b28e44e360302195b4983\": rpc error: code = NotFound desc = could not find container \"57dee7f0140bb9eb6c64184aa7957d4c5fa8b3dd914b28e44e360302195b4983\": container with ID starting with 57dee7f0140bb9eb6c64184aa7957d4c5fa8b3dd914b28e44e360302195b4983 not found: ID does not exist" Oct 30 14:13:01.960562 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:01.960541 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:02.020330 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:02.020280 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v"] Oct 30 14:13:02.037536 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:02.036335 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-trq2v"] Oct 30 14:13:02.400844 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:02.400815 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68c5e59c-004d-4751-bed1-f7943e0548fd" path="/var/lib/kubelet/pods/68c5e59c-004d-4751-bed1-f7943e0548fd/volumes" Oct 30 14:13:03.926653 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:03.926609 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-25917498-7b65854ffc-57t5n" Oct 30 14:13:04.195895 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:04.195822 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:06.024632 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:06.024594 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:06.225963 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:06.225931 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:07.893975 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:07.893938 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:09.838301 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:09.838268 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:11.750331 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:11.750291 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:13.735605 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:13.735562 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:13.900954 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:13.900924 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:14.415472 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:14.415427 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:15.833573 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:15.833533 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:17.706041 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:17.706003 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:19.668794 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:19.668757 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:20.772435 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:20.772393 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:20.918752 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:20.918713 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:21.170379 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:21.170328 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:21.562385 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:21.562351 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:21.730355 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:21.730302 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:22.148223 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:22.148182 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:22.301115 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:22.301066 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:22.463159 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:22.463056 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:22.630121 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:22.630084 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:23.243331 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:23.243289 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:24.334673 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:24.330614 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:24.977663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:24.952009 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ea3e7b1d-6fb74b859f-kxc7w"] Oct 30 14:13:25.062509 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:25.062467 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:25.217063 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:25.217014 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:25.323050 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:25.313717 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" podUID="946696b9-91bd-437a-9f23-c16b73a5a9a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 14:13:25.350208 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:25.350158 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" podUID="15e7b7f9-2570-410a-a398-4c07c2e465ea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 14:13:25.407044 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:25.406998 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" podUID="ab822fe6-4349-4051-b2fd-c6e5ceb50d08" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:13:25.558161 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:25.558097 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:25.677882 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:25.677788 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" podUID="cbd754bd-771d-4007-8ac0-7e413dc24915" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:13:25.725996 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:25.725943 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-pvbq6" podUID="d13287fc-d0ba-4231-a5c5-7a0a128e620a" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.26:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:13:25.857797 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:25.857750 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" podUID="cc3a3a28-d723-459a-93d0-ff115eac479a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded" Oct 30 14:13:26.327741 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:26.327696 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" podUID="946696b9-91bd-437a-9f23-c16b73a5a9a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 14:13:26.365660 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:26.347907 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" Oct 30 14:13:26.375913 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:26.371374 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" Oct 30 14:13:26.678554 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:26.678458 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" podUID="cbd754bd-771d-4007-8ac0-7e413dc24915" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:13:26.691868 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:26.691834 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" Oct 30 14:13:26.896663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:26.884764 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" podUID="00fdf663-9bc9-4349-a565-6a8dc01c0842" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 14:13:27.017176 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:27.016428 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" Oct 30 14:13:27.212428 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:27.210814 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/distinct-w3-00926c-kserve-774df5b58-cv8vt" podUID="b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:13:27.690707 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:27.682819 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" podUID="4fa94d86-57ce-4d23-95d5-4efa965f6864" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:13:27.903662 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:27.901862 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" podUID="cc3a3a28-d723-459a-93d0-ff115eac479a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:13:27.903662 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:27.901799 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" podUID="00fdf663-9bc9-4349-a565-6a8dc01c0842" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 14:13:27.904566 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:27.904420 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" Oct 30 14:13:28.026485 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:28.023759 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:28.054936 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:28.053725 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" podUID="d65dee9b-4f8b-4f09-b00b-0793d732d3ab" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:13:28.213341 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:28.212194 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/distinct-w3-00926c-kserve-774df5b58-cv8vt" podUID="b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 30 14:13:28.236806 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:28.213983 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-00926c/distinct-w3-00926c-kserve-774df5b58-cv8vt" Oct 30 14:13:28.325716 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:13:28.300819 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:13:28.387790 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:28.387745 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" podUID="946696b9-91bd-437a-9f23-c16b73a5a9a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 14:13:28.497665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:28.482499 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" podUID="ab822fe6-4349-4051-b2fd-c6e5ceb50d08" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:13:28.746838 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:28.746737 4982 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" podUID="cc3a3a28-d723-459a-93d0-ff115eac479a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": context deadline exceeded" Oct 30 14:13:28.904794 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:28.904746 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" podUID="cc3a3a28-d723-459a-93d0-ff115eac479a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:13:28.905382 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:28.905027 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" podUID="00fdf663-9bc9-4349-a565-6a8dc01c0842" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 14:13:28.907111 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:28.906375 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" Oct 30 14:13:29.008614 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:29.007813 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8" podUID="f3093abe-9b1e-492a-8837-4bd42ea39d2d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Oct 30 14:13:29.059944 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:29.057766 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" podUID="d65dee9b-4f8b-4f09-b00b-0793d732d3ab" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:13:29.059944 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:29.058203 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" Oct 30 14:13:29.121822 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:29.121750 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" podUID="cbd754bd-771d-4007-8ac0-7e413dc24915" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 14:13:29.167233 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:29.163979 4982 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-5842c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:13:29.167233 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:29.165344 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5842c" podUID="f56cfd10-4ad6-4c14-b614-ae3b48b87dbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:13:29.171831 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:29.171151 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/distinct-2-w3-00926c-kserve-746b64d6b5-z66nw" podUID="91f37a37-70fe-4627-b2dc-77a9668d9a5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:13:29.192720 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:29.192685 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" podUID="15e7b7f9-2570-410a-a398-4c07c2e465ea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:13:29.447181 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:29.442993 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/distinct-w3-00926c-kserve-774df5b58-cv8vt" podUID="b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 14:13:29.447875 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:29.445283 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:29.478660 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:29.475477 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-00926c/distinct-w3-00926c-kserve-774df5b58-cv8vt" Oct 30 14:13:29.535990 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:29.535960 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" Oct 30 14:13:29.573490 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:29.573463 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" Oct 30 14:13:29.575498 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:29.575470 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" Oct 30 14:13:29.622112 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:29.622079 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:29.878966 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:29.878931 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:29.978379 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:29.978318 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-ea3e7b1d-6fb74b859f-kxc7w" podUID="3029292f-1aa0-49ce-bda4-a033a001e53e" containerName="payload-pre-processing" containerID="cri-o://6e3609571012334c0245ebd1748bf8959fd0cbbd5150a3f6a7a4f6a1444d7e80" gracePeriod=25 Oct 30 14:13:30.378272 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:30.378246 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ea3e7b1d-6fb74b859f-kxc7w" Oct 30 14:13:30.517240 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:30.517152 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3029292f-1aa0-49ce-bda4-a033a001e53e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3029292f-1aa0-49ce-bda4-a033a001e53e-plugins-config-volume\") pod \"3029292f-1aa0-49ce-bda4-a033a001e53e\" (UID: \"3029292f-1aa0-49ce-bda4-a033a001e53e\") " Oct 30 14:13:30.517465 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:30.517446 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3029292f-1aa0-49ce-bda4-a033a001e53e-kube-api-access-dd6nt\" (UniqueName: \"kubernetes.io/projected/3029292f-1aa0-49ce-bda4-a033a001e53e-kube-api-access-dd6nt\") pod \"3029292f-1aa0-49ce-bda4-a033a001e53e\" (UID: \"3029292f-1aa0-49ce-bda4-a033a001e53e\") " Oct 30 14:13:30.517559 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:30.517536 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3029292f-1aa0-49ce-bda4-a033a001e53e-plugins-config-volume" pod "3029292f-1aa0-49ce-bda4-a033a001e53e" (UID: "3029292f-1aa0-49ce-bda4-a033a001e53e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:13:30.518126 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:30.518082 4982 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3029292f-1aa0-49ce-bda4-a033a001e53e-plugins-config-volume\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:13:30.520057 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:30.520015 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3029292f-1aa0-49ce-bda4-a033a001e53e-kube-api-access-dd6nt" pod "3029292f-1aa0-49ce-bda4-a033a001e53e" (UID: "3029292f-1aa0-49ce-bda4-a033a001e53e"). InnerVolumeSpecName "kube-api-access-dd6nt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:13:30.619697 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:30.619667 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dd6nt\" (UniqueName: \"kubernetes.io/projected/3029292f-1aa0-49ce-bda4-a033a001e53e-kube-api-access-dd6nt\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:13:30.694914 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:30.694873 4982 generic.go:356] "Generic (PLEG): container finished" podID="3029292f-1aa0-49ce-bda4-a033a001e53e" containerID="6e3609571012334c0245ebd1748bf8959fd0cbbd5150a3f6a7a4f6a1444d7e80" exitCode=0 Oct 30 14:13:30.695071 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:30.695021 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ea3e7b1d-6fb74b859f-kxc7w" event={"ID":"3029292f-1aa0-49ce-bda4-a033a001e53e","Type":"ContainerDied","Data":"6e3609571012334c0245ebd1748bf8959fd0cbbd5150a3f6a7a4f6a1444d7e80"} Oct 30 14:13:30.695134 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:30.695071 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ea3e7b1d-6fb74b859f-kxc7w" event={"ID":"3029292f-1aa0-49ce-bda4-a033a001e53e","Type":"ContainerDied","Data":"074a1cac2da214a20f8bf8afcf911ef1059c2e44c1a0d77194466aab56a9821e"} Oct 30 14:13:30.695134 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:30.695102 4982 scope.go:122] "RemoveContainer" containerID="6e3609571012334c0245ebd1748bf8959fd0cbbd5150a3f6a7a4f6a1444d7e80" Oct 30 14:13:30.695289 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:30.695275 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ea3e7b1d-6fb74b859f-kxc7w" Oct 30 14:13:30.708950 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:30.708921 4982 scope.go:122] "RemoveContainer" containerID="6e3609571012334c0245ebd1748bf8959fd0cbbd5150a3f6a7a4f6a1444d7e80" Oct 30 14:13:30.709285 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:13:30.709261 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e3609571012334c0245ebd1748bf8959fd0cbbd5150a3f6a7a4f6a1444d7e80\": container with ID starting with 6e3609571012334c0245ebd1748bf8959fd0cbbd5150a3f6a7a4f6a1444d7e80 not found: ID does not exist" containerID="6e3609571012334c0245ebd1748bf8959fd0cbbd5150a3f6a7a4f6a1444d7e80" Oct 30 14:13:30.709355 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:30.709297 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e3609571012334c0245ebd1748bf8959fd0cbbd5150a3f6a7a4f6a1444d7e80"} err="failed to get container status \"6e3609571012334c0245ebd1748bf8959fd0cbbd5150a3f6a7a4f6a1444d7e80\": rpc error: code = NotFound desc = could not find container \"6e3609571012334c0245ebd1748bf8959fd0cbbd5150a3f6a7a4f6a1444d7e80\": container with ID starting with 6e3609571012334c0245ebd1748bf8959fd0cbbd5150a3f6a7a4f6a1444d7e80 not found: ID does not exist" Oct 30 14:13:30.723318 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:30.723289 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:30.756587 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:30.756561 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ea3e7b1d-6fb74b859f-kxc7w"] Oct 30 14:13:30.769563 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:30.769505 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ea3e7b1d-6fb74b859f-kxc7w"] Oct 30 14:13:31.279325 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:31.279290 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:31.803195 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:31.803156 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:31.967992 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:31.967913 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:32.095076 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:32.094940 4982 patch_prober.go:29] interesting pod/e2e-ait-25917498-openshift-default-77dc9bb847-smvqn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:13:32.095076 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:32.095015 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-25917498-openshift-default-77dc9bb847-smvqn" podUID="8104c709-90b5-4c71-9540-a766b2add736" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:13:32.402979 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:32.402902 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3029292f-1aa0-49ce-bda4-a033a001e53e" path="/var/lib/kubelet/pods/3029292f-1aa0-49ce-bda4-a033a001e53e/volumes" Oct 30 14:13:33.744381 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:33.744291 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:34.744838 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:34.744789 4982 generic.go:356] "Generic (PLEG): container finished" podID="8104c709-90b5-4c71-9540-a766b2add736" containerID="e5f5e55f4a55842ed1cc2b3e9fa9f75aa58b37706b78d0b3ae43c10f6900c918" exitCode=0 Oct 30 14:13:34.745352 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:34.744860 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-25917498-openshift-default-77dc9bb847-smvqn" event={"ID":"8104c709-90b5-4c71-9540-a766b2add736","Type":"ContainerDied","Data":"e5f5e55f4a55842ed1cc2b3e9fa9f75aa58b37706b78d0b3ae43c10f6900c918"} Oct 30 14:13:34.745424 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:34.745411 4982 scope.go:122] "RemoveContainer" containerID="e5f5e55f4a55842ed1cc2b3e9fa9f75aa58b37706b78d0b3ae43c10f6900c918" Oct 30 14:13:34.747071 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:34.747042 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:13:34.747185 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:34.747109 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:13:34.747185 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:34.747142 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:13:35.516598 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:35.516554 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:35.738951 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:35.738919 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:35.753550 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:35.753520 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-25917498-openshift-default-77dc9bb847-smvqn" event={"ID":"8104c709-90b5-4c71-9540-a766b2add736","Type":"ContainerStarted","Data":"c54d7dce00f9fcc8da0dff7036529e72da6b485dd9c7e6b6c9056cb20000656c"} Oct 30 14:13:35.964129 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:35.964093 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:36.733491 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:36.733452 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-25917498-openshift-default-77dc9bb847-smvqn" Oct 30 14:13:36.733829 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:36.733802 4982 patch_prober.go:29] interesting pod/e2e-ait-25917498-openshift-default-77dc9bb847-smvqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 14:13:36.734034 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:36.734007 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-25917498-openshift-default-77dc9bb847-smvqn" podUID="8104c709-90b5-4c71-9540-a766b2add736" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 14:13:37.416686 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:37.412232 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:37.632586 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:37.632523 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:37.847677 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:37.817877 4982 patch_prober.go:29] interesting pod/e2e-ait-25917498-openshift-default-77dc9bb847-smvqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 14:13:37.847677 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:37.817981 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-25917498-openshift-default-77dc9bb847-smvqn" podUID="8104c709-90b5-4c71-9540-a766b2add736" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 14:13:38.157552 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:38.157471 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:38.733913 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:38.733880 4982 patch_prober.go:29] interesting pod/e2e-ait-25917498-openshift-default-77dc9bb847-smvqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 14:13:38.734404 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:38.733941 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-25917498-openshift-default-77dc9bb847-smvqn" podUID="8104c709-90b5-4c71-9540-a766b2add736" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 14:13:39.734146 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:39.734095 4982 patch_prober.go:29] interesting pod/e2e-ait-25917498-openshift-default-77dc9bb847-smvqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 14:13:39.734676 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:39.734165 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-25917498-openshift-default-77dc9bb847-smvqn" podUID="8104c709-90b5-4c71-9540-a766b2add736" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 14:13:40.138260 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:40.138227 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:40.740058 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:40.733761 4982 patch_prober.go:29] interesting pod/e2e-ait-25917498-openshift-default-77dc9bb847-smvqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 14:13:40.740058 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:40.733825 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-25917498-openshift-default-77dc9bb847-smvqn" podUID="8104c709-90b5-4c71-9540-a766b2add736" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 14:13:41.737005 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:41.736970 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ea3e7b1d-66757797d5-54tg9"] Oct 30 14:13:41.737480 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:41.737441 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea3e7b1d-66757797d5-54tg9" podUID="aa0a51b1-d2bd-4321-acfe-ddb5ccd955ae" containerName="maas-api" containerID="cri-o://ed7497714e2a9f1582e0bb2026279641c57413d640da35d42050a9ea9ce3b9d8" gracePeriod=30 Oct 30 14:13:41.764760 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:41.764734 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-25917498-openshift-default-77dc9bb847-smvqn" Oct 30 14:13:41.765769 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:41.765699 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-25917498-openshift-default-77dc9bb847-smvqn" Oct 30 14:13:41.767234 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:41.767167 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-25917498-openshift-default-77dc9bb847-smvqn" Oct 30 14:13:41.824964 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:41.824926 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:41.952257 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:41.952220 4982 generic.go:356] "Generic (PLEG): container finished" podID="aa0a51b1-d2bd-4321-acfe-ddb5ccd955ae" containerID="ed7497714e2a9f1582e0bb2026279641c57413d640da35d42050a9ea9ce3b9d8" exitCode=0 Oct 30 14:13:41.952453 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:41.952274 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea3e7b1d-66757797d5-54tg9" event={"ID":"aa0a51b1-d2bd-4321-acfe-ddb5ccd955ae","Type":"ContainerDied","Data":"ed7497714e2a9f1582e0bb2026279641c57413d640da35d42050a9ea9ce3b9d8"} Oct 30 14:13:42.157113 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:42.157083 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea3e7b1d-66757797d5-54tg9" Oct 30 14:13:42.254100 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:42.254060 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa0a51b1-d2bd-4321-acfe-ddb5ccd955ae-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa0a51b1-d2bd-4321-acfe-ddb5ccd955ae-maas-api-tls\") pod \"aa0a51b1-d2bd-4321-acfe-ddb5ccd955ae\" (UID: \"aa0a51b1-d2bd-4321-acfe-ddb5ccd955ae\") " Oct 30 14:13:42.254330 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:42.254308 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa0a51b1-d2bd-4321-acfe-ddb5ccd955ae-metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa0a51b1-d2bd-4321-acfe-ddb5ccd955ae-metrics-certs\") pod \"aa0a51b1-d2bd-4321-acfe-ddb5ccd955ae\" (UID: \"aa0a51b1-d2bd-4321-acfe-ddb5ccd955ae\") " Oct 30 14:13:42.254446 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:42.254342 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa0a51b1-d2bd-4321-acfe-ddb5ccd955ae-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa0a51b1-d2bd-4321-acfe-ddb5ccd955ae-openshift-service-ca\") pod \"aa0a51b1-d2bd-4321-acfe-ddb5ccd955ae\" (UID: \"aa0a51b1-d2bd-4321-acfe-ddb5ccd955ae\") " Oct 30 14:13:42.254446 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:42.254402 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa0a51b1-d2bd-4321-acfe-ddb5ccd955ae-kube-api-access-x5vvh\" (UniqueName: \"kubernetes.io/projected/aa0a51b1-d2bd-4321-acfe-ddb5ccd955ae-kube-api-access-x5vvh\") pod \"aa0a51b1-d2bd-4321-acfe-ddb5ccd955ae\" (UID: \"aa0a51b1-d2bd-4321-acfe-ddb5ccd955ae\") " Oct 30 14:13:42.255858 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:42.255824 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa0a51b1-d2bd-4321-acfe-ddb5ccd955ae-openshift-service-ca" pod "aa0a51b1-d2bd-4321-acfe-ddb5ccd955ae" (UID: "aa0a51b1-d2bd-4321-acfe-ddb5ccd955ae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:13:42.268674 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:42.266949 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa0a51b1-d2bd-4321-acfe-ddb5ccd955ae-metrics-certs" pod "aa0a51b1-d2bd-4321-acfe-ddb5ccd955ae" (UID: "aa0a51b1-d2bd-4321-acfe-ddb5ccd955ae"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:13:42.273195 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:42.272908 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa0a51b1-d2bd-4321-acfe-ddb5ccd955ae-maas-api-tls" pod "aa0a51b1-d2bd-4321-acfe-ddb5ccd955ae" (UID: "aa0a51b1-d2bd-4321-acfe-ddb5ccd955ae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:13:42.279682 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:42.279506 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa0a51b1-d2bd-4321-acfe-ddb5ccd955ae-kube-api-access-x5vvh" pod "aa0a51b1-d2bd-4321-acfe-ddb5ccd955ae" (UID: "aa0a51b1-d2bd-4321-acfe-ddb5ccd955ae"). InnerVolumeSpecName "kube-api-access-x5vvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:13:42.356740 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:42.356708 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5vvh\" (UniqueName: \"kubernetes.io/projected/aa0a51b1-d2bd-4321-acfe-ddb5ccd955ae-kube-api-access-x5vvh\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:13:42.356740 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:42.356737 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa0a51b1-d2bd-4321-acfe-ddb5ccd955ae-maas-api-tls\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:13:42.356740 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:42.356748 4982 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa0a51b1-d2bd-4321-acfe-ddb5ccd955ae-metrics-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:13:42.356973 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:42.356757 4982 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa0a51b1-d2bd-4321-acfe-ddb5ccd955ae-openshift-service-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:13:42.377077 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:42.376994 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:42.542449 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:42.542410 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:42.787716 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:42.787195 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:42.954544 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:42.954502 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:42.963631 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:42.963604 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea3e7b1d-66757797d5-54tg9" Oct 30 14:13:42.963631 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:42.963607 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea3e7b1d-66757797d5-54tg9" event={"ID":"aa0a51b1-d2bd-4321-acfe-ddb5ccd955ae","Type":"ContainerDied","Data":"0568471c5045b375f0785b023bda120fdc4c8838003dfa6be25c47d44b613b8f"} Oct 30 14:13:42.963873 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:42.963722 4982 scope.go:122] "RemoveContainer" containerID="ed7497714e2a9f1582e0bb2026279641c57413d640da35d42050a9ea9ce3b9d8" Oct 30 14:13:43.025181 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:43.025147 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ea3e7b1d-66757797d5-54tg9"] Oct 30 14:13:43.034528 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:43.034496 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ea3e7b1d-66757797d5-54tg9"] Oct 30 14:13:43.121232 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:43.121193 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:43.780126 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:43.780087 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:43.936479 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:43.936448 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:44.093846 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:44.093819 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:44.408660 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:44.408562 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa0a51b1-d2bd-4321-acfe-ddb5ccd955ae" path="/var/lib/kubelet/pods/aa0a51b1-d2bd-4321-acfe-ddb5ccd955ae/volumes" Oct 30 14:13:44.452749 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:44.452723 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:44.603863 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:44.603823 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:44.758971 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:44.758890 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:46.406907 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:46.406870 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:46.565766 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:46.565738 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:46.742097 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:46.741982 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:48.310133 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:48.310095 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:48.444515 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:48.444480 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-75443e-7c475f6fbc-d7wdc"] Oct 30 14:13:48.445190 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:48.445168 4982 cpu_manager.go:414] "containerMap: removing container" podUID="3029292f-1aa0-49ce-bda4-a033a001e53e" containerName="payload-pre-processing" Oct 30 14:13:48.445312 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:48.445191 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3029292f-1aa0-49ce-bda4-a033a001e53e" containerName="payload-pre-processing" Oct 30 14:13:48.445312 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:48.445203 4982 cpu_manager.go:414] "containerMap: removing container" podUID="aa0a51b1-d2bd-4321-acfe-ddb5ccd955ae" containerName="maas-api" Oct 30 14:13:48.445312 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:48.445210 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa0a51b1-d2bd-4321-acfe-ddb5ccd955ae" containerName="maas-api" Oct 30 14:13:48.445312 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:48.445225 4982 cpu_manager.go:414] "containerMap: removing container" podUID="68c5e59c-004d-4751-bed1-f7943e0548fd" containerName="istio-proxy" Oct 30 14:13:48.445312 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:48.445233 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68c5e59c-004d-4751-bed1-f7943e0548fd" containerName="istio-proxy" Oct 30 14:13:48.445549 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:48.445402 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="3029292f-1aa0-49ce-bda4-a033a001e53e" containerName="payload-pre-processing" Oct 30 14:13:48.445549 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:48.445417 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa0a51b1-d2bd-4321-acfe-ddb5ccd955ae" containerName="maas-api" Oct 30 14:13:48.445549 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:48.445429 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="68c5e59c-004d-4751-bed1-f7943e0548fd" containerName="istio-proxy" Oct 30 14:13:48.450247 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:48.450226 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-75443e-7c475f6fbc-d7wdc" Oct 30 14:13:48.474632 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:48.474608 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-75443e-7c475f6fbc-d7wdc"] Oct 30 14:13:48.481336 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:48.481312 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:48.483416 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:48.483396 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-75443e-dockercfg-c54th\"" Oct 30 14:13:48.483416 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:48.483396 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-75443e\"" Oct 30 14:13:48.525965 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:48.525508 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-75443e-6b6cd45ccb-fnql4"] Oct 30 14:13:48.532933 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:48.532911 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-75443e-6b6cd45ccb-fnql4" Oct 30 14:13:48.551109 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:48.551080 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-75443e-6b6cd45ccb-fnql4"] Oct 30 14:13:48.672754 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:48.656306 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42ebc63e-3f5f-4aa3-b2cc-b9d92e3f9e0c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-75443e-6b6cd45ccb-fnql4\" (UID: \"42ebc63e-3f5f-4aa3-b2cc-b9d92e3f9e0c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-75443e-6b6cd45ccb-fnql4" Oct 30 14:13:48.672754 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:48.656380 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg22h\" (UniqueName: \"kubernetes.io/projected/42ebc63e-3f5f-4aa3-b2cc-b9d92e3f9e0c-kube-api-access-sg22h\") pod \"payload-processing-e2e-shared-a-w6-75443e-6b6cd45ccb-fnql4\" (UID: \"42ebc63e-3f5f-4aa3-b2cc-b9d92e3f9e0c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-75443e-6b6cd45ccb-fnql4" Oct 30 14:13:48.672754 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:48.656498 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxvj6\" (UniqueName: \"kubernetes.io/projected/02ab3f3b-3aac-4acd-8340-3f63c789ec8c-kube-api-access-fxvj6\") pod \"payload-pre-processing-e2e-shared-a-w6-75443e-7c475f6fbc-d7wdc\" (UID: \"02ab3f3b-3aac-4acd-8340-3f63c789ec8c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-75443e-7c475f6fbc-d7wdc" Oct 30 14:13:48.672754 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:48.656758 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02ab3f3b-3aac-4acd-8340-3f63c789ec8c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-75443e-7c475f6fbc-d7wdc\" (UID: \"02ab3f3b-3aac-4acd-8340-3f63c789ec8c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-75443e-7c475f6fbc-d7wdc" Oct 30 14:13:48.758785 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:48.758746 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42ebc63e-3f5f-4aa3-b2cc-b9d92e3f9e0c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-75443e-6b6cd45ccb-fnql4\" (UID: \"42ebc63e-3f5f-4aa3-b2cc-b9d92e3f9e0c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-75443e-6b6cd45ccb-fnql4" Oct 30 14:13:48.758956 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:48.758808 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg22h\" (UniqueName: \"kubernetes.io/projected/42ebc63e-3f5f-4aa3-b2cc-b9d92e3f9e0c-kube-api-access-sg22h\") pod \"payload-processing-e2e-shared-a-w6-75443e-6b6cd45ccb-fnql4\" (UID: \"42ebc63e-3f5f-4aa3-b2cc-b9d92e3f9e0c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-75443e-6b6cd45ccb-fnql4" Oct 30 14:13:48.758956 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:48.758936 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxvj6\" (UniqueName: \"kubernetes.io/projected/02ab3f3b-3aac-4acd-8340-3f63c789ec8c-kube-api-access-fxvj6\") pod \"payload-pre-processing-e2e-shared-a-w6-75443e-7c475f6fbc-d7wdc\" (UID: \"02ab3f3b-3aac-4acd-8340-3f63c789ec8c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-75443e-7c475f6fbc-d7wdc" Oct 30 14:13:48.759115 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:48.759051 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02ab3f3b-3aac-4acd-8340-3f63c789ec8c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-75443e-7c475f6fbc-d7wdc\" (UID: \"02ab3f3b-3aac-4acd-8340-3f63c789ec8c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-75443e-7c475f6fbc-d7wdc" Oct 30 14:13:48.759695 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:48.759660 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42ebc63e-3f5f-4aa3-b2cc-b9d92e3f9e0c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-75443e-6b6cd45ccb-fnql4\" (UID: \"42ebc63e-3f5f-4aa3-b2cc-b9d92e3f9e0c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-75443e-6b6cd45ccb-fnql4" Oct 30 14:13:48.759918 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:48.759884 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02ab3f3b-3aac-4acd-8340-3f63c789ec8c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-75443e-7c475f6fbc-d7wdc\" (UID: \"02ab3f3b-3aac-4acd-8340-3f63c789ec8c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-75443e-7c475f6fbc-d7wdc" Oct 30 14:13:48.787114 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:48.787068 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg22h\" (UniqueName: \"kubernetes.io/projected/42ebc63e-3f5f-4aa3-b2cc-b9d92e3f9e0c-kube-api-access-sg22h\") pod \"payload-processing-e2e-shared-a-w6-75443e-6b6cd45ccb-fnql4\" (UID: \"42ebc63e-3f5f-4aa3-b2cc-b9d92e3f9e0c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-75443e-6b6cd45ccb-fnql4" Oct 30 14:13:48.787277 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:48.787238 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxvj6\" (UniqueName: \"kubernetes.io/projected/02ab3f3b-3aac-4acd-8340-3f63c789ec8c-kube-api-access-fxvj6\") pod \"payload-pre-processing-e2e-shared-a-w6-75443e-7c475f6fbc-d7wdc\" (UID: \"02ab3f3b-3aac-4acd-8340-3f63c789ec8c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-75443e-7c475f6fbc-d7wdc" Oct 30 14:13:48.868657 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:48.868614 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-75443e-6b6cd45ccb-fnql4" Oct 30 14:13:48.977616 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:48.977519 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:49.063971 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:49.063929 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-75443e-7c475f6fbc-d7wdc" Oct 30 14:13:49.251927 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:49.251786 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-75443e-6b6cd45ccb-fnql4"] Oct 30 14:13:49.260524 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:13:49.255292 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42ebc63e_3f5f_4aa3_b2cc_b9d92e3f9e0c.slice/crio-b6365238ae88307a361849612b1244e0ca1f9229e749bece7d7da00fb9945edb WatchSource:0}: Error finding container b6365238ae88307a361849612b1244e0ca1f9229e749bece7d7da00fb9945edb: Status 404 returned error can't find the container with id b6365238ae88307a361849612b1244e0ca1f9229e749bece7d7da00fb9945edb Oct 30 14:13:49.400435 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:13:49.400402 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02ab3f3b_3aac_4acd_8340_3f63c789ec8c.slice/crio-3b450840fb30f2d4d0edb16c61782617308c248d6aa7cb2345bec420b69e157f WatchSource:0}: Error finding container 3b450840fb30f2d4d0edb16c61782617308c248d6aa7cb2345bec420b69e157f: Status 404 returned error can't find the container with id 3b450840fb30f2d4d0edb16c61782617308c248d6aa7cb2345bec420b69e157f Oct 30 14:13:49.401575 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:49.401544 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-75443e-7c475f6fbc-d7wdc"] Oct 30 14:13:50.018554 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:50.018518 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-75443e-6b6cd45ccb-fnql4" event={"ID":"42ebc63e-3f5f-4aa3-b2cc-b9d92e3f9e0c","Type":"ContainerStarted","Data":"504ade04f400781a538cd542d18de720c7dadb4351a7d4c6470084648d32f1a6"} Oct 30 14:13:50.018772 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:50.018578 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-75443e-6b6cd45ccb-fnql4" event={"ID":"42ebc63e-3f5f-4aa3-b2cc-b9d92e3f9e0c","Type":"ContainerStarted","Data":"b6365238ae88307a361849612b1244e0ca1f9229e749bece7d7da00fb9945edb"} Oct 30 14:13:50.018772 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:50.018697 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-75443e-6b6cd45ccb-fnql4" Oct 30 14:13:50.020746 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:50.020699 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-75443e-7c475f6fbc-d7wdc" event={"ID":"02ab3f3b-3aac-4acd-8340-3f63c789ec8c","Type":"ContainerStarted","Data":"287ae69eacdb8c07d50ca49b7e5029e6777a3d56d09f2668eaca1fb2aa8960a3"} Oct 30 14:13:50.020855 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:50.020769 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-75443e-7c475f6fbc-d7wdc" Oct 30 14:13:50.020855 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:50.020793 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-75443e-7c475f6fbc-d7wdc" event={"ID":"02ab3f3b-3aac-4acd-8340-3f63c789ec8c","Type":"ContainerStarted","Data":"3b450840fb30f2d4d0edb16c61782617308c248d6aa7cb2345bec420b69e157f"} Oct 30 14:13:50.069593 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:50.069549 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-75443e-6b6cd45ccb-fnql4" podStartSLOduration=2.069536557 podStartE2EDuration="2.069536557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:13:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:13:50.065831905 +0000 UTC m=+1499.809965947" watchObservedRunningTime="2026-10-30 14:13:50.069536557 +0000 UTC m=+1499.813670599" Oct 30 14:13:50.112281 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:50.112231 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-75443e-7c475f6fbc-d7wdc" podStartSLOduration=2.112214383 podStartE2EDuration="2.112214383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:13:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:13:50.110298855 +0000 UTC m=+1499.854432909" watchObservedRunningTime="2026-10-30 14:13:50.112214383 +0000 UTC m=+1499.856348428" Oct 30 14:13:50.583003 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:50.582967 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ntlqq_dc967875-1546-40bb-8376-fde09416c204/console-operator/2.log" Oct 30 14:13:50.583511 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:50.583197 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ntlqq_dc967875-1546-40bb-8376-fde09416c204/console-operator/2.log" Oct 30 14:13:50.588672 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:50.588613 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhxjf_aca47fe8-7068-4743-b6ab-283ee0874f66/ovn-acl-logging/0.log" Oct 30 14:13:50.588909 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:50.588892 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhxjf_aca47fe8-7068-4743-b6ab-283ee0874f66/ovn-acl-logging/0.log" Oct 30 14:13:50.865085 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:50.865006 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:52.716271 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:52.716218 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:54.703361 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:54.703311 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:54.851519 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:54.851484 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:55.002041 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:55.001937 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:56.404080 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:56.404044 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:56.628142 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:56.628107 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:57.395743 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:57.395674 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:57.542822 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:57.542778 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:58.017357 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:13:58.017317 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:13:58.167405 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:58.167366 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-25917498-7b65854ffc-57t5n"] Oct 30 14:13:58.167729 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:58.167680 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-25917498-7b65854ffc-57t5n" podUID="3b12b40f-da8a-46d3-a34f-9a1bb0420656" containerName="maas-api" containerID="cri-o://3cc7fde94e17aafe1319820a07bdd7990b30e1cb256ef7e0752048c75be9c49d" gracePeriod=30 Oct 30 14:13:58.278360 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:58.278248 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:58.717910 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:58.717872 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-25917498-7b65854ffc-57t5n" Oct 30 14:13:58.837663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:58.833032 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:58.848594 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:58.843823 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b12b40f-da8a-46d3-a34f-9a1bb0420656-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b12b40f-da8a-46d3-a34f-9a1bb0420656-openshift-service-ca\") pod \"3b12b40f-da8a-46d3-a34f-9a1bb0420656\" (UID: \"3b12b40f-da8a-46d3-a34f-9a1bb0420656\") " Oct 30 14:13:58.848594 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:58.844047 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b12b40f-da8a-46d3-a34f-9a1bb0420656-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b12b40f-da8a-46d3-a34f-9a1bb0420656-maas-api-tls\") pod \"3b12b40f-da8a-46d3-a34f-9a1bb0420656\" (UID: \"3b12b40f-da8a-46d3-a34f-9a1bb0420656\") " Oct 30 14:13:58.848594 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:58.844209 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b12b40f-da8a-46d3-a34f-9a1bb0420656-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b12b40f-da8a-46d3-a34f-9a1bb0420656-metrics-certs\") pod \"3b12b40f-da8a-46d3-a34f-9a1bb0420656\" (UID: \"3b12b40f-da8a-46d3-a34f-9a1bb0420656\") " Oct 30 14:13:58.848594 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:58.844295 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b12b40f-da8a-46d3-a34f-9a1bb0420656-kube-api-access-5wdfw\" (UniqueName: \"kubernetes.io/projected/3b12b40f-da8a-46d3-a34f-9a1bb0420656-kube-api-access-5wdfw\") pod \"3b12b40f-da8a-46d3-a34f-9a1bb0420656\" (UID: \"3b12b40f-da8a-46d3-a34f-9a1bb0420656\") " Oct 30 14:13:58.848594 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:58.845668 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b12b40f-da8a-46d3-a34f-9a1bb0420656-openshift-service-ca" pod "3b12b40f-da8a-46d3-a34f-9a1bb0420656" (UID: "3b12b40f-da8a-46d3-a34f-9a1bb0420656"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:13:58.858745 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:58.857200 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b12b40f-da8a-46d3-a34f-9a1bb0420656-maas-api-tls" pod "3b12b40f-da8a-46d3-a34f-9a1bb0420656" (UID: "3b12b40f-da8a-46d3-a34f-9a1bb0420656"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:13:58.868206 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:58.868170 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b12b40f-da8a-46d3-a34f-9a1bb0420656-kube-api-access-5wdfw" pod "3b12b40f-da8a-46d3-a34f-9a1bb0420656" (UID: "3b12b40f-da8a-46d3-a34f-9a1bb0420656"). InnerVolumeSpecName "kube-api-access-5wdfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:13:58.870516 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:58.868974 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b12b40f-da8a-46d3-a34f-9a1bb0420656-metrics-certs" pod "3b12b40f-da8a-46d3-a34f-9a1bb0420656" (UID: "3b12b40f-da8a-46d3-a34f-9a1bb0420656"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:13:58.947197 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:58.947098 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b12b40f-da8a-46d3-a34f-9a1bb0420656-maas-api-tls\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:13:58.947197 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:58.947144 4982 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b12b40f-da8a-46d3-a34f-9a1bb0420656-metrics-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:13:58.947197 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:58.947159 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wdfw\" (UniqueName: \"kubernetes.io/projected/3b12b40f-da8a-46d3-a34f-9a1bb0420656-kube-api-access-5wdfw\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:13:58.947197 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:58.947174 4982 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b12b40f-da8a-46d3-a34f-9a1bb0420656-openshift-service-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:13:59.038819 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:59.038779 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:59.103053 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:59.102139 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-25917498-openshift-default-77dc9bb847-smvqn"] Oct 30 14:13:59.103053 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:59.102445 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-25917498-openshift-default-77dc9bb847-smvqn" podUID="8104c709-90b5-4c71-9540-a766b2add736" containerName="istio-proxy" containerID="cri-o://c54d7dce00f9fcc8da0dff7036529e72da6b485dd9c7e6b6c9056cb20000656c" gracePeriod=30 Oct 30 14:13:59.145194 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:59.145153 4982 generic.go:356] "Generic (PLEG): container finished" podID="3b12b40f-da8a-46d3-a34f-9a1bb0420656" containerID="3cc7fde94e17aafe1319820a07bdd7990b30e1cb256ef7e0752048c75be9c49d" exitCode=0 Oct 30 14:13:59.146001 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:59.145927 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-25917498-7b65854ffc-57t5n" Oct 30 14:13:59.150076 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:59.149855 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-25917498-7b65854ffc-57t5n" event={"ID":"3b12b40f-da8a-46d3-a34f-9a1bb0420656","Type":"ContainerDied","Data":"3cc7fde94e17aafe1319820a07bdd7990b30e1cb256ef7e0752048c75be9c49d"} Oct 30 14:13:59.150076 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:59.149920 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-25917498-7b65854ffc-57t5n" event={"ID":"3b12b40f-da8a-46d3-a34f-9a1bb0420656","Type":"ContainerDied","Data":"8c2f0ca2c1ebf6625d602c75f530b189c5f26cf3ca1eb10500f0da30dc7c212a"} Oct 30 14:13:59.150076 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:59.149947 4982 scope.go:122] "RemoveContainer" containerID="3cc7fde94e17aafe1319820a07bdd7990b30e1cb256ef7e0752048c75be9c49d" Oct 30 14:13:59.182574 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:59.178097 4982 scope.go:122] "RemoveContainer" containerID="3cc7fde94e17aafe1319820a07bdd7990b30e1cb256ef7e0752048c75be9c49d" Oct 30 14:13:59.196773 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:13:59.196373 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cc7fde94e17aafe1319820a07bdd7990b30e1cb256ef7e0752048c75be9c49d\": container with ID starting with 3cc7fde94e17aafe1319820a07bdd7990b30e1cb256ef7e0752048c75be9c49d not found: ID does not exist" containerID="3cc7fde94e17aafe1319820a07bdd7990b30e1cb256ef7e0752048c75be9c49d" Oct 30 14:13:59.196773 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:59.196425 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cc7fde94e17aafe1319820a07bdd7990b30e1cb256ef7e0752048c75be9c49d"} err="failed to get container status \"3cc7fde94e17aafe1319820a07bdd7990b30e1cb256ef7e0752048c75be9c49d\": rpc error: code = NotFound desc = could not find container \"3cc7fde94e17aafe1319820a07bdd7990b30e1cb256ef7e0752048c75be9c49d\": container with ID starting with 3cc7fde94e17aafe1319820a07bdd7990b30e1cb256ef7e0752048c75be9c49d not found: ID does not exist" Oct 30 14:13:59.233679 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:59.233394 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:13:59.237534 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:59.234702 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-25917498-7b65854ffc-57t5n"] Oct 30 14:13:59.240979 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:13:59.240940 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-25917498-7b65854ffc-57t5n"] Oct 30 14:14:00.400524 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:00.400490 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b12b40f-da8a-46d3-a34f-9a1bb0420656" path="/var/lib/kubelet/pods/3b12b40f-da8a-46d3-a34f-9a1bb0420656/volumes" Oct 30 14:14:01.033496 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:01.033459 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-75443e-6b6cd45ccb-fnql4" Oct 30 14:14:01.033752 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:01.033538 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-75443e-7c475f6fbc-d7wdc" Oct 30 14:14:01.126575 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:01.126537 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:02.972831 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:02.972793 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:03.471411 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:03.471376 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:04.446900 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:04.446875 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-25917498-openshift-default-77dc9bb847-smvqn" Oct 30 14:14:04.548977 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:04.548933 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8104c709-90b5-4c71-9540-a766b2add736-kube-api-access-jhtfn\" (UniqueName: \"kubernetes.io/projected/8104c709-90b5-4c71-9540-a766b2add736-kube-api-access-jhtfn\") pod \"8104c709-90b5-4c71-9540-a766b2add736\" (UID: \"8104c709-90b5-4c71-9540-a766b2add736\") " Oct 30 14:14:04.549165 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:04.549131 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8104c709-90b5-4c71-9540-a766b2add736-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8104c709-90b5-4c71-9540-a766b2add736-workload-socket\") pod \"8104c709-90b5-4c71-9540-a766b2add736\" (UID: \"8104c709-90b5-4c71-9540-a766b2add736\") " Oct 30 14:14:04.549237 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:04.549173 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8104c709-90b5-4c71-9540-a766b2add736-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8104c709-90b5-4c71-9540-a766b2add736-istio-envoy\") pod \"8104c709-90b5-4c71-9540-a766b2add736\" (UID: \"8104c709-90b5-4c71-9540-a766b2add736\") " Oct 30 14:14:04.549291 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:04.549260 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8104c709-90b5-4c71-9540-a766b2add736-istio-token\" (UniqueName: \"kubernetes.io/projected/8104c709-90b5-4c71-9540-a766b2add736-istio-token\") pod \"8104c709-90b5-4c71-9540-a766b2add736\" (UID: \"8104c709-90b5-4c71-9540-a766b2add736\") " Oct 30 14:14:04.549352 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:04.549298 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8104c709-90b5-4c71-9540-a766b2add736-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8104c709-90b5-4c71-9540-a766b2add736-istio-podinfo\") pod \"8104c709-90b5-4c71-9540-a766b2add736\" (UID: \"8104c709-90b5-4c71-9540-a766b2add736\") " Oct 30 14:14:04.549463 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:04.549434 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8104c709-90b5-4c71-9540-a766b2add736-workload-socket" pod "8104c709-90b5-4c71-9540-a766b2add736" (UID: "8104c709-90b5-4c71-9540-a766b2add736"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:14:04.549547 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:04.549442 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8104c709-90b5-4c71-9540-a766b2add736-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8104c709-90b5-4c71-9540-a766b2add736-istio-data\") pod \"8104c709-90b5-4c71-9540-a766b2add736\" (UID: \"8104c709-90b5-4c71-9540-a766b2add736\") " Oct 30 14:14:04.549547 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:04.549525 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8104c709-90b5-4c71-9540-a766b2add736-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8104c709-90b5-4c71-9540-a766b2add736-istiod-ca-cert\") pod \"8104c709-90b5-4c71-9540-a766b2add736\" (UID: \"8104c709-90b5-4c71-9540-a766b2add736\") " Oct 30 14:14:04.549676 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:04.549618 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8104c709-90b5-4c71-9540-a766b2add736-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8104c709-90b5-4c71-9540-a766b2add736-workload-certs\") pod \"8104c709-90b5-4c71-9540-a766b2add736\" (UID: \"8104c709-90b5-4c71-9540-a766b2add736\") " Oct 30 14:14:04.549741 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:04.549676 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8104c709-90b5-4c71-9540-a766b2add736-istio-data" pod "8104c709-90b5-4c71-9540-a766b2add736" (UID: "8104c709-90b5-4c71-9540-a766b2add736"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:14:04.549741 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:04.549705 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8104c709-90b5-4c71-9540-a766b2add736-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8104c709-90b5-4c71-9540-a766b2add736-credential-socket\") pod \"8104c709-90b5-4c71-9540-a766b2add736\" (UID: \"8104c709-90b5-4c71-9540-a766b2add736\") " Oct 30 14:14:04.549900 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:04.549882 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8104c709-90b5-4c71-9540-a766b2add736-workload-certs" pod "8104c709-90b5-4c71-9540-a766b2add736" (UID: "8104c709-90b5-4c71-9540-a766b2add736"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:14:04.550002 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:04.549981 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8104c709-90b5-4c71-9540-a766b2add736-istiod-ca-cert" pod "8104c709-90b5-4c71-9540-a766b2add736" (UID: "8104c709-90b5-4c71-9540-a766b2add736"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:14:04.550154 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:04.550135 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8104c709-90b5-4c71-9540-a766b2add736-credential-socket" pod "8104c709-90b5-4c71-9540-a766b2add736" (UID: "8104c709-90b5-4c71-9540-a766b2add736"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:14:04.550883 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:04.550859 4982 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8104c709-90b5-4c71-9540-a766b2add736-istio-data\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:14:04.551023 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:04.550888 4982 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8104c709-90b5-4c71-9540-a766b2add736-istiod-ca-cert\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:14:04.551023 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:04.550903 4982 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8104c709-90b5-4c71-9540-a766b2add736-workload-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:14:04.551023 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:04.550917 4982 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8104c709-90b5-4c71-9540-a766b2add736-credential-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:14:04.551023 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:04.550929 4982 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8104c709-90b5-4c71-9540-a766b2add736-workload-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:14:04.551929 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:04.551908 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8104c709-90b5-4c71-9540-a766b2add736-istio-envoy" pod "8104c709-90b5-4c71-9540-a766b2add736" (UID: "8104c709-90b5-4c71-9540-a766b2add736"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:14:04.553274 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:04.553244 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8104c709-90b5-4c71-9540-a766b2add736-kube-api-access-jhtfn" pod "8104c709-90b5-4c71-9540-a766b2add736" (UID: "8104c709-90b5-4c71-9540-a766b2add736"). InnerVolumeSpecName "kube-api-access-jhtfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:14:04.553373 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:04.553297 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8104c709-90b5-4c71-9540-a766b2add736-istio-token" pod "8104c709-90b5-4c71-9540-a766b2add736" (UID: "8104c709-90b5-4c71-9540-a766b2add736"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:14:04.553911 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:04.553891 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8104c709-90b5-4c71-9540-a766b2add736-istio-podinfo" pod "8104c709-90b5-4c71-9540-a766b2add736" (UID: "8104c709-90b5-4c71-9540-a766b2add736"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:14:04.652610 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:04.652561 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhtfn\" (UniqueName: \"kubernetes.io/projected/8104c709-90b5-4c71-9540-a766b2add736-kube-api-access-jhtfn\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:14:04.652610 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:04.652604 4982 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8104c709-90b5-4c71-9540-a766b2add736-istio-envoy\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:14:04.652854 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:04.652620 4982 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8104c709-90b5-4c71-9540-a766b2add736-istio-token\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:14:04.652854 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:04.652634 4982 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8104c709-90b5-4c71-9540-a766b2add736-istio-podinfo\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:14:04.825170 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:04.825131 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:04.990235 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:04.990190 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:05.216247 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:05.216169 4982 generic.go:356] "Generic (PLEG): container finished" podID="8104c709-90b5-4c71-9540-a766b2add736" containerID="c54d7dce00f9fcc8da0dff7036529e72da6b485dd9c7e6b6c9056cb20000656c" exitCode=0 Oct 30 14:14:05.216247 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:05.216231 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-25917498-openshift-default-77dc9bb847-smvqn" Oct 30 14:14:05.216486 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:05.216250 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-25917498-openshift-default-77dc9bb847-smvqn" event={"ID":"8104c709-90b5-4c71-9540-a766b2add736","Type":"ContainerDied","Data":"c54d7dce00f9fcc8da0dff7036529e72da6b485dd9c7e6b6c9056cb20000656c"} Oct 30 14:14:05.216486 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:05.216338 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-25917498-openshift-default-77dc9bb847-smvqn" event={"ID":"8104c709-90b5-4c71-9540-a766b2add736","Type":"ContainerDied","Data":"d0bb23cfedcb7891fbc6fb93fbf134f11b4c2d7231baab311a24f04fdb2b40eb"} Oct 30 14:14:05.216486 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:05.216362 4982 scope.go:122] "RemoveContainer" containerID="c54d7dce00f9fcc8da0dff7036529e72da6b485dd9c7e6b6c9056cb20000656c" Oct 30 14:14:05.228065 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:05.228048 4982 scope.go:122] "RemoveContainer" containerID="e5f5e55f4a55842ed1cc2b3e9fa9f75aa58b37706b78d0b3ae43c10f6900c918" Oct 30 14:14:05.238131 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:05.238108 4982 scope.go:122] "RemoveContainer" containerID="c54d7dce00f9fcc8da0dff7036529e72da6b485dd9c7e6b6c9056cb20000656c" Oct 30 14:14:05.238419 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:14:05.238401 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c54d7dce00f9fcc8da0dff7036529e72da6b485dd9c7e6b6c9056cb20000656c\": container with ID starting with c54d7dce00f9fcc8da0dff7036529e72da6b485dd9c7e6b6c9056cb20000656c not found: ID does not exist" containerID="c54d7dce00f9fcc8da0dff7036529e72da6b485dd9c7e6b6c9056cb20000656c" Oct 30 14:14:05.238467 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:05.238428 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c54d7dce00f9fcc8da0dff7036529e72da6b485dd9c7e6b6c9056cb20000656c"} err="failed to get container status \"c54d7dce00f9fcc8da0dff7036529e72da6b485dd9c7e6b6c9056cb20000656c\": rpc error: code = NotFound desc = could not find container \"c54d7dce00f9fcc8da0dff7036529e72da6b485dd9c7e6b6c9056cb20000656c\": container with ID starting with c54d7dce00f9fcc8da0dff7036529e72da6b485dd9c7e6b6c9056cb20000656c not found: ID does not exist" Oct 30 14:14:05.238467 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:05.238444 4982 scope.go:122] "RemoveContainer" containerID="e5f5e55f4a55842ed1cc2b3e9fa9f75aa58b37706b78d0b3ae43c10f6900c918" Oct 30 14:14:05.238715 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:14:05.238696 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5f5e55f4a55842ed1cc2b3e9fa9f75aa58b37706b78d0b3ae43c10f6900c918\": container with ID starting with e5f5e55f4a55842ed1cc2b3e9fa9f75aa58b37706b78d0b3ae43c10f6900c918 not found: ID does not exist" containerID="e5f5e55f4a55842ed1cc2b3e9fa9f75aa58b37706b78d0b3ae43c10f6900c918" Oct 30 14:14:05.238770 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:05.238721 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5f5e55f4a55842ed1cc2b3e9fa9f75aa58b37706b78d0b3ae43c10f6900c918"} err="failed to get container status \"e5f5e55f4a55842ed1cc2b3e9fa9f75aa58b37706b78d0b3ae43c10f6900c918\": rpc error: code = NotFound desc = could not find container \"e5f5e55f4a55842ed1cc2b3e9fa9f75aa58b37706b78d0b3ae43c10f6900c918\": container with ID starting with e5f5e55f4a55842ed1cc2b3e9fa9f75aa58b37706b78d0b3ae43c10f6900c918 not found: ID does not exist" Oct 30 14:14:05.282070 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:05.282045 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-25917498-openshift-default-77dc9bb847-smvqn"] Oct 30 14:14:05.296933 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:05.296908 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-25917498-openshift-default-77dc9bb847-smvqn"] Oct 30 14:14:06.400314 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:06.400263 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8104c709-90b5-4c71-9540-a766b2add736" path="/var/lib/kubelet/pods/8104c709-90b5-4c71-9540-a766b2add736/volumes" Oct 30 14:14:06.954513 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:06.954477 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:10.573949 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:10.573915 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:11.147008 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:11.146972 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:13.191515 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:13.188398 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:15.303010 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:15.302971 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:15.977574 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:15.977540 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:16.426393 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:16.426356 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg"] Oct 30 14:14:16.427885 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:16.427855 4982 cpu_manager.go:414] "containerMap: removing container" podUID="8104c709-90b5-4c71-9540-a766b2add736" containerName="istio-proxy" Oct 30 14:14:16.428034 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:16.428020 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8104c709-90b5-4c71-9540-a766b2add736" containerName="istio-proxy" Oct 30 14:14:16.428151 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:16.428141 4982 cpu_manager.go:414] "containerMap: removing container" podUID="3b12b40f-da8a-46d3-a34f-9a1bb0420656" containerName="maas-api" Oct 30 14:14:16.428223 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:16.428214 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b12b40f-da8a-46d3-a34f-9a1bb0420656" containerName="maas-api" Oct 30 14:14:16.428304 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:16.428295 4982 cpu_manager.go:414] "containerMap: removing container" podUID="8104c709-90b5-4c71-9540-a766b2add736" containerName="istio-proxy" Oct 30 14:14:16.428392 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:16.428380 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8104c709-90b5-4c71-9540-a766b2add736" containerName="istio-proxy" Oct 30 14:14:16.428651 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:16.428623 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="8104c709-90b5-4c71-9540-a766b2add736" containerName="istio-proxy" Oct 30 14:14:16.428756 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:16.428745 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b12b40f-da8a-46d3-a34f-9a1bb0420656" containerName="maas-api" Oct 30 14:14:16.428837 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:16.428828 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="8104c709-90b5-4c71-9540-a766b2add736" containerName="istio-proxy" Oct 30 14:14:16.433442 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:16.433418 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg" Oct 30 14:14:16.456595 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:16.456564 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg"] Oct 30 14:14:16.468427 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:16.465260 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-9dba4af9-openshift-default-dockercfg-qf6sh\"" Oct 30 14:14:16.499160 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:16.499088 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:16.508442 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:16.508410 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7057acc6-78aa-43fa-bc57-6e5ecacf2e50-istio-podinfo\") pod \"e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg\" (UID: \"7057acc6-78aa-43fa-bc57-6e5ecacf2e50\") " pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg" Oct 30 14:14:16.508705 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:16.508685 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7057acc6-78aa-43fa-bc57-6e5ecacf2e50-credential-socket\") pod \"e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg\" (UID: \"7057acc6-78aa-43fa-bc57-6e5ecacf2e50\") " pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg" Oct 30 14:14:16.508932 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:16.508915 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7057acc6-78aa-43fa-bc57-6e5ecacf2e50-istio-data\") pod \"e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg\" (UID: \"7057acc6-78aa-43fa-bc57-6e5ecacf2e50\") " pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg" Oct 30 14:14:16.509126 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:16.509109 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7057acc6-78aa-43fa-bc57-6e5ecacf2e50-istio-token\") pod \"e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg\" (UID: \"7057acc6-78aa-43fa-bc57-6e5ecacf2e50\") " pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg" Oct 30 14:14:16.509368 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:16.509353 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7057acc6-78aa-43fa-bc57-6e5ecacf2e50-istio-envoy\") pod \"e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg\" (UID: \"7057acc6-78aa-43fa-bc57-6e5ecacf2e50\") " pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg" Oct 30 14:14:16.509685 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:16.509631 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7057acc6-78aa-43fa-bc57-6e5ecacf2e50-workload-certs\") pod \"e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg\" (UID: \"7057acc6-78aa-43fa-bc57-6e5ecacf2e50\") " pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg" Oct 30 14:14:16.509824 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:16.509810 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7057acc6-78aa-43fa-bc57-6e5ecacf2e50-istiod-ca-cert\") pod \"e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg\" (UID: \"7057acc6-78aa-43fa-bc57-6e5ecacf2e50\") " pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg" Oct 30 14:14:16.510013 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:16.509996 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcxpd\" (UniqueName: \"kubernetes.io/projected/7057acc6-78aa-43fa-bc57-6e5ecacf2e50-kube-api-access-wcxpd\") pod \"e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg\" (UID: \"7057acc6-78aa-43fa-bc57-6e5ecacf2e50\") " pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg" Oct 30 14:14:16.510176 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:16.510144 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7057acc6-78aa-43fa-bc57-6e5ecacf2e50-workload-socket\") pod \"e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg\" (UID: \"7057acc6-78aa-43fa-bc57-6e5ecacf2e50\") " pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg" Oct 30 14:14:16.643665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:16.629512 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7057acc6-78aa-43fa-bc57-6e5ecacf2e50-istio-podinfo\") pod \"e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg\" (UID: \"7057acc6-78aa-43fa-bc57-6e5ecacf2e50\") " pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg" Oct 30 14:14:16.643665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:16.629582 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7057acc6-78aa-43fa-bc57-6e5ecacf2e50-credential-socket\") pod \"e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg\" (UID: \"7057acc6-78aa-43fa-bc57-6e5ecacf2e50\") " pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg" Oct 30 14:14:16.643665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:16.629658 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7057acc6-78aa-43fa-bc57-6e5ecacf2e50-istio-data\") pod \"e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg\" (UID: \"7057acc6-78aa-43fa-bc57-6e5ecacf2e50\") " pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg" Oct 30 14:14:16.643665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:16.629721 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7057acc6-78aa-43fa-bc57-6e5ecacf2e50-istio-token\") pod \"e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg\" (UID: \"7057acc6-78aa-43fa-bc57-6e5ecacf2e50\") " pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg" Oct 30 14:14:16.643665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:16.629849 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7057acc6-78aa-43fa-bc57-6e5ecacf2e50-istio-envoy\") pod \"e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg\" (UID: \"7057acc6-78aa-43fa-bc57-6e5ecacf2e50\") " pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg" Oct 30 14:14:16.643665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:16.630026 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7057acc6-78aa-43fa-bc57-6e5ecacf2e50-workload-certs\") pod \"e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg\" (UID: \"7057acc6-78aa-43fa-bc57-6e5ecacf2e50\") " pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg" Oct 30 14:14:16.643665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:16.630058 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7057acc6-78aa-43fa-bc57-6e5ecacf2e50-istiod-ca-cert\") pod \"e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg\" (UID: \"7057acc6-78aa-43fa-bc57-6e5ecacf2e50\") " pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg" Oct 30 14:14:16.643665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:16.635740 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcxpd\" (UniqueName: \"kubernetes.io/projected/7057acc6-78aa-43fa-bc57-6e5ecacf2e50-kube-api-access-wcxpd\") pod \"e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg\" (UID: \"7057acc6-78aa-43fa-bc57-6e5ecacf2e50\") " pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg" Oct 30 14:14:16.643665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:16.635778 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7057acc6-78aa-43fa-bc57-6e5ecacf2e50-workload-socket\") pod \"e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg\" (UID: \"7057acc6-78aa-43fa-bc57-6e5ecacf2e50\") " pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg" Oct 30 14:14:16.643665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:16.636301 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7057acc6-78aa-43fa-bc57-6e5ecacf2e50-workload-socket\") pod \"e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg\" (UID: \"7057acc6-78aa-43fa-bc57-6e5ecacf2e50\") " pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg" Oct 30 14:14:16.643665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:16.637039 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7057acc6-78aa-43fa-bc57-6e5ecacf2e50-workload-certs\") pod \"e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg\" (UID: \"7057acc6-78aa-43fa-bc57-6e5ecacf2e50\") " pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg" Oct 30 14:14:16.643665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:16.637270 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7057acc6-78aa-43fa-bc57-6e5ecacf2e50-credential-socket\") pod \"e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg\" (UID: \"7057acc6-78aa-43fa-bc57-6e5ecacf2e50\") " pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg" Oct 30 14:14:16.643665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:16.637487 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7057acc6-78aa-43fa-bc57-6e5ecacf2e50-istio-data\") pod \"e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg\" (UID: \"7057acc6-78aa-43fa-bc57-6e5ecacf2e50\") " pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg" Oct 30 14:14:16.643665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:16.642615 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7057acc6-78aa-43fa-bc57-6e5ecacf2e50-istiod-ca-cert\") pod \"e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg\" (UID: \"7057acc6-78aa-43fa-bc57-6e5ecacf2e50\") " pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg" Oct 30 14:14:16.664708 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:16.664450 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7057acc6-78aa-43fa-bc57-6e5ecacf2e50-istio-podinfo\") pod \"e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg\" (UID: \"7057acc6-78aa-43fa-bc57-6e5ecacf2e50\") " pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg" Oct 30 14:14:16.680707 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:16.671250 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7057acc6-78aa-43fa-bc57-6e5ecacf2e50-istio-envoy\") pod \"e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg\" (UID: \"7057acc6-78aa-43fa-bc57-6e5ecacf2e50\") " pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg" Oct 30 14:14:16.688743 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:16.688713 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcxpd\" (UniqueName: \"kubernetes.io/projected/7057acc6-78aa-43fa-bc57-6e5ecacf2e50-kube-api-access-wcxpd\") pod \"e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg\" (UID: \"7057acc6-78aa-43fa-bc57-6e5ecacf2e50\") " pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg" Oct 30 14:14:16.704663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:16.704567 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7057acc6-78aa-43fa-bc57-6e5ecacf2e50-istio-token\") pod \"e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg\" (UID: \"7057acc6-78aa-43fa-bc57-6e5ecacf2e50\") " pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg" Oct 30 14:14:16.752396 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:16.751869 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg" Oct 30 14:14:17.212684 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:17.212021 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg"] Oct 30 14:14:17.217190 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:17.217154 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:14:17.217449 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:17.217244 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:14:17.217449 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:17.217289 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:14:17.319289 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:17.319218 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg" event={"ID":"7057acc6-78aa-43fa-bc57-6e5ecacf2e50","Type":"ContainerStarted","Data":"ad6543ac955d66904b53a21d988117a4636a57451e9d2407dd6c912eace2e435"} Oct 30 14:14:17.319561 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:17.319517 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg" event={"ID":"7057acc6-78aa-43fa-bc57-6e5ecacf2e50","Type":"ContainerStarted","Data":"9c51b51be948bfd3b56f33003ee7a8e0018774021addfc1875c77b95ba271580"} Oct 30 14:14:17.356773 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:17.356737 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:17.384396 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:17.384333 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg" podStartSLOduration=1.3843070499999999 podStartE2EDuration="1.38430705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:14:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:14:17.380846348 +0000 UTC m=+1527.124980395" watchObservedRunningTime="2026-10-30 14:14:17.38430705 +0000 UTC m=+1527.128441099" Oct 30 14:14:17.753760 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:17.753724 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg" Oct 30 14:14:17.981369 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:17.981320 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:18.378344 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:18.378290 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:18.560709 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:18.560670 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:18.758784 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:18.758705 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg" Oct 30 14:14:19.345742 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:19.345703 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg" Oct 30 14:14:19.346917 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:19.346892 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg" Oct 30 14:14:20.637792 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:20.637759 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:22.532769 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:22.532730 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:24.731444 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:24.731411 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:25.355944 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:25.355908 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:26.803588 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:26.803549 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:28.012626 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:14:28.012595 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:14:29.101484 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:29.101423 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:29.368549 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:29.368459 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:30.349828 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:30.349791 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:30.852679 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:30.852623 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:31.093331 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:31.093297 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9dba4af9-6d98fd4964-4fmbr"] Oct 30 14:14:31.099892 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:31.099865 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9dba4af9-6d98fd4964-4fmbr" Oct 30 14:14:31.135121 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:31.135037 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-9dba4af9-dockercfg-r7jw7\"" Oct 30 14:14:31.135474 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:31.135444 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-9dba4af9\"" Oct 30 14:14:31.188700 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:31.182458 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w7gt\" (UniqueName: \"kubernetes.io/projected/83f22477-fe63-43ef-9752-9cddc4c93627-kube-api-access-5w7gt\") pod \"payload-processing-e2e-mt-9dba4af9-6d98fd4964-4fmbr\" (UID: \"83f22477-fe63-43ef-9752-9cddc4c93627\") " pod="openshift-ingress/payload-processing-e2e-mt-9dba4af9-6d98fd4964-4fmbr" Oct 30 14:14:31.188700 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:31.182564 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83f22477-fe63-43ef-9752-9cddc4c93627-plugins-config-volume\") pod \"payload-processing-e2e-mt-9dba4af9-6d98fd4964-4fmbr\" (UID: \"83f22477-fe63-43ef-9752-9cddc4c93627\") " pod="openshift-ingress/payload-processing-e2e-mt-9dba4af9-6d98fd4964-4fmbr" Oct 30 14:14:31.228264 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:31.228173 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9dba4af9-6d98fd4964-4fmbr"] Oct 30 14:14:31.284840 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:31.284787 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w7gt\" (UniqueName: \"kubernetes.io/projected/83f22477-fe63-43ef-9752-9cddc4c93627-kube-api-access-5w7gt\") pod \"payload-processing-e2e-mt-9dba4af9-6d98fd4964-4fmbr\" (UID: \"83f22477-fe63-43ef-9752-9cddc4c93627\") " pod="openshift-ingress/payload-processing-e2e-mt-9dba4af9-6d98fd4964-4fmbr" Oct 30 14:14:31.285267 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:31.285239 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83f22477-fe63-43ef-9752-9cddc4c93627-plugins-config-volume\") pod \"payload-processing-e2e-mt-9dba4af9-6d98fd4964-4fmbr\" (UID: \"83f22477-fe63-43ef-9752-9cddc4c93627\") " pod="openshift-ingress/payload-processing-e2e-mt-9dba4af9-6d98fd4964-4fmbr" Oct 30 14:14:31.286296 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:31.286267 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83f22477-fe63-43ef-9752-9cddc4c93627-plugins-config-volume\") pod \"payload-processing-e2e-mt-9dba4af9-6d98fd4964-4fmbr\" (UID: \"83f22477-fe63-43ef-9752-9cddc4c93627\") " pod="openshift-ingress/payload-processing-e2e-mt-9dba4af9-6d98fd4964-4fmbr" Oct 30 14:14:31.329972 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:31.329942 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w7gt\" (UniqueName: \"kubernetes.io/projected/83f22477-fe63-43ef-9752-9cddc4c93627-kube-api-access-5w7gt\") pod \"payload-processing-e2e-mt-9dba4af9-6d98fd4964-4fmbr\" (UID: \"83f22477-fe63-43ef-9752-9cddc4c93627\") " pod="openshift-ingress/payload-processing-e2e-mt-9dba4af9-6d98fd4964-4fmbr" Oct 30 14:14:31.532618 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:31.532526 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9dba4af9-6d98fd4964-4fmbr" Oct 30 14:14:31.872678 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:14:31.872607 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83f22477_fe63_43ef_9752_9cddc4c93627.slice/crio-3ba46ea7b6865035b0cd8c38bf9f88cfdbfd1a2af83b91c52aeac690886a7592 WatchSource:0}: Error finding container 3ba46ea7b6865035b0cd8c38bf9f88cfdbfd1a2af83b91c52aeac690886a7592: Status 404 returned error can't find the container with id 3ba46ea7b6865035b0cd8c38bf9f88cfdbfd1a2af83b91c52aeac690886a7592 Oct 30 14:14:31.872955 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:31.872929 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9dba4af9-6d98fd4964-4fmbr"] Oct 30 14:14:32.186268 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:32.186130 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:32.339765 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:32.339705 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:32.434314 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:32.434271 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9dba4af9-6d98fd4964-4fmbr" event={"ID":"83f22477-fe63-43ef-9752-9cddc4c93627","Type":"ContainerStarted","Data":"06c105066840871b3164e6e308d1e37ca90f661bdd7b66e1994946f339485dd3"} Oct 30 14:14:32.434507 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:32.434332 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9dba4af9-6d98fd4964-4fmbr" event={"ID":"83f22477-fe63-43ef-9752-9cddc4c93627","Type":"ContainerStarted","Data":"3ba46ea7b6865035b0cd8c38bf9f88cfdbfd1a2af83b91c52aeac690886a7592"} Oct 30 14:14:32.434507 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:32.434365 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-9dba4af9-6d98fd4964-4fmbr" Oct 30 14:14:32.493868 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:32.493752 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-9dba4af9-6d98fd4964-4fmbr" podStartSLOduration=1.493735268 podStartE2EDuration="1.493735268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:14:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:14:32.488801575 +0000 UTC m=+1542.232935618" watchObservedRunningTime="2026-10-30 14:14:32.493735268 +0000 UTC m=+1542.237869311" Oct 30 14:14:32.785037 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:32.783633 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:33.089141 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:33.089108 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:33.239811 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:33.239776 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b"] Oct 30 14:14:33.244552 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:33.244526 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b" Oct 30 14:14:33.259958 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:33.259929 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:33.277471 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:33.277436 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-0efe43-openshift-default-dockercfg-bthxx\"" Oct 30 14:14:33.282581 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:33.282551 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b"] Oct 30 14:14:33.340103 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:33.340019 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c99896fe-79b1-4368-aeb5-a81182e7ae13-istio-data\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b\" (UID: \"c99896fe-79b1-4368-aeb5-a81182e7ae13\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b" Oct 30 14:14:33.340267 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:33.340174 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c99896fe-79b1-4368-aeb5-a81182e7ae13-istio-envoy\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b\" (UID: \"c99896fe-79b1-4368-aeb5-a81182e7ae13\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b" Oct 30 14:14:33.340267 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:33.340225 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c99896fe-79b1-4368-aeb5-a81182e7ae13-istio-podinfo\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b\" (UID: \"c99896fe-79b1-4368-aeb5-a81182e7ae13\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b" Oct 30 14:14:33.340365 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:33.340345 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c99896fe-79b1-4368-aeb5-a81182e7ae13-workload-socket\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b\" (UID: \"c99896fe-79b1-4368-aeb5-a81182e7ae13\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b" Oct 30 14:14:33.340400 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:33.340390 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c99896fe-79b1-4368-aeb5-a81182e7ae13-istio-token\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b\" (UID: \"c99896fe-79b1-4368-aeb5-a81182e7ae13\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b" Oct 30 14:14:33.340463 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:33.340446 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c99896fe-79b1-4368-aeb5-a81182e7ae13-istiod-ca-cert\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b\" (UID: \"c99896fe-79b1-4368-aeb5-a81182e7ae13\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b" Oct 30 14:14:33.340572 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:33.340556 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c99896fe-79b1-4368-aeb5-a81182e7ae13-workload-certs\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b\" (UID: \"c99896fe-79b1-4368-aeb5-a81182e7ae13\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b" Oct 30 14:14:33.340627 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:33.340598 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c99896fe-79b1-4368-aeb5-a81182e7ae13-credential-socket\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b\" (UID: \"c99896fe-79b1-4368-aeb5-a81182e7ae13\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b" Oct 30 14:14:33.340711 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:33.340689 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkbww\" (UniqueName: \"kubernetes.io/projected/c99896fe-79b1-4368-aeb5-a81182e7ae13-kube-api-access-zkbww\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b\" (UID: \"c99896fe-79b1-4368-aeb5-a81182e7ae13\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b" Oct 30 14:14:33.446163 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:33.446127 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c99896fe-79b1-4368-aeb5-a81182e7ae13-istio-envoy\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b\" (UID: \"c99896fe-79b1-4368-aeb5-a81182e7ae13\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b" Oct 30 14:14:33.446472 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:33.446189 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c99896fe-79b1-4368-aeb5-a81182e7ae13-istio-podinfo\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b\" (UID: \"c99896fe-79b1-4368-aeb5-a81182e7ae13\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b" Oct 30 14:14:33.446472 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:33.446212 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c99896fe-79b1-4368-aeb5-a81182e7ae13-workload-socket\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b\" (UID: \"c99896fe-79b1-4368-aeb5-a81182e7ae13\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b" Oct 30 14:14:33.446472 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:33.446237 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c99896fe-79b1-4368-aeb5-a81182e7ae13-istio-token\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b\" (UID: \"c99896fe-79b1-4368-aeb5-a81182e7ae13\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b" Oct 30 14:14:33.446472 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:33.446281 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c99896fe-79b1-4368-aeb5-a81182e7ae13-istiod-ca-cert\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b\" (UID: \"c99896fe-79b1-4368-aeb5-a81182e7ae13\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b" Oct 30 14:14:33.446472 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:33.446351 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c99896fe-79b1-4368-aeb5-a81182e7ae13-workload-certs\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b\" (UID: \"c99896fe-79b1-4368-aeb5-a81182e7ae13\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b" Oct 30 14:14:33.446472 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:33.446385 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c99896fe-79b1-4368-aeb5-a81182e7ae13-credential-socket\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b\" (UID: \"c99896fe-79b1-4368-aeb5-a81182e7ae13\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b" Oct 30 14:14:33.446472 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:33.446453 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkbww\" (UniqueName: \"kubernetes.io/projected/c99896fe-79b1-4368-aeb5-a81182e7ae13-kube-api-access-zkbww\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b\" (UID: \"c99896fe-79b1-4368-aeb5-a81182e7ae13\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b" Oct 30 14:14:33.446910 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:33.446521 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c99896fe-79b1-4368-aeb5-a81182e7ae13-istio-data\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b\" (UID: \"c99896fe-79b1-4368-aeb5-a81182e7ae13\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b" Oct 30 14:14:33.446969 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:33.446903 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c99896fe-79b1-4368-aeb5-a81182e7ae13-istio-data\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b\" (UID: \"c99896fe-79b1-4368-aeb5-a81182e7ae13\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b" Oct 30 14:14:33.448619 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:33.448592 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c99896fe-79b1-4368-aeb5-a81182e7ae13-istiod-ca-cert\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b\" (UID: \"c99896fe-79b1-4368-aeb5-a81182e7ae13\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b" Oct 30 14:14:33.450322 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:33.449427 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c99896fe-79b1-4368-aeb5-a81182e7ae13-workload-certs\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b\" (UID: \"c99896fe-79b1-4368-aeb5-a81182e7ae13\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b" Oct 30 14:14:33.450322 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:33.449604 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c99896fe-79b1-4368-aeb5-a81182e7ae13-credential-socket\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b\" (UID: \"c99896fe-79b1-4368-aeb5-a81182e7ae13\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b" Oct 30 14:14:33.450322 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:33.449813 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c99896fe-79b1-4368-aeb5-a81182e7ae13-workload-socket\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b\" (UID: \"c99896fe-79b1-4368-aeb5-a81182e7ae13\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b" Oct 30 14:14:33.455818 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:33.451791 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c99896fe-79b1-4368-aeb5-a81182e7ae13-istio-envoy\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b\" (UID: \"c99896fe-79b1-4368-aeb5-a81182e7ae13\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b" Oct 30 14:14:33.455818 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:33.452005 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c99896fe-79b1-4368-aeb5-a81182e7ae13-istio-podinfo\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b\" (UID: \"c99896fe-79b1-4368-aeb5-a81182e7ae13\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b" Oct 30 14:14:33.475944 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:33.475914 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkbww\" (UniqueName: \"kubernetes.io/projected/c99896fe-79b1-4368-aeb5-a81182e7ae13-kube-api-access-zkbww\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b\" (UID: \"c99896fe-79b1-4368-aeb5-a81182e7ae13\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b" Oct 30 14:14:33.476554 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:33.476525 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c99896fe-79b1-4368-aeb5-a81182e7ae13-istio-token\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b\" (UID: \"c99896fe-79b1-4368-aeb5-a81182e7ae13\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b" Oct 30 14:14:33.486821 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:33.486792 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:33.562747 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:33.561848 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b" Oct 30 14:14:33.894666 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:33.894557 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:34.026893 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:34.026857 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b"] Oct 30 14:14:34.034487 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:14:34.034460 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc99896fe_79b1_4368_aeb5_a81182e7ae13.slice/crio-b55df68921fc279754136cb60172008d1f4ad1a6a137f6d3f8eb89954650a007 WatchSource:0}: Error finding container b55df68921fc279754136cb60172008d1f4ad1a6a137f6d3f8eb89954650a007: Status 404 returned error can't find the container with id b55df68921fc279754136cb60172008d1f4ad1a6a137f6d3f8eb89954650a007 Oct 30 14:14:34.037737 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:34.037687 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:14:34.037855 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:34.037782 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:14:34.037855 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:34.037830 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:14:34.397806 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:34.397715 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:34.468586 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:34.468547 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b" event={"ID":"c99896fe-79b1-4368-aeb5-a81182e7ae13","Type":"ContainerStarted","Data":"26ba0205e31735dc2452611301762a84e2575620db52858330c39d279e7b54f8"} Oct 30 14:14:34.468782 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:34.468618 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b" event={"ID":"c99896fe-79b1-4368-aeb5-a81182e7ae13","Type":"ContainerStarted","Data":"b55df68921fc279754136cb60172008d1f4ad1a6a137f6d3f8eb89954650a007"} Oct 30 14:14:34.562744 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:34.562703 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b" Oct 30 14:14:34.869083 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:34.869044 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:35.568843 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:35.568810 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b" Oct 30 14:14:35.635579 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:35.635524 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b" podStartSLOduration=2.635504794 podStartE2EDuration="2.635504794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:14:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:14:34.535516612 +0000 UTC m=+1544.279650655" watchObservedRunningTime="2026-10-30 14:14:35.635504794 +0000 UTC m=+1545.379638837" Oct 30 14:14:35.952214 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:35.952127 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:36.376492 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:36.376428 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:36.481010 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:36.480979 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b" Oct 30 14:14:36.482025 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:36.482000 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b" Oct 30 14:14:36.528253 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:36.528224 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:37.087095 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:37.087060 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:37.238852 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:37.238816 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:37.915276 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:37.915239 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:38.566909 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:38.566877 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:38.945834 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:38.945755 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:39.100369 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:39.100334 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:39.257559 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:39.257471 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:40.904736 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:40.904695 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:41.072760 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:41.072728 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:41.726785 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:41.726737 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:42.224009 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:42.223977 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:42.857930 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:42.857892 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:43.012240 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:43.012203 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:43.273455 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:43.273348 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:43.461517 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:43.461472 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-9dba4af9-6d98fd4964-4fmbr" Oct 30 14:14:45.172720 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:45.172682 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:46.730608 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:46.730567 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:47.101227 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:47.101189 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:47.381735 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:47.376284 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:48.762402 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:48.762365 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:48.919311 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:48.919275 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:49.494026 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:49.493996 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:49.649822 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:49.649787 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:51.535272 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:51.535230 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:53.552333 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:53.552290 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:53.876903 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:53.876873 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:54.062402 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:54.062349 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:54.521948 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:54.521910 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:54.697397 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:54.697367 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:55.537118 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:55.537079 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:57.330200 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:57.330165 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:57.534746 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:57.534690 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:58.014600 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:14:58.014568 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:14:58.084990 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:58.084955 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjqps"] Oct 30 14:14:58.090990 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:58.090968 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjqps" Oct 30 14:14:58.110317 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:58.110295 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 14:14:58.110317 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:58.110298 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 14:14:58.110476 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:58.110330 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 14:14:58.110476 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:58.110338 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-zzhpw\"" Oct 30 14:14:58.128765 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:58.128730 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjqps"] Oct 30 14:14:58.175889 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:58.175100 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9c00c972-1e46-4182-aaf4-163a6960fb70-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wjqps\" (UID: \"9c00c972-1e46-4182-aaf4-163a6960fb70\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjqps" Oct 30 14:14:58.175889 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:58.175216 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9c00c972-1e46-4182-aaf4-163a6960fb70-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wjqps\" (UID: \"9c00c972-1e46-4182-aaf4-163a6960fb70\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjqps" Oct 30 14:14:58.175889 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:58.175402 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9c00c972-1e46-4182-aaf4-163a6960fb70-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wjqps\" (UID: \"9c00c972-1e46-4182-aaf4-163a6960fb70\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjqps" Oct 30 14:14:58.175889 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:58.175487 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9c00c972-1e46-4182-aaf4-163a6960fb70-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wjqps\" (UID: \"9c00c972-1e46-4182-aaf4-163a6960fb70\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjqps" Oct 30 14:14:58.175889 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:58.175555 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c00c972-1e46-4182-aaf4-163a6960fb70-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wjqps\" (UID: \"9c00c972-1e46-4182-aaf4-163a6960fb70\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjqps" Oct 30 14:14:58.175889 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:58.175585 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zz5l\" (UniqueName: \"kubernetes.io/projected/9c00c972-1e46-4182-aaf4-163a6960fb70-kube-api-access-5zz5l\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wjqps\" (UID: \"9c00c972-1e46-4182-aaf4-163a6960fb70\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjqps" Oct 30 14:14:58.175889 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:58.175683 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c00c972-1e46-4182-aaf4-163a6960fb70-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wjqps\" (UID: \"9c00c972-1e46-4182-aaf4-163a6960fb70\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjqps" Oct 30 14:14:58.277560 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:58.277443 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c00c972-1e46-4182-aaf4-163a6960fb70-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wjqps\" (UID: \"9c00c972-1e46-4182-aaf4-163a6960fb70\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjqps" Oct 30 14:14:58.277560 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:58.277505 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9c00c972-1e46-4182-aaf4-163a6960fb70-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wjqps\" (UID: \"9c00c972-1e46-4182-aaf4-163a6960fb70\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjqps" Oct 30 14:14:58.277844 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:58.277591 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9c00c972-1e46-4182-aaf4-163a6960fb70-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wjqps\" (UID: \"9c00c972-1e46-4182-aaf4-163a6960fb70\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjqps" Oct 30 14:14:58.277844 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:58.277802 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9c00c972-1e46-4182-aaf4-163a6960fb70-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wjqps\" (UID: \"9c00c972-1e46-4182-aaf4-163a6960fb70\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjqps" Oct 30 14:14:58.277972 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:58.277882 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9c00c972-1e46-4182-aaf4-163a6960fb70-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wjqps\" (UID: \"9c00c972-1e46-4182-aaf4-163a6960fb70\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjqps" Oct 30 14:14:58.277972 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:58.277942 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c00c972-1e46-4182-aaf4-163a6960fb70-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wjqps\" (UID: \"9c00c972-1e46-4182-aaf4-163a6960fb70\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjqps" Oct 30 14:14:58.278073 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:58.277971 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zz5l\" (UniqueName: \"kubernetes.io/projected/9c00c972-1e46-4182-aaf4-163a6960fb70-kube-api-access-5zz5l\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wjqps\" (UID: \"9c00c972-1e46-4182-aaf4-163a6960fb70\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjqps" Oct 30 14:14:58.279569 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:58.279538 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9c00c972-1e46-4182-aaf4-163a6960fb70-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wjqps\" (UID: \"9c00c972-1e46-4182-aaf4-163a6960fb70\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjqps" Oct 30 14:14:58.281004 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:58.280981 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c00c972-1e46-4182-aaf4-163a6960fb70-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wjqps\" (UID: \"9c00c972-1e46-4182-aaf4-163a6960fb70\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjqps" Oct 30 14:14:58.282017 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:58.281985 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9c00c972-1e46-4182-aaf4-163a6960fb70-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wjqps\" (UID: \"9c00c972-1e46-4182-aaf4-163a6960fb70\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjqps" Oct 30 14:14:58.282580 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:58.282552 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9c00c972-1e46-4182-aaf4-163a6960fb70-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wjqps\" (UID: \"9c00c972-1e46-4182-aaf4-163a6960fb70\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjqps" Oct 30 14:14:58.283714 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:58.283693 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9c00c972-1e46-4182-aaf4-163a6960fb70-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wjqps\" (UID: \"9c00c972-1e46-4182-aaf4-163a6960fb70\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjqps" Oct 30 14:14:58.314673 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:58.312900 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zz5l\" (UniqueName: \"kubernetes.io/projected/9c00c972-1e46-4182-aaf4-163a6960fb70-kube-api-access-5zz5l\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wjqps\" (UID: \"9c00c972-1e46-4182-aaf4-163a6960fb70\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjqps" Oct 30 14:14:58.315561 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:58.315530 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c00c972-1e46-4182-aaf4-163a6960fb70-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wjqps\" (UID: \"9c00c972-1e46-4182-aaf4-163a6960fb70\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjqps" Oct 30 14:14:58.403557 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:58.403524 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjqps" Oct 30 14:14:58.746368 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:58.746053 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjqps"] Oct 30 14:14:58.747251 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:14:58.747218 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c00c972_1e46_4182_aaf4_163a6960fb70.slice/crio-bfefdf0ff84a3d647c7ad4414dd5a7b4ee161672e175f4470fc491e264375ca9 WatchSource:0}: Error finding container bfefdf0ff84a3d647c7ad4414dd5a7b4ee161672e175f4470fc491e264375ca9: Status 404 returned error can't find the container with id bfefdf0ff84a3d647c7ad4414dd5a7b4ee161672e175f4470fc491e264375ca9 Oct 30 14:14:59.487879 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:59.487840 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:14:59.756744 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:14:59.756624 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjqps" event={"ID":"9c00c972-1e46-4182-aaf4-163a6960fb70","Type":"ContainerStarted","Data":"bfefdf0ff84a3d647c7ad4414dd5a7b4ee161672e175f4470fc491e264375ca9"} Oct 30 14:15:00.179413 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:00.179377 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889495-dqtn6"] Oct 30 14:15:00.184564 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:00.184539 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889495-dqtn6" Oct 30 14:15:00.213369 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:00.213333 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889495-95w96"] Oct 30 14:15:00.224592 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:00.224377 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889495-dqtn6"] Oct 30 14:15:00.224592 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:00.224450 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889495-95w96" Oct 30 14:15:00.226923 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:00.226897 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9dba4af9-29889495-cr2cl"] Oct 30 14:15:00.256920 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:00.256887 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889495-95w96"] Oct 30 14:15:00.257074 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:00.256943 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9dba4af9-29889495-cr2cl"] Oct 30 14:15:00.257074 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:00.257040 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9dba4af9-29889495-cr2cl" Oct 30 14:15:00.294045 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:00.294021 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:15:00.318540 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:00.318509 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b7bcb9c-a287-4cf1-86f3-3ed486f210e1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889495-dqtn6\" (UID: \"8b7bcb9c-a287-4cf1-86f3-3ed486f210e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889495-dqtn6" Oct 30 14:15:00.318698 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:00.318570 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pswz\" (UniqueName: \"kubernetes.io/projected/74c90d32-f9a9-40f4-9834-77268ed4a2d9-kube-api-access-5pswz\") pod \"maas-api-key-cleanup-e2e-mt-9dba4af9-29889495-cr2cl\" (UID: \"74c90d32-f9a9-40f4-9834-77268ed4a2d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9dba4af9-29889495-cr2cl" Oct 30 14:15:00.318753 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:00.318707 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74c90d32-f9a9-40f4-9834-77268ed4a2d9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-9dba4af9-29889495-cr2cl\" (UID: \"74c90d32-f9a9-40f4-9834-77268ed4a2d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9dba4af9-29889495-cr2cl" Oct 30 14:15:00.318856 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:00.318831 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/198d48ae-abd7-4d73-81c1-60c3b826ed0c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-ca229c-29889495-95w96\" (UID: \"198d48ae-abd7-4d73-81c1-60c3b826ed0c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889495-95w96" Oct 30 14:15:00.318919 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:00.318886 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnhc5\" (UniqueName: \"kubernetes.io/projected/198d48ae-abd7-4d73-81c1-60c3b826ed0c-kube-api-access-cnhc5\") pod \"maas-api-key-cleanup-e2e-worker-w1-ca229c-29889495-95w96\" (UID: \"198d48ae-abd7-4d73-81c1-60c3b826ed0c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889495-95w96" Oct 30 14:15:00.319064 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:00.319042 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbmx9\" (UniqueName: \"kubernetes.io/projected/8b7bcb9c-a287-4cf1-86f3-3ed486f210e1-kube-api-access-dbmx9\") pod \"maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889495-dqtn6\" (UID: \"8b7bcb9c-a287-4cf1-86f3-3ed486f210e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889495-dqtn6" Oct 30 14:15:00.420771 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:00.420733 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbmx9\" (UniqueName: \"kubernetes.io/projected/8b7bcb9c-a287-4cf1-86f3-3ed486f210e1-kube-api-access-dbmx9\") pod \"maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889495-dqtn6\" (UID: \"8b7bcb9c-a287-4cf1-86f3-3ed486f210e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889495-dqtn6" Oct 30 14:15:00.420959 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:00.420823 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b7bcb9c-a287-4cf1-86f3-3ed486f210e1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889495-dqtn6\" (UID: \"8b7bcb9c-a287-4cf1-86f3-3ed486f210e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889495-dqtn6" Oct 30 14:15:00.420959 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:00.420857 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pswz\" (UniqueName: \"kubernetes.io/projected/74c90d32-f9a9-40f4-9834-77268ed4a2d9-kube-api-access-5pswz\") pod \"maas-api-key-cleanup-e2e-mt-9dba4af9-29889495-cr2cl\" (UID: \"74c90d32-f9a9-40f4-9834-77268ed4a2d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9dba4af9-29889495-cr2cl" Oct 30 14:15:00.420959 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:00.420907 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74c90d32-f9a9-40f4-9834-77268ed4a2d9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-9dba4af9-29889495-cr2cl\" (UID: \"74c90d32-f9a9-40f4-9834-77268ed4a2d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9dba4af9-29889495-cr2cl" Oct 30 14:15:00.421465 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:00.421257 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/198d48ae-abd7-4d73-81c1-60c3b826ed0c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-ca229c-29889495-95w96\" (UID: \"198d48ae-abd7-4d73-81c1-60c3b826ed0c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889495-95w96" Oct 30 14:15:00.421465 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:00.421324 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnhc5\" (UniqueName: \"kubernetes.io/projected/198d48ae-abd7-4d73-81c1-60c3b826ed0c-kube-api-access-cnhc5\") pod \"maas-api-key-cleanup-e2e-worker-w1-ca229c-29889495-95w96\" (UID: \"198d48ae-abd7-4d73-81c1-60c3b826ed0c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889495-95w96" Oct 30 14:15:00.421782 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:00.421752 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74c90d32-f9a9-40f4-9834-77268ed4a2d9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-9dba4af9-29889495-cr2cl\" (UID: \"74c90d32-f9a9-40f4-9834-77268ed4a2d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9dba4af9-29889495-cr2cl" Oct 30 14:15:00.421972 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:00.421951 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b7bcb9c-a287-4cf1-86f3-3ed486f210e1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889495-dqtn6\" (UID: \"8b7bcb9c-a287-4cf1-86f3-3ed486f210e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889495-dqtn6" Oct 30 14:15:00.422055 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:00.421963 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/198d48ae-abd7-4d73-81c1-60c3b826ed0c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-ca229c-29889495-95w96\" (UID: \"198d48ae-abd7-4d73-81c1-60c3b826ed0c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889495-95w96" Oct 30 14:15:00.449519 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:00.449463 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbmx9\" (UniqueName: \"kubernetes.io/projected/8b7bcb9c-a287-4cf1-86f3-3ed486f210e1-kube-api-access-dbmx9\") pod \"maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889495-dqtn6\" (UID: \"8b7bcb9c-a287-4cf1-86f3-3ed486f210e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889495-dqtn6" Oct 30 14:15:00.449867 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:00.449847 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnhc5\" (UniqueName: \"kubernetes.io/projected/198d48ae-abd7-4d73-81c1-60c3b826ed0c-kube-api-access-cnhc5\") pod \"maas-api-key-cleanup-e2e-worker-w1-ca229c-29889495-95w96\" (UID: \"198d48ae-abd7-4d73-81c1-60c3b826ed0c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889495-95w96" Oct 30 14:15:00.450459 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:00.450442 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pswz\" (UniqueName: \"kubernetes.io/projected/74c90d32-f9a9-40f4-9834-77268ed4a2d9-kube-api-access-5pswz\") pod \"maas-api-key-cleanup-e2e-mt-9dba4af9-29889495-cr2cl\" (UID: \"74c90d32-f9a9-40f4-9834-77268ed4a2d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9dba4af9-29889495-cr2cl" Oct 30 14:15:00.505366 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:00.505338 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889495-dqtn6" Oct 30 14:15:00.553436 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:00.553375 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889495-95w96" Oct 30 14:15:00.583721 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:00.583459 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9dba4af9-29889495-cr2cl" Oct 30 14:15:00.845981 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:00.845950 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889495-dqtn6"] Oct 30 14:15:00.924740 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:00.924708 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889495-95w96"] Oct 30 14:15:00.931083 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:15:00.931052 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod198d48ae_abd7_4d73_81c1_60c3b826ed0c.slice/crio-451a5db903d4623c5a3b75ac1b4139ec111421f2d4bc02e900f62db3a197ef92 WatchSource:0}: Error finding container 451a5db903d4623c5a3b75ac1b4139ec111421f2d4bc02e900f62db3a197ef92: Status 404 returned error can't find the container with id 451a5db903d4623c5a3b75ac1b4139ec111421f2d4bc02e900f62db3a197ef92 Oct 30 14:15:00.960199 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:00.960140 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9dba4af9-29889495-cr2cl"] Oct 30 14:15:00.961254 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:15:00.961216 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74c90d32_f9a9_40f4_9834_77268ed4a2d9.slice/crio-8ce2545d6894043a5034e23438634afd01e3f9cef357b3ef43752a3596077f1b WatchSource:0}: Error finding container 8ce2545d6894043a5034e23438634afd01e3f9cef357b3ef43752a3596077f1b: Status 404 returned error can't find the container with id 8ce2545d6894043a5034e23438634afd01e3f9cef357b3ef43752a3596077f1b Oct 30 14:15:01.568129 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:01.568094 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:15:01.794404 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:01.794363 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889495-dqtn6" event={"ID":"8b7bcb9c-a287-4cf1-86f3-3ed486f210e1","Type":"ContainerStarted","Data":"2958def2c2f8a5484428f39880bd4fa78de0b6a5dc08371946cbfe1d21e6170f"} Oct 30 14:15:01.796519 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:01.796486 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889495-95w96" event={"ID":"198d48ae-abd7-4d73-81c1-60c3b826ed0c","Type":"ContainerStarted","Data":"451a5db903d4623c5a3b75ac1b4139ec111421f2d4bc02e900f62db3a197ef92"} Oct 30 14:15:01.799061 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:01.799031 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9dba4af9-29889495-cr2cl" event={"ID":"74c90d32-f9a9-40f4-9834-77268ed4a2d9","Type":"ContainerStarted","Data":"8ce2545d6894043a5034e23438634afd01e3f9cef357b3ef43752a3596077f1b"} Oct 30 14:15:02.199247 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:02.199201 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:15:02.199396 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:02.199294 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:15:02.812653 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:02.812595 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjqps" event={"ID":"9c00c972-1e46-4182-aaf4-163a6960fb70","Type":"ContainerStarted","Data":"5f7b659e00513b90380612c899dfa9f10e29ddeb3d97ec4166160c8f991158b6"} Oct 30 14:15:02.813143 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:02.812713 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjqps" Oct 30 14:15:03.510019 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:03.509983 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:15:03.824596 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:03.824560 4982 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-wjqps container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 14:15:03.825094 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:03.824619 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjqps" podUID="9c00c972-1e46-4182-aaf4-163a6960fb70" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 14:15:05.338790 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:05.338738 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:15:05.838055 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:05.838029 4982 generic.go:356] "Generic (PLEG): container finished" podID="8b7bcb9c-a287-4cf1-86f3-3ed486f210e1" containerID="32a96252cfa4f6b91085761ee722132b664129db431401e7d21a3b7870d0b4af" exitCode=60 Oct 30 14:15:05.838175 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:05.838114 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889495-dqtn6" event={"ID":"8b7bcb9c-a287-4cf1-86f3-3ed486f210e1","Type":"ContainerDied","Data":"32a96252cfa4f6b91085761ee722132b664129db431401e7d21a3b7870d0b4af"} Oct 30 14:15:05.838415 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:05.838398 4982 scope.go:122] "RemoveContainer" containerID="32a96252cfa4f6b91085761ee722132b664129db431401e7d21a3b7870d0b4af" Oct 30 14:15:05.839897 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:05.839878 4982 generic.go:356] "Generic (PLEG): container finished" podID="198d48ae-abd7-4d73-81c1-60c3b826ed0c" containerID="fe97583a40787669d4274acd21709102ea2c42798fe717fa1278e567dae4175e" exitCode=60 Oct 30 14:15:05.839999 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:05.839962 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889495-95w96" event={"ID":"198d48ae-abd7-4d73-81c1-60c3b826ed0c","Type":"ContainerDied","Data":"fe97583a40787669d4274acd21709102ea2c42798fe717fa1278e567dae4175e"} Oct 30 14:15:05.840200 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:05.840179 4982 scope.go:122] "RemoveContainer" containerID="fe97583a40787669d4274acd21709102ea2c42798fe717fa1278e567dae4175e" Oct 30 14:15:05.841789 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:05.841768 4982 generic.go:356] "Generic (PLEG): container finished" podID="74c90d32-f9a9-40f4-9834-77268ed4a2d9" containerID="cb19374f6f127c53374db70a661eea0c389880f9a32512ad80f5519738098926" exitCode=60 Oct 30 14:15:05.841900 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:05.841811 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9dba4af9-29889495-cr2cl" event={"ID":"74c90d32-f9a9-40f4-9834-77268ed4a2d9","Type":"ContainerDied","Data":"cb19374f6f127c53374db70a661eea0c389880f9a32512ad80f5519738098926"} Oct 30 14:15:05.842106 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:05.842083 4982 scope.go:122] "RemoveContainer" containerID="cb19374f6f127c53374db70a661eea0c389880f9a32512ad80f5519738098926" Oct 30 14:15:05.885846 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:05.885798 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjqps" podStartSLOduration=4.437317237 podStartE2EDuration="7.885781608s" totalImagesPullingTime="3.448464371s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:14:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:15:02.906134477 +0000 UTC m=+1572.650268524" watchObservedRunningTime="2026-10-30 14:15:05.885781608 +0000 UTC m=+1575.629915651" Oct 30 14:15:06.824231 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:06.824186 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjqps" Oct 30 14:15:06.850842 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:06.850798 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889495-95w96" event={"ID":"198d48ae-abd7-4d73-81c1-60c3b826ed0c","Type":"ContainerStarted","Data":"9a28921b6c40209330ff9690909f58357996461b5ca7ef5d42028904607e41af"} Oct 30 14:15:06.852654 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:06.852610 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9dba4af9-29889495-cr2cl" event={"ID":"74c90d32-f9a9-40f4-9834-77268ed4a2d9","Type":"ContainerStarted","Data":"13b666597e19e487cfc01f5a9995c81fa7ef5f4957a7e840b4882d174a92f012"} Oct 30 14:15:06.854350 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:06.854329 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889495-dqtn6" event={"ID":"8b7bcb9c-a287-4cf1-86f3-3ed486f210e1","Type":"ContainerStarted","Data":"115444e9c154b3beb19266620e4fd8c417730b318d54cc70e4b00129770ebf24"} Oct 30 14:15:06.976810 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:06.976761 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9dba4af9-29889495-cr2cl" podStartSLOduration=3.3478168249999998 podStartE2EDuration="6.976747675s" totalImagesPullingTime="3.62893085s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:15:06.975428416 +0000 UTC m=+1576.719562460" watchObservedRunningTime="2026-10-30 14:15:06.976747675 +0000 UTC m=+1576.720881716" Oct 30 14:15:06.979035 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:06.979000 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889495-95w96" podStartSLOduration=3.210230337 podStartE2EDuration="6.978988299s" totalImagesPullingTime="3.768757962s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:15:06.932923018 +0000 UTC m=+1576.677057062" watchObservedRunningTime="2026-10-30 14:15:06.978988299 +0000 UTC m=+1576.723122385" Oct 30 14:15:07.022930 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:07.022876 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889495-dqtn6" podStartSLOduration=3.230235423 podStartE2EDuration="7.02286048s" totalImagesPullingTime="3.792625057s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:15:07.02110621 +0000 UTC m=+1576.765240252" watchObservedRunningTime="2026-10-30 14:15:07.02286048 +0000 UTC m=+1576.766994521" Oct 30 14:15:07.338363 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:07.338277 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:15:07.863266 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:07.863233 4982 generic.go:356] "Generic (PLEG): container finished" podID="74c90d32-f9a9-40f4-9834-77268ed4a2d9" containerID="13b666597e19e487cfc01f5a9995c81fa7ef5f4957a7e840b4882d174a92f012" exitCode=60 Oct 30 14:15:07.863748 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:07.863330 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9dba4af9-29889495-cr2cl" event={"ID":"74c90d32-f9a9-40f4-9834-77268ed4a2d9","Type":"ContainerDied","Data":"13b666597e19e487cfc01f5a9995c81fa7ef5f4957a7e840b4882d174a92f012"} Oct 30 14:15:07.863748 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:07.863412 4982 scope.go:122] "RemoveContainer" containerID="cb19374f6f127c53374db70a661eea0c389880f9a32512ad80f5519738098926" Oct 30 14:15:07.863748 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:07.863661 4982 scope.go:122] "RemoveContainer" containerID="13b666597e19e487cfc01f5a9995c81fa7ef5f4957a7e840b4882d174a92f012" Oct 30 14:15:07.864083 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:15:07.864061 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-9dba4af9-29889495-cr2cl_odh-ai-gateway-infra(74c90d32-f9a9-40f4-9834-77268ed4a2d9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9dba4af9-29889495-cr2cl" podUID="74c90d32-f9a9-40f4-9834-77268ed4a2d9" Oct 30 14:15:07.865840 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:07.865818 4982 generic.go:356] "Generic (PLEG): container finished" podID="8b7bcb9c-a287-4cf1-86f3-3ed486f210e1" containerID="115444e9c154b3beb19266620e4fd8c417730b318d54cc70e4b00129770ebf24" exitCode=60 Oct 30 14:15:07.866014 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:07.865848 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889495-dqtn6" event={"ID":"8b7bcb9c-a287-4cf1-86f3-3ed486f210e1","Type":"ContainerDied","Data":"115444e9c154b3beb19266620e4fd8c417730b318d54cc70e4b00129770ebf24"} Oct 30 14:15:07.866239 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:07.866222 4982 scope.go:122] "RemoveContainer" containerID="115444e9c154b3beb19266620e4fd8c417730b318d54cc70e4b00129770ebf24" Oct 30 14:15:07.866553 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:15:07.866533 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889495-dqtn6_odh-ai-gateway-infra(8b7bcb9c-a287-4cf1-86f3-3ed486f210e1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889495-dqtn6" podUID="8b7bcb9c-a287-4cf1-86f3-3ed486f210e1" Oct 30 14:15:07.867959 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:07.867936 4982 generic.go:356] "Generic (PLEG): container finished" podID="198d48ae-abd7-4d73-81c1-60c3b826ed0c" containerID="9a28921b6c40209330ff9690909f58357996461b5ca7ef5d42028904607e41af" exitCode=60 Oct 30 14:15:07.868045 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:07.867960 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889495-95w96" event={"ID":"198d48ae-abd7-4d73-81c1-60c3b826ed0c","Type":"ContainerDied","Data":"9a28921b6c40209330ff9690909f58357996461b5ca7ef5d42028904607e41af"} Oct 30 14:15:07.868273 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:07.868258 4982 scope.go:122] "RemoveContainer" containerID="9a28921b6c40209330ff9690909f58357996461b5ca7ef5d42028904607e41af" Oct 30 14:15:07.868567 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:15:07.868549 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-ca229c-29889495-95w96_odh-ai-gateway-infra(198d48ae-abd7-4d73-81c1-60c3b826ed0c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889495-95w96" podUID="198d48ae-abd7-4d73-81c1-60c3b826ed0c" Oct 30 14:15:07.876509 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:07.876479 4982 scope.go:122] "RemoveContainer" containerID="32a96252cfa4f6b91085761ee722132b664129db431401e7d21a3b7870d0b4af" Oct 30 14:15:07.892222 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:07.892195 4982 scope.go:122] "RemoveContainer" containerID="fe97583a40787669d4274acd21709102ea2c42798fe717fa1278e567dae4175e" Oct 30 14:15:08.437483 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:08.437452 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:15:08.877777 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:08.877236 4982 scope.go:122] "RemoveContainer" containerID="115444e9c154b3beb19266620e4fd8c417730b318d54cc70e4b00129770ebf24" Oct 30 14:15:08.877777 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:15:08.877598 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889495-dqtn6_odh-ai-gateway-infra(8b7bcb9c-a287-4cf1-86f3-3ed486f210e1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889495-dqtn6" podUID="8b7bcb9c-a287-4cf1-86f3-3ed486f210e1" Oct 30 14:15:08.879274 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:08.879244 4982 scope.go:122] "RemoveContainer" containerID="9a28921b6c40209330ff9690909f58357996461b5ca7ef5d42028904607e41af" Oct 30 14:15:08.879562 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:15:08.879537 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-ca229c-29889495-95w96_odh-ai-gateway-infra(198d48ae-abd7-4d73-81c1-60c3b826ed0c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889495-95w96" podUID="198d48ae-abd7-4d73-81c1-60c3b826ed0c" Oct 30 14:15:08.880999 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:08.880979 4982 scope.go:122] "RemoveContainer" containerID="13b666597e19e487cfc01f5a9995c81fa7ef5f4957a7e840b4882d174a92f012" Oct 30 14:15:08.881272 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:15:08.881253 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-9dba4af9-29889495-cr2cl_odh-ai-gateway-infra(74c90d32-f9a9-40f4-9834-77268ed4a2d9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9dba4af9-29889495-cr2cl" podUID="74c90d32-f9a9-40f4-9834-77268ed4a2d9" Oct 30 14:15:09.185173 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:09.185074 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:15:09.327757 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:09.327718 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:15:09.477019 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:09.476935 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:15:11.334448 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:11.334408 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:15:13.246371 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:13.246320 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:15:15.308665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:15.305258 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:15:15.471670 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:15.471608 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:15:15.656720 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:15.656598 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:15:17.476103 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:17.476061 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:15:19.342503 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:19.342459 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:15:19.396461 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:19.392157 4982 scope.go:122] "RemoveContainer" containerID="9a28921b6c40209330ff9690909f58357996461b5ca7ef5d42028904607e41af" Oct 30 14:15:19.446002 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:19.445939 4982 scope.go:122] "RemoveContainer" containerID="13b666597e19e487cfc01f5a9995c81fa7ef5f4957a7e840b4882d174a92f012" Oct 30 14:15:20.391687 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:20.389761 4982 patch_prober.go:29] interesting pod/e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:15:20.391687 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:20.389834 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg" podUID="7057acc6-78aa-43fa-bc57-6e5ecacf2e50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:15:20.399166 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:20.399130 4982 scope.go:122] "RemoveContainer" containerID="115444e9c154b3beb19266620e4fd8c417730b318d54cc70e4b00129770ebf24" Oct 30 14:15:21.005273 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:21.005181 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889495-dqtn6" event={"ID":"8b7bcb9c-a287-4cf1-86f3-3ed486f210e1","Type":"ContainerStarted","Data":"e2c637d67124012ff3c03972c9d45751ae11247c25ebffd43865b79dc4c6dc01"} Oct 30 14:15:21.008193 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:21.008159 4982 generic.go:356] "Generic (PLEG): container finished" podID="198d48ae-abd7-4d73-81c1-60c3b826ed0c" containerID="24eb2c320dbf35a9ccb192649354e459cb3140dd89001df78faefc1789b963bb" exitCode=60 Oct 30 14:15:21.008350 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:21.008234 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889495-95w96" event={"ID":"198d48ae-abd7-4d73-81c1-60c3b826ed0c","Type":"ContainerDied","Data":"24eb2c320dbf35a9ccb192649354e459cb3140dd89001df78faefc1789b963bb"} Oct 30 14:15:21.008350 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:21.008309 4982 scope.go:122] "RemoveContainer" containerID="9a28921b6c40209330ff9690909f58357996461b5ca7ef5d42028904607e41af" Oct 30 14:15:21.009199 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:21.008849 4982 scope.go:122] "RemoveContainer" containerID="24eb2c320dbf35a9ccb192649354e459cb3140dd89001df78faefc1789b963bb" Oct 30 14:15:21.010230 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:15:21.010073 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-ca229c-29889495-95w96_odh-ai-gateway-infra(198d48ae-abd7-4d73-81c1-60c3b826ed0c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889495-95w96" podUID="198d48ae-abd7-4d73-81c1-60c3b826ed0c" Oct 30 14:15:21.011301 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:21.011282 4982 generic.go:356] "Generic (PLEG): container finished" podID="74c90d32-f9a9-40f4-9834-77268ed4a2d9" containerID="51bbf486ce4745def9ba6df095bc26eb46ebc59d1c4587bbbd2bc774b5727f02" exitCode=60 Oct 30 14:15:21.011451 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:21.011434 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9dba4af9-29889495-cr2cl" event={"ID":"74c90d32-f9a9-40f4-9834-77268ed4a2d9","Type":"ContainerDied","Data":"51bbf486ce4745def9ba6df095bc26eb46ebc59d1c4587bbbd2bc774b5727f02"} Oct 30 14:15:21.011885 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:21.011865 4982 scope.go:122] "RemoveContainer" containerID="51bbf486ce4745def9ba6df095bc26eb46ebc59d1c4587bbbd2bc774b5727f02" Oct 30 14:15:21.012194 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:15:21.012175 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-9dba4af9-29889495-cr2cl_odh-ai-gateway-infra(74c90d32-f9a9-40f4-9834-77268ed4a2d9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9dba4af9-29889495-cr2cl" podUID="74c90d32-f9a9-40f4-9834-77268ed4a2d9" Oct 30 14:15:21.025829 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:21.025803 4982 scope.go:122] "RemoveContainer" containerID="13b666597e19e487cfc01f5a9995c81fa7ef5f4957a7e840b4882d174a92f012" Oct 30 14:15:21.178188 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:21.178110 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:15:22.028668 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:22.027433 4982 generic.go:356] "Generic (PLEG): container finished" podID="8b7bcb9c-a287-4cf1-86f3-3ed486f210e1" containerID="e2c637d67124012ff3c03972c9d45751ae11247c25ebffd43865b79dc4c6dc01" exitCode=60 Oct 30 14:15:22.028668 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:22.027544 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889495-dqtn6" event={"ID":"8b7bcb9c-a287-4cf1-86f3-3ed486f210e1","Type":"ContainerDied","Data":"e2c637d67124012ff3c03972c9d45751ae11247c25ebffd43865b79dc4c6dc01"} Oct 30 14:15:22.028668 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:22.027611 4982 scope.go:122] "RemoveContainer" containerID="115444e9c154b3beb19266620e4fd8c417730b318d54cc70e4b00129770ebf24" Oct 30 14:15:22.028668 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:22.028277 4982 scope.go:122] "RemoveContainer" containerID="e2c637d67124012ff3c03972c9d45751ae11247c25ebffd43865b79dc4c6dc01" Oct 30 14:15:22.029323 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:15:22.028814 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889495-dqtn6_odh-ai-gateway-infra(8b7bcb9c-a287-4cf1-86f3-3ed486f210e1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889495-dqtn6" podUID="8b7bcb9c-a287-4cf1-86f3-3ed486f210e1" Oct 30 14:15:22.070787 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:22.070746 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889495-dqtn6"] Oct 30 14:15:22.117103 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:22.117049 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889495-95w96"] Oct 30 14:15:22.252537 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:22.252487 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9dba4af9-29889495-cr2cl"] Oct 30 14:15:22.453590 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:22.453534 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889495-95w96" Oct 30 14:15:22.523117 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:22.522770 4982 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:15:22.523117 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:22.522836 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b" podUID="c99896fe-79b1-4368-aeb5-a81182e7ae13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:15:22.536959 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:22.536923 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/198d48ae-abd7-4d73-81c1-60c3b826ed0c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/198d48ae-abd7-4d73-81c1-60c3b826ed0c-maas-api-service-ca\") pod \"198d48ae-abd7-4d73-81c1-60c3b826ed0c\" (UID: \"198d48ae-abd7-4d73-81c1-60c3b826ed0c\") " Oct 30 14:15:22.537152 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:22.537012 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/198d48ae-abd7-4d73-81c1-60c3b826ed0c-kube-api-access-cnhc5\" (UniqueName: \"kubernetes.io/projected/198d48ae-abd7-4d73-81c1-60c3b826ed0c-kube-api-access-cnhc5\") pod \"198d48ae-abd7-4d73-81c1-60c3b826ed0c\" (UID: \"198d48ae-abd7-4d73-81c1-60c3b826ed0c\") " Oct 30 14:15:22.539269 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:22.539237 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/198d48ae-abd7-4d73-81c1-60c3b826ed0c-maas-api-service-ca" pod "198d48ae-abd7-4d73-81c1-60c3b826ed0c" (UID: "198d48ae-abd7-4d73-81c1-60c3b826ed0c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:15:22.541291 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:22.540986 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/198d48ae-abd7-4d73-81c1-60c3b826ed0c-maas-api-service-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:15:22.542884 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:22.541734 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9dba4af9-29889495-cr2cl" Oct 30 14:15:22.545523 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:22.545490 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/198d48ae-abd7-4d73-81c1-60c3b826ed0c-kube-api-access-cnhc5" pod "198d48ae-abd7-4d73-81c1-60c3b826ed0c" (UID: "198d48ae-abd7-4d73-81c1-60c3b826ed0c"). InnerVolumeSpecName "kube-api-access-cnhc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:15:22.642517 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:22.642424 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74c90d32-f9a9-40f4-9834-77268ed4a2d9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74c90d32-f9a9-40f4-9834-77268ed4a2d9-maas-api-service-ca\") pod \"74c90d32-f9a9-40f4-9834-77268ed4a2d9\" (UID: \"74c90d32-f9a9-40f4-9834-77268ed4a2d9\") " Oct 30 14:15:22.642735 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:22.642546 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74c90d32-f9a9-40f4-9834-77268ed4a2d9-kube-api-access-5pswz\" (UniqueName: \"kubernetes.io/projected/74c90d32-f9a9-40f4-9834-77268ed4a2d9-kube-api-access-5pswz\") pod \"74c90d32-f9a9-40f4-9834-77268ed4a2d9\" (UID: \"74c90d32-f9a9-40f4-9834-77268ed4a2d9\") " Oct 30 14:15:22.642918 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:22.642888 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74c90d32-f9a9-40f4-9834-77268ed4a2d9-maas-api-service-ca" pod "74c90d32-f9a9-40f4-9834-77268ed4a2d9" (UID: "74c90d32-f9a9-40f4-9834-77268ed4a2d9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:15:22.643506 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:22.643470 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74c90d32-f9a9-40f4-9834-77268ed4a2d9-maas-api-service-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:15:22.643506 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:22.643503 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnhc5\" (UniqueName: \"kubernetes.io/projected/198d48ae-abd7-4d73-81c1-60c3b826ed0c-kube-api-access-cnhc5\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:15:22.645722 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:22.645687 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74c90d32-f9a9-40f4-9834-77268ed4a2d9-kube-api-access-5pswz" pod "74c90d32-f9a9-40f4-9834-77268ed4a2d9" (UID: "74c90d32-f9a9-40f4-9834-77268ed4a2d9"). InnerVolumeSpecName "kube-api-access-5pswz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:15:22.745607 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:22.745561 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pswz\" (UniqueName: \"kubernetes.io/projected/74c90d32-f9a9-40f4-9834-77268ed4a2d9-kube-api-access-5pswz\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:15:23.072686 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:23.052452 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889495-95w96" Oct 30 14:15:23.072686 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:23.058954 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889495-95w96" event={"ID":"198d48ae-abd7-4d73-81c1-60c3b826ed0c","Type":"ContainerDied","Data":"451a5db903d4623c5a3b75ac1b4139ec111421f2d4bc02e900f62db3a197ef92"} Oct 30 14:15:23.072686 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:23.059044 4982 scope.go:122] "RemoveContainer" containerID="24eb2c320dbf35a9ccb192649354e459cb3140dd89001df78faefc1789b963bb" Oct 30 14:15:23.129277 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:23.129239 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9dba4af9-29889495-cr2cl" event={"ID":"74c90d32-f9a9-40f4-9834-77268ed4a2d9","Type":"ContainerDied","Data":"8ce2545d6894043a5034e23438634afd01e3f9cef357b3ef43752a3596077f1b"} Oct 30 14:15:23.129449 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:23.129373 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9dba4af9-29889495-cr2cl" Oct 30 14:15:23.137186 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:23.137070 4982 scope.go:122] "RemoveContainer" containerID="51bbf486ce4745def9ba6df095bc26eb46ebc59d1c4587bbbd2bc774b5727f02" Oct 30 14:15:23.198717 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:23.184378 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889495-95w96"] Oct 30 14:15:23.198717 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:23.192927 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889495-95w96"] Oct 30 14:15:23.236833 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:23.236793 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:15:23.254382 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:23.253933 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9dba4af9-29889495-cr2cl"] Oct 30 14:15:23.278964 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:23.278928 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9dba4af9-29889495-cr2cl"] Oct 30 14:15:23.330278 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:23.329882 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889495-dqtn6" Oct 30 14:15:23.470378 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:23.470340 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b7bcb9c-a287-4cf1-86f3-3ed486f210e1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b7bcb9c-a287-4cf1-86f3-3ed486f210e1-maas-api-service-ca\") pod \"8b7bcb9c-a287-4cf1-86f3-3ed486f210e1\" (UID: \"8b7bcb9c-a287-4cf1-86f3-3ed486f210e1\") " Oct 30 14:15:23.470568 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:23.470408 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b7bcb9c-a287-4cf1-86f3-3ed486f210e1-kube-api-access-dbmx9\" (UniqueName: \"kubernetes.io/projected/8b7bcb9c-a287-4cf1-86f3-3ed486f210e1-kube-api-access-dbmx9\") pod \"8b7bcb9c-a287-4cf1-86f3-3ed486f210e1\" (UID: \"8b7bcb9c-a287-4cf1-86f3-3ed486f210e1\") " Oct 30 14:15:23.470801 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:23.470774 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b7bcb9c-a287-4cf1-86f3-3ed486f210e1-maas-api-service-ca" pod "8b7bcb9c-a287-4cf1-86f3-3ed486f210e1" (UID: "8b7bcb9c-a287-4cf1-86f3-3ed486f210e1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:15:23.471807 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:23.471777 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b7bcb9c-a287-4cf1-86f3-3ed486f210e1-maas-api-service-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:15:23.473516 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:23.473481 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b7bcb9c-a287-4cf1-86f3-3ed486f210e1-kube-api-access-dbmx9" pod "8b7bcb9c-a287-4cf1-86f3-3ed486f210e1" (UID: "8b7bcb9c-a287-4cf1-86f3-3ed486f210e1"). InnerVolumeSpecName "kube-api-access-dbmx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:15:23.573828 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:23.573782 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbmx9\" (UniqueName: \"kubernetes.io/projected/8b7bcb9c-a287-4cf1-86f3-3ed486f210e1-kube-api-access-dbmx9\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:15:24.139015 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:24.138970 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889495-dqtn6" event={"ID":"8b7bcb9c-a287-4cf1-86f3-3ed486f210e1","Type":"ContainerDied","Data":"2958def2c2f8a5484428f39880bd4fa78de0b6a5dc08371946cbfe1d21e6170f"} Oct 30 14:15:24.925877 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:24.138986 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889495-dqtn6" Oct 30 14:15:24.925877 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:24.139045 4982 scope.go:122] "RemoveContainer" containerID="e2c637d67124012ff3c03972c9d45751ae11247c25ebffd43865b79dc4c6dc01" Oct 30 14:15:24.925877 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:24.197753 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889495-dqtn6"] Oct 30 14:15:24.925877 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:24.213058 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889495-dqtn6"] Oct 30 14:15:24.925877 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:24.403461 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="198d48ae-abd7-4d73-81c1-60c3b826ed0c" path="/var/lib/kubelet/pods/198d48ae-abd7-4d73-81c1-60c3b826ed0c/volumes" Oct 30 14:15:24.925877 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:24.403957 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74c90d32-f9a9-40f4-9834-77268ed4a2d9" path="/var/lib/kubelet/pods/74c90d32-f9a9-40f4-9834-77268ed4a2d9/volumes" Oct 30 14:15:24.925877 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:24.404361 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b7bcb9c-a287-4cf1-86f3-3ed486f210e1" path="/var/lib/kubelet/pods/8b7bcb9c-a287-4cf1-86f3-3ed486f210e1/volumes" Oct 30 14:15:25.150419 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:25.150386 4982 generic.go:356] "Generic (PLEG): container finished" podID="c99896fe-79b1-4368-aeb5-a81182e7ae13" containerID="26ba0205e31735dc2452611301762a84e2575620db52858330c39d279e7b54f8" exitCode=0 Oct 30 14:15:25.150933 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:25.150455 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b" event={"ID":"c99896fe-79b1-4368-aeb5-a81182e7ae13","Type":"ContainerDied","Data":"26ba0205e31735dc2452611301762a84e2575620db52858330c39d279e7b54f8"} Oct 30 14:15:25.151054 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:25.151037 4982 scope.go:122] "RemoveContainer" containerID="26ba0205e31735dc2452611301762a84e2575620db52858330c39d279e7b54f8" Oct 30 14:15:25.152988 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:25.152954 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:15:25.153125 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:25.153045 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:15:25.153125 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:25.153098 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:15:25.153466 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:25.153443 4982 generic.go:356] "Generic (PLEG): container finished" podID="7057acc6-78aa-43fa-bc57-6e5ecacf2e50" containerID="ad6543ac955d66904b53a21d988117a4636a57451e9d2407dd6c912eace2e435" exitCode=0 Oct 30 14:15:25.153612 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:25.153507 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg" event={"ID":"7057acc6-78aa-43fa-bc57-6e5ecacf2e50","Type":"ContainerDied","Data":"ad6543ac955d66904b53a21d988117a4636a57451e9d2407dd6c912eace2e435"} Oct 30 14:15:25.154063 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:25.153947 4982 scope.go:122] "RemoveContainer" containerID="ad6543ac955d66904b53a21d988117a4636a57451e9d2407dd6c912eace2e435" Oct 30 14:15:25.155634 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:25.155606 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:15:25.155763 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:25.155695 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:15:25.155763 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:25.155732 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:15:25.309445 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:25.308985 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:15:26.168426 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:26.168379 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b" event={"ID":"c99896fe-79b1-4368-aeb5-a81182e7ae13","Type":"ContainerStarted","Data":"8702c319b369afd4d5d80fad6ade9493e1cdb09f96cbf3a40b0cdb727cf5759d"} Oct 30 14:15:26.171466 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:26.171349 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg" event={"ID":"7057acc6-78aa-43fa-bc57-6e5ecacf2e50","Type":"ContainerStarted","Data":"6a68340dffb035a63f91af47e2f3bbcba54e742ee429f812b080b6162d8b73ac"} Oct 30 14:15:26.563223 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:26.563176 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b" Oct 30 14:15:26.563504 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:26.563197 4982 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 14:15:26.563712 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:26.563684 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b" podUID="c99896fe-79b1-4368-aeb5-a81182e7ae13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 14:15:26.752971 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:26.752933 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg" Oct 30 14:15:26.753159 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:26.753038 4982 patch_prober.go:29] interesting pod/e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 14:15:26.753159 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:26.753073 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg" podUID="7057acc6-78aa-43fa-bc57-6e5ecacf2e50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 14:15:26.922763 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:26.922225 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:15:27.202720 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:27.198002 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:15:27.575726 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:27.565659 4982 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 14:15:27.575726 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:27.565719 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b" podUID="c99896fe-79b1-4368-aeb5-a81182e7ae13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 14:15:27.611349 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:27.593130 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:15:27.767432 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:27.767381 4982 patch_prober.go:29] interesting pod/e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 14:15:27.767716 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:27.767688 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg" podUID="7057acc6-78aa-43fa-bc57-6e5ecacf2e50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 14:15:28.142665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:28.137139 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:15:28.310306 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:15:28.310201 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:15:28.563899 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:28.563866 4982 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 14:15:28.564173 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:28.564146 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b" podUID="c99896fe-79b1-4368-aeb5-a81182e7ae13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 14:15:28.754722 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:28.754674 4982 patch_prober.go:29] interesting pod/e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 14:15:28.754949 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:28.754752 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg" podUID="7057acc6-78aa-43fa-bc57-6e5ecacf2e50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 14:15:29.119745 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:29.119595 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:15:29.563050 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:29.563010 4982 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 14:15:29.563685 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:29.563622 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b" podUID="c99896fe-79b1-4368-aeb5-a81182e7ae13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 14:15:29.757667 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:29.753097 4982 patch_prober.go:29] interesting pod/e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 14:15:29.757667 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:29.753168 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg" podUID="7057acc6-78aa-43fa-bc57-6e5ecacf2e50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 14:15:29.833715 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:29.833234 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:15:30.563496 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:30.563460 4982 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 14:15:30.564177 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:30.564100 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b" podUID="c99896fe-79b1-4368-aeb5-a81182e7ae13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 14:15:30.752924 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:30.752887 4982 patch_prober.go:29] interesting pod/e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 14:15:30.753200 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:30.753174 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg" podUID="7057acc6-78aa-43fa-bc57-6e5ecacf2e50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 14:15:31.223502 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:31.223447 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:15:31.563099 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:31.563059 4982 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 14:15:31.563276 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:31.563121 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b" podUID="c99896fe-79b1-4368-aeb5-a81182e7ae13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 14:15:31.753260 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:31.753217 4982 patch_prober.go:29] interesting pod/e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 14:15:31.753676 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:31.753277 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg" podUID="7057acc6-78aa-43fa-bc57-6e5ecacf2e50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 14:15:32.321247 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:32.321204 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:15:32.465222 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:32.465186 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:15:32.563154 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:32.563110 4982 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 14:15:32.563358 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:32.563188 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b" podUID="c99896fe-79b1-4368-aeb5-a81182e7ae13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 14:15:32.818509 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:32.818473 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg" Oct 30 14:15:32.819326 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:32.819246 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg" Oct 30 14:15:32.820185 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:32.820164 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg" Oct 30 14:15:33.325483 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:33.325446 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:15:33.567324 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:33.567290 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b" Oct 30 14:15:33.567595 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:33.567575 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b" Oct 30 14:15:33.568385 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:33.568363 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b" Oct 30 14:15:34.039408 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:34.039372 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:15:34.196278 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:34.196239 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:15:35.557950 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:35.557905 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:15:37.443274 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:37.443234 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:15:37.963071 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:37.963037 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:15:38.108670 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:38.108622 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:15:38.437850 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:38.437815 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:15:38.929268 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:38.929220 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:15:39.278784 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:39.278432 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:15:39.463249 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:39.463208 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:15:41.546900 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:41.546849 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:15:42.204753 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:42.204715 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:15:43.346826 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:43.346790 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:15:43.505075 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:43.505042 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:15:43.670634 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:43.670523 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:15:45.169578 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:45.169508 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:15:45.411389 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:45.411354 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:15:45.800174 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:45.800127 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:15:46.986534 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:46.986483 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:15:47.843019 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:47.842980 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:15:49.844696 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:49.844659 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:15:51.646223 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:51.646181 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:15:53.568039 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:53.567999 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:15:54.072106 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:54.072055 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:15:55.597877 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:55.597842 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:15:55.740655 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:55.740617 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:15:55.906814 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:55.906719 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:15:56.171964 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:56.171872 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:15:56.866219 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:56.866186 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:15:57.640467 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:57.640396 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:15:58.048923 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:15:58.048841 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:15:59.657870 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:15:59.657834 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:16:00.337115 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:00.337075 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:16:00.648618 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:00.648544 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:16:01.050596 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:01.050511 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:16:01.248338 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:01.248297 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:16:01.779273 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:01.779235 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:16:02.126123 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:02.126090 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:16:02.324776 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:02.324737 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:16:02.504713 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:02.503744 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:16:04.036705 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:04.036657 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:16:05.972239 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:05.972204 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:16:07.794811 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:07.794773 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:16:09.933663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:09.933612 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:16:10.092691 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:10.092620 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:16:11.848079 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:11.848040 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:16:12.420385 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:12.420347 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:16:13.662052 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:13.662015 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:16:15.646443 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:15.646401 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:16:17.534860 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:17.534818 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:16:19.385750 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:19.385711 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:16:20.348582 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:20.348551 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:16:21.324070 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:21.324024 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:16:21.466245 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:21.466207 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:16:23.267611 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:23.267574 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:16:23.977698 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:23.977659 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcb32943-75vlb"] Oct 30 14:16:23.978291 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:23.978276 4982 cpu_manager.go:414] "containerMap: removing container" podUID="74c90d32-f9a9-40f4-9834-77268ed4a2d9" containerName="cleanup" Oct 30 14:16:23.978340 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:23.978293 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74c90d32-f9a9-40f4-9834-77268ed4a2d9" containerName="cleanup" Oct 30 14:16:23.978340 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:23.978310 4982 cpu_manager.go:414] "containerMap: removing container" podUID="74c90d32-f9a9-40f4-9834-77268ed4a2d9" containerName="cleanup" Oct 30 14:16:23.978340 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:23.978327 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74c90d32-f9a9-40f4-9834-77268ed4a2d9" containerName="cleanup" Oct 30 14:16:23.978436 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:23.978342 4982 cpu_manager.go:414] "containerMap: removing container" podUID="198d48ae-abd7-4d73-81c1-60c3b826ed0c" containerName="cleanup" Oct 30 14:16:23.978436 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:23.978348 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="198d48ae-abd7-4d73-81c1-60c3b826ed0c" containerName="cleanup" Oct 30 14:16:23.978436 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:23.978363 4982 cpu_manager.go:414] "containerMap: removing container" podUID="8b7bcb9c-a287-4cf1-86f3-3ed486f210e1" containerName="cleanup" Oct 30 14:16:23.978436 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:23.978367 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b7bcb9c-a287-4cf1-86f3-3ed486f210e1" containerName="cleanup" Oct 30 14:16:23.978436 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:23.978400 4982 cpu_manager.go:414] "containerMap: removing container" podUID="8b7bcb9c-a287-4cf1-86f3-3ed486f210e1" containerName="cleanup" Oct 30 14:16:23.978436 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:23.978405 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b7bcb9c-a287-4cf1-86f3-3ed486f210e1" containerName="cleanup" Oct 30 14:16:23.978436 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:23.978413 4982 cpu_manager.go:414] "containerMap: removing container" podUID="198d48ae-abd7-4d73-81c1-60c3b826ed0c" containerName="cleanup" Oct 30 14:16:23.978436 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:23.978417 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="198d48ae-abd7-4d73-81c1-60c3b826ed0c" containerName="cleanup" Oct 30 14:16:23.978752 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:23.978528 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="74c90d32-f9a9-40f4-9834-77268ed4a2d9" containerName="cleanup" Oct 30 14:16:23.978752 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:23.978537 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="198d48ae-abd7-4d73-81c1-60c3b826ed0c" containerName="cleanup" Oct 30 14:16:23.978752 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:23.978545 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b7bcb9c-a287-4cf1-86f3-3ed486f210e1" containerName="cleanup" Oct 30 14:16:23.978752 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:23.978551 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b7bcb9c-a287-4cf1-86f3-3ed486f210e1" containerName="cleanup" Oct 30 14:16:23.978752 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:23.978558 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="74c90d32-f9a9-40f4-9834-77268ed4a2d9" containerName="cleanup" Oct 30 14:16:23.978752 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:23.978563 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="198d48ae-abd7-4d73-81c1-60c3b826ed0c" containerName="cleanup" Oct 30 14:16:23.982590 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:23.982564 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb32943-75vlb" Oct 30 14:16:24.003943 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:24.003916 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcb32943-75vlb"] Oct 30 14:16:24.010370 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:24.010343 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:16:24.010513 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:24.010394 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:16:24.084203 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:24.084160 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d77678e-4328-4e2e-8dca-19d17111b3cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bcb32943-75vlb\" (UID: \"1d77678e-4328-4e2e-8dca-19d17111b3cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb32943-75vlb" Oct 30 14:16:24.084418 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:24.084332 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt7gh\" (UniqueName: \"kubernetes.io/projected/1d77678e-4328-4e2e-8dca-19d17111b3cb-kube-api-access-mt7gh\") pod \"maas-api-revoke-sub-bcb32943-75vlb\" (UID: \"1d77678e-4328-4e2e-8dca-19d17111b3cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb32943-75vlb" Oct 30 14:16:24.084418 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:24.084405 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d77678e-4328-4e2e-8dca-19d17111b3cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bcb32943-75vlb\" (UID: \"1d77678e-4328-4e2e-8dca-19d17111b3cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb32943-75vlb" Oct 30 14:16:24.190671 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:24.190612 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt7gh\" (UniqueName: \"kubernetes.io/projected/1d77678e-4328-4e2e-8dca-19d17111b3cb-kube-api-access-mt7gh\") pod \"maas-api-revoke-sub-bcb32943-75vlb\" (UID: \"1d77678e-4328-4e2e-8dca-19d17111b3cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb32943-75vlb" Oct 30 14:16:24.204788 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:24.204756 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d77678e-4328-4e2e-8dca-19d17111b3cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bcb32943-75vlb\" (UID: \"1d77678e-4328-4e2e-8dca-19d17111b3cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb32943-75vlb" Oct 30 14:16:24.205293 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:24.205267 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d77678e-4328-4e2e-8dca-19d17111b3cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bcb32943-75vlb\" (UID: \"1d77678e-4328-4e2e-8dca-19d17111b3cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb32943-75vlb" Oct 30 14:16:24.208843 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:24.206275 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d77678e-4328-4e2e-8dca-19d17111b3cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bcb32943-75vlb\" (UID: \"1d77678e-4328-4e2e-8dca-19d17111b3cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb32943-75vlb" Oct 30 14:16:24.229782 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:24.229688 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d77678e-4328-4e2e-8dca-19d17111b3cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bcb32943-75vlb\" (UID: \"1d77678e-4328-4e2e-8dca-19d17111b3cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb32943-75vlb" Oct 30 14:16:24.241347 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:24.241315 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt7gh\" (UniqueName: \"kubernetes.io/projected/1d77678e-4328-4e2e-8dca-19d17111b3cb-kube-api-access-mt7gh\") pod \"maas-api-revoke-sub-bcb32943-75vlb\" (UID: \"1d77678e-4328-4e2e-8dca-19d17111b3cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb32943-75vlb" Oct 30 14:16:24.299025 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:24.298984 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb32943-75vlb" Oct 30 14:16:24.721981 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:24.721951 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcb32943-75vlb"] Oct 30 14:16:24.722839 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:16:24.722814 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d77678e_4328_4e2e_8dca_19d17111b3cb.slice/crio-90622bbc2dcc424568522e5c3bb8439ecb6f62197bc27abf9dac5be72819e183 WatchSource:0}: Error finding container 90622bbc2dcc424568522e5c3bb8439ecb6f62197bc27abf9dac5be72819e183: Status 404 returned error can't find the container with id 90622bbc2dcc424568522e5c3bb8439ecb6f62197bc27abf9dac5be72819e183 Oct 30 14:16:24.980504 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:24.980311 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb32943-75vlb" event={"ID":"1d77678e-4328-4e2e-8dca-19d17111b3cb","Type":"ContainerStarted","Data":"90622bbc2dcc424568522e5c3bb8439ecb6f62197bc27abf9dac5be72819e183"} Oct 30 14:16:25.498583 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:25.498540 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:16:26.042660 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:26.042060 4982 generic.go:356] "Generic (PLEG): container finished" podID="1d77678e-4328-4e2e-8dca-19d17111b3cb" containerID="71f3493b5caff4cbf22b653bec40cdfea317252cd104b3dd2cd1bc6de8086b5c" exitCode=0 Oct 30 14:16:26.042660 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:26.042205 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb32943-75vlb" event={"ID":"1d77678e-4328-4e2e-8dca-19d17111b3cb","Type":"ContainerDied","Data":"71f3493b5caff4cbf22b653bec40cdfea317252cd104b3dd2cd1bc6de8086b5c"} Oct 30 14:16:26.851762 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:26.851723 4982 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-mqc7s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:16:26.852354 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:26.852308 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mqc7s" podUID="f784c488-9f50-4924-a54b-811e5be8661f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:16:27.561587 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:27.561548 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-01955278-7cfd4f7cbd-sj77j"] Oct 30 14:16:27.563242 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:27.563209 4982 cpu_manager.go:414] "containerMap: removing container" podUID="74c90d32-f9a9-40f4-9834-77268ed4a2d9" containerName="cleanup" Oct 30 14:16:27.563242 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:27.563239 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74c90d32-f9a9-40f4-9834-77268ed4a2d9" containerName="cleanup" Oct 30 14:16:27.563433 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:27.563263 4982 cpu_manager.go:414] "containerMap: removing container" podUID="198d48ae-abd7-4d73-81c1-60c3b826ed0c" containerName="cleanup" Oct 30 14:16:27.563433 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:27.563272 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="198d48ae-abd7-4d73-81c1-60c3b826ed0c" containerName="cleanup" Oct 30 14:16:27.563433 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:27.563288 4982 cpu_manager.go:414] "containerMap: removing container" podUID="8b7bcb9c-a287-4cf1-86f3-3ed486f210e1" containerName="cleanup" Oct 30 14:16:27.563433 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:27.563293 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b7bcb9c-a287-4cf1-86f3-3ed486f210e1" containerName="cleanup" Oct 30 14:16:27.563711 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:27.563455 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b7bcb9c-a287-4cf1-86f3-3ed486f210e1" containerName="cleanup" Oct 30 14:16:27.563711 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:27.563467 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="74c90d32-f9a9-40f4-9834-77268ed4a2d9" containerName="cleanup" Oct 30 14:16:27.563711 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:27.563474 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="198d48ae-abd7-4d73-81c1-60c3b826ed0c" containerName="cleanup" Oct 30 14:16:27.567552 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:27.567528 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-01955278-7cfd4f7cbd-sj77j" Oct 30 14:16:27.590256 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:27.583243 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb32943-75vlb" Oct 30 14:16:27.590256 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:27.586970 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-01955278-serving-cert\"" Oct 30 14:16:27.610907 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:27.610713 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-01955278-7cfd4f7cbd-sj77j"] Oct 30 14:16:27.693162 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:27.693127 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:16:27.746447 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:27.738912 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d77678e-4328-4e2e-8dca-19d17111b3cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d77678e-4328-4e2e-8dca-19d17111b3cb-maas-api-openshift-ca\") pod \"1d77678e-4328-4e2e-8dca-19d17111b3cb\" (UID: \"1d77678e-4328-4e2e-8dca-19d17111b3cb\") " Oct 30 14:16:27.746447 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:27.739066 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d77678e-4328-4e2e-8dca-19d17111b3cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d77678e-4328-4e2e-8dca-19d17111b3cb-maas-api-xks-ca\") pod \"1d77678e-4328-4e2e-8dca-19d17111b3cb\" (UID: \"1d77678e-4328-4e2e-8dca-19d17111b3cb\") " Oct 30 14:16:27.746447 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:27.739106 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d77678e-4328-4e2e-8dca-19d17111b3cb-kube-api-access-mt7gh\" (UniqueName: \"kubernetes.io/projected/1d77678e-4328-4e2e-8dca-19d17111b3cb-kube-api-access-mt7gh\") pod \"1d77678e-4328-4e2e-8dca-19d17111b3cb\" (UID: \"1d77678e-4328-4e2e-8dca-19d17111b3cb\") " Oct 30 14:16:27.746447 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:27.739342 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/708748da-ecb9-4028-bb6e-c4ed4b889921-metrics-certs\") pod \"maas-api-e2e-mt-01955278-7cfd4f7cbd-sj77j\" (UID: \"708748da-ecb9-4028-bb6e-c4ed4b889921\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-01955278-7cfd4f7cbd-sj77j" Oct 30 14:16:27.746447 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:27.739426 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/708748da-ecb9-4028-bb6e-c4ed4b889921-maas-api-tls\") pod \"maas-api-e2e-mt-01955278-7cfd4f7cbd-sj77j\" (UID: \"708748da-ecb9-4028-bb6e-c4ed4b889921\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-01955278-7cfd4f7cbd-sj77j" Oct 30 14:16:27.746447 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:27.739450 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsvg9\" (UniqueName: \"kubernetes.io/projected/708748da-ecb9-4028-bb6e-c4ed4b889921-kube-api-access-hsvg9\") pod \"maas-api-e2e-mt-01955278-7cfd4f7cbd-sj77j\" (UID: \"708748da-ecb9-4028-bb6e-c4ed4b889921\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-01955278-7cfd4f7cbd-sj77j" Oct 30 14:16:27.746447 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:27.739487 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/708748da-ecb9-4028-bb6e-c4ed4b889921-openshift-service-ca\") pod \"maas-api-e2e-mt-01955278-7cfd4f7cbd-sj77j\" (UID: \"708748da-ecb9-4028-bb6e-c4ed4b889921\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-01955278-7cfd4f7cbd-sj77j" Oct 30 14:16:27.746447 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:27.740367 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d77678e-4328-4e2e-8dca-19d17111b3cb-maas-api-openshift-ca" pod "1d77678e-4328-4e2e-8dca-19d17111b3cb" (UID: "1d77678e-4328-4e2e-8dca-19d17111b3cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:16:27.746447 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:27.743454 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d77678e-4328-4e2e-8dca-19d17111b3cb-kube-api-access-mt7gh" pod "1d77678e-4328-4e2e-8dca-19d17111b3cb" (UID: "1d77678e-4328-4e2e-8dca-19d17111b3cb"). InnerVolumeSpecName "kube-api-access-mt7gh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:16:27.746447 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:27.744026 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d77678e-4328-4e2e-8dca-19d17111b3cb-maas-api-xks-ca" pod "1d77678e-4328-4e2e-8dca-19d17111b3cb" (UID: "1d77678e-4328-4e2e-8dca-19d17111b3cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:16:27.844165 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:27.841789 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/708748da-ecb9-4028-bb6e-c4ed4b889921-metrics-certs\") pod \"maas-api-e2e-mt-01955278-7cfd4f7cbd-sj77j\" (UID: \"708748da-ecb9-4028-bb6e-c4ed4b889921\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-01955278-7cfd4f7cbd-sj77j" Oct 30 14:16:27.844165 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:27.842454 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/708748da-ecb9-4028-bb6e-c4ed4b889921-maas-api-tls\") pod \"maas-api-e2e-mt-01955278-7cfd4f7cbd-sj77j\" (UID: \"708748da-ecb9-4028-bb6e-c4ed4b889921\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-01955278-7cfd4f7cbd-sj77j" Oct 30 14:16:27.844165 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:27.842502 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsvg9\" (UniqueName: \"kubernetes.io/projected/708748da-ecb9-4028-bb6e-c4ed4b889921-kube-api-access-hsvg9\") pod \"maas-api-e2e-mt-01955278-7cfd4f7cbd-sj77j\" (UID: \"708748da-ecb9-4028-bb6e-c4ed4b889921\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-01955278-7cfd4f7cbd-sj77j" Oct 30 14:16:27.844165 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:27.842557 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/708748da-ecb9-4028-bb6e-c4ed4b889921-openshift-service-ca\") pod \"maas-api-e2e-mt-01955278-7cfd4f7cbd-sj77j\" (UID: \"708748da-ecb9-4028-bb6e-c4ed4b889921\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-01955278-7cfd4f7cbd-sj77j" Oct 30 14:16:27.844165 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:16:27.842601 4982 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-01955278-serving-cert: secret "maas-api-e2e-mt-01955278-serving-cert" not found Oct 30 14:16:27.844165 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:16:27.842708 4982 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/708748da-ecb9-4028-bb6e-c4ed4b889921-maas-api-tls podName:708748da-ecb9-4028-bb6e-c4ed4b889921 nodeName:}" failed. No retries permitted until 2026-10-30 14:16:28.342686362 +0000 UTC m=+1658.086820399 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/708748da-ecb9-4028-bb6e-c4ed4b889921-maas-api-tls") pod "maas-api-e2e-mt-01955278-7cfd4f7cbd-sj77j" (UID: "708748da-ecb9-4028-bb6e-c4ed4b889921") : secret "maas-api-e2e-mt-01955278-serving-cert" not found Oct 30 14:16:27.844165 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:27.842890 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mt7gh\" (UniqueName: \"kubernetes.io/projected/1d77678e-4328-4e2e-8dca-19d17111b3cb-kube-api-access-mt7gh\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:16:27.844165 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:27.842911 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d77678e-4328-4e2e-8dca-19d17111b3cb-maas-api-openshift-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:16:27.844165 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:27.842926 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d77678e-4328-4e2e-8dca-19d17111b3cb-maas-api-xks-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:16:27.844165 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:27.843840 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/708748da-ecb9-4028-bb6e-c4ed4b889921-openshift-service-ca\") pod \"maas-api-e2e-mt-01955278-7cfd4f7cbd-sj77j\" (UID: \"708748da-ecb9-4028-bb6e-c4ed4b889921\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-01955278-7cfd4f7cbd-sj77j" Oct 30 14:16:27.845092 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:27.845071 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/708748da-ecb9-4028-bb6e-c4ed4b889921-metrics-certs\") pod \"maas-api-e2e-mt-01955278-7cfd4f7cbd-sj77j\" (UID: \"708748da-ecb9-4028-bb6e-c4ed4b889921\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-01955278-7cfd4f7cbd-sj77j" Oct 30 14:16:27.859286 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:27.859253 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:16:27.871915 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:27.871891 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsvg9\" (UniqueName: \"kubernetes.io/projected/708748da-ecb9-4028-bb6e-c4ed4b889921-kube-api-access-hsvg9\") pod \"maas-api-e2e-mt-01955278-7cfd4f7cbd-sj77j\" (UID: \"708748da-ecb9-4028-bb6e-c4ed4b889921\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-01955278-7cfd4f7cbd-sj77j" Oct 30 14:16:28.034239 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:16:28.034193 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:16:28.069961 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:28.069927 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb32943-75vlb" Oct 30 14:16:28.070121 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:28.069959 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb32943-75vlb" event={"ID":"1d77678e-4328-4e2e-8dca-19d17111b3cb","Type":"ContainerDied","Data":"90622bbc2dcc424568522e5c3bb8439ecb6f62197bc27abf9dac5be72819e183"} Oct 30 14:16:28.070121 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:28.070006 4982 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90622bbc2dcc424568522e5c3bb8439ecb6f62197bc27abf9dac5be72819e183" Oct 30 14:16:28.247072 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:28.246993 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:16:28.350712 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:28.350676 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/708748da-ecb9-4028-bb6e-c4ed4b889921-maas-api-tls\") pod \"maas-api-e2e-mt-01955278-7cfd4f7cbd-sj77j\" (UID: \"708748da-ecb9-4028-bb6e-c4ed4b889921\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-01955278-7cfd4f7cbd-sj77j" Oct 30 14:16:28.353390 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:28.353358 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/708748da-ecb9-4028-bb6e-c4ed4b889921-maas-api-tls\") pod \"maas-api-e2e-mt-01955278-7cfd4f7cbd-sj77j\" (UID: \"708748da-ecb9-4028-bb6e-c4ed4b889921\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-01955278-7cfd4f7cbd-sj77j" Oct 30 14:16:28.397055 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:28.396976 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:16:28.511873 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:28.511779 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-01955278-7cfd4f7cbd-sj77j" Oct 30 14:16:28.674789 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:28.674744 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:16:28.916633 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:28.916592 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-01955278-7cfd4f7cbd-sj77j"] Oct 30 14:16:29.079896 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:29.079847 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-01955278-7cfd4f7cbd-sj77j" event={"ID":"708748da-ecb9-4028-bb6e-c4ed4b889921","Type":"ContainerStarted","Data":"4d6171ca7dd2d82c5f54b0698a6b4c4ac4fee55b07f9809a8e00339d3b1a0aa1"} Oct 30 14:16:29.131560 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:29.131524 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b"] Oct 30 14:16:29.133726 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:29.133699 4982 cpu_manager.go:414] "containerMap: removing container" podUID="1d77678e-4328-4e2e-8dca-19d17111b3cb" containerName="revoke-keys" Oct 30 14:16:29.133726 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:29.133724 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d77678e-4328-4e2e-8dca-19d17111b3cb" containerName="revoke-keys" Oct 30 14:16:29.133926 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:29.133913 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d77678e-4328-4e2e-8dca-19d17111b3cb" containerName="revoke-keys" Oct 30 14:16:29.139316 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:29.139293 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b" Oct 30 14:16:29.182578 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:29.182489 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b"] Oct 30 14:16:29.189409 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:29.187799 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-228454-openshift-default-dockercfg-nlnt2\"" Oct 30 14:16:29.217700 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:29.216701 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52b260c5-122f-4e2a-8344-56d8efbb66f6-istio-envoy\") pod \"e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b\" (UID: \"52b260c5-122f-4e2a-8344-56d8efbb66f6\") " pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b" Oct 30 14:16:29.217700 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:29.216781 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52b260c5-122f-4e2a-8344-56d8efbb66f6-workload-socket\") pod \"e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b\" (UID: \"52b260c5-122f-4e2a-8344-56d8efbb66f6\") " pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b" Oct 30 14:16:29.217700 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:29.216836 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52b260c5-122f-4e2a-8344-56d8efbb66f6-istiod-ca-cert\") pod \"e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b\" (UID: \"52b260c5-122f-4e2a-8344-56d8efbb66f6\") " pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b" Oct 30 14:16:29.217700 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:29.216902 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52b260c5-122f-4e2a-8344-56d8efbb66f6-credential-socket\") pod \"e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b\" (UID: \"52b260c5-122f-4e2a-8344-56d8efbb66f6\") " pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b" Oct 30 14:16:29.217700 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:29.216965 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52b260c5-122f-4e2a-8344-56d8efbb66f6-istio-podinfo\") pod \"e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b\" (UID: \"52b260c5-122f-4e2a-8344-56d8efbb66f6\") " pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b" Oct 30 14:16:29.217700 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:29.217073 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52b260c5-122f-4e2a-8344-56d8efbb66f6-istio-data\") pod \"e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b\" (UID: \"52b260c5-122f-4e2a-8344-56d8efbb66f6\") " pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b" Oct 30 14:16:29.217700 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:29.217153 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52b260c5-122f-4e2a-8344-56d8efbb66f6-istio-token\") pod \"e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b\" (UID: \"52b260c5-122f-4e2a-8344-56d8efbb66f6\") " pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b" Oct 30 14:16:29.217700 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:29.217204 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq6s6\" (UniqueName: \"kubernetes.io/projected/52b260c5-122f-4e2a-8344-56d8efbb66f6-kube-api-access-hq6s6\") pod \"e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b\" (UID: \"52b260c5-122f-4e2a-8344-56d8efbb66f6\") " pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b" Oct 30 14:16:29.217700 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:29.217371 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52b260c5-122f-4e2a-8344-56d8efbb66f6-workload-certs\") pod \"e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b\" (UID: \"52b260c5-122f-4e2a-8344-56d8efbb66f6\") " pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b" Oct 30 14:16:29.237135 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:29.237104 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:16:29.318765 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:29.318728 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52b260c5-122f-4e2a-8344-56d8efbb66f6-workload-certs\") pod \"e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b\" (UID: \"52b260c5-122f-4e2a-8344-56d8efbb66f6\") " pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b" Oct 30 14:16:29.323525 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:29.323475 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52b260c5-122f-4e2a-8344-56d8efbb66f6-workload-certs\") pod \"e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b\" (UID: \"52b260c5-122f-4e2a-8344-56d8efbb66f6\") " pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b" Oct 30 14:16:29.324927 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:29.323796 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52b260c5-122f-4e2a-8344-56d8efbb66f6-istio-envoy\") pod \"e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b\" (UID: \"52b260c5-122f-4e2a-8344-56d8efbb66f6\") " pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b" Oct 30 14:16:29.324927 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:29.323861 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52b260c5-122f-4e2a-8344-56d8efbb66f6-workload-socket\") pod \"e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b\" (UID: \"52b260c5-122f-4e2a-8344-56d8efbb66f6\") " pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b" Oct 30 14:16:29.324927 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:29.323902 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52b260c5-122f-4e2a-8344-56d8efbb66f6-istiod-ca-cert\") pod \"e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b\" (UID: \"52b260c5-122f-4e2a-8344-56d8efbb66f6\") " pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b" Oct 30 14:16:29.324927 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:29.323967 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52b260c5-122f-4e2a-8344-56d8efbb66f6-credential-socket\") pod \"e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b\" (UID: \"52b260c5-122f-4e2a-8344-56d8efbb66f6\") " pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b" Oct 30 14:16:29.324927 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:29.324023 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52b260c5-122f-4e2a-8344-56d8efbb66f6-istio-podinfo\") pod \"e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b\" (UID: \"52b260c5-122f-4e2a-8344-56d8efbb66f6\") " pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b" Oct 30 14:16:29.324927 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:29.324060 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52b260c5-122f-4e2a-8344-56d8efbb66f6-istio-data\") pod \"e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b\" (UID: \"52b260c5-122f-4e2a-8344-56d8efbb66f6\") " pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b" Oct 30 14:16:29.324927 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:29.324122 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52b260c5-122f-4e2a-8344-56d8efbb66f6-istio-token\") pod \"e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b\" (UID: \"52b260c5-122f-4e2a-8344-56d8efbb66f6\") " pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b" Oct 30 14:16:29.324927 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:29.324166 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq6s6\" (UniqueName: \"kubernetes.io/projected/52b260c5-122f-4e2a-8344-56d8efbb66f6-kube-api-access-hq6s6\") pod \"e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b\" (UID: \"52b260c5-122f-4e2a-8344-56d8efbb66f6\") " pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b" Oct 30 14:16:29.327294 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:29.325687 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52b260c5-122f-4e2a-8344-56d8efbb66f6-credential-socket\") pod \"e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b\" (UID: \"52b260c5-122f-4e2a-8344-56d8efbb66f6\") " pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b" Oct 30 14:16:29.327294 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:29.325944 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52b260c5-122f-4e2a-8344-56d8efbb66f6-workload-socket\") pod \"e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b\" (UID: \"52b260c5-122f-4e2a-8344-56d8efbb66f6\") " pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b" Oct 30 14:16:29.327294 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:29.326572 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52b260c5-122f-4e2a-8344-56d8efbb66f6-istiod-ca-cert\") pod \"e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b\" (UID: \"52b260c5-122f-4e2a-8344-56d8efbb66f6\") " pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b" Oct 30 14:16:29.327294 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:29.326854 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52b260c5-122f-4e2a-8344-56d8efbb66f6-istio-data\") pod \"e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b\" (UID: \"52b260c5-122f-4e2a-8344-56d8efbb66f6\") " pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b" Oct 30 14:16:29.336206 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:29.336153 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52b260c5-122f-4e2a-8344-56d8efbb66f6-istio-envoy\") pod \"e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b\" (UID: \"52b260c5-122f-4e2a-8344-56d8efbb66f6\") " pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b" Oct 30 14:16:29.378502 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:29.378144 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52b260c5-122f-4e2a-8344-56d8efbb66f6-istio-podinfo\") pod \"e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b\" (UID: \"52b260c5-122f-4e2a-8344-56d8efbb66f6\") " pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b" Oct 30 14:16:29.468771 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:29.467821 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:16:29.489669 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:29.489561 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq6s6\" (UniqueName: \"kubernetes.io/projected/52b260c5-122f-4e2a-8344-56d8efbb66f6-kube-api-access-hq6s6\") pod \"e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b\" (UID: \"52b260c5-122f-4e2a-8344-56d8efbb66f6\") " pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b" Oct 30 14:16:29.528227 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:29.528148 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52b260c5-122f-4e2a-8344-56d8efbb66f6-istio-token\") pod \"e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b\" (UID: \"52b260c5-122f-4e2a-8344-56d8efbb66f6\") " pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b" Oct 30 14:16:29.548572 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:29.548264 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b" Oct 30 14:16:29.764619 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:29.764469 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:16:30.071696 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:30.071665 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b"] Oct 30 14:16:30.073915 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:16:30.073884 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52b260c5_122f_4e2a_8344_56d8efbb66f6.slice/crio-98bc3ae3d4b9fa4ebcd8a3022f9e9a8e99ec3232c5168e72971ce1d7dcb0fb52 WatchSource:0}: Error finding container 98bc3ae3d4b9fa4ebcd8a3022f9e9a8e99ec3232c5168e72971ce1d7dcb0fb52: Status 404 returned error can't find the container with id 98bc3ae3d4b9fa4ebcd8a3022f9e9a8e99ec3232c5168e72971ce1d7dcb0fb52 Oct 30 14:16:30.079999 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:30.077771 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:16:30.079999 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:30.077859 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:16:30.079999 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:30.077901 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:16:30.118548 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:30.118509 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-01955278-7cfd4f7cbd-sj77j" event={"ID":"708748da-ecb9-4028-bb6e-c4ed4b889921","Type":"ContainerStarted","Data":"95e9079256043fc8e4a7d94ad5241e9ead21b6c8c9f3d7b2cdde024352704931"} Oct 30 14:16:30.118745 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:30.118613 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-01955278-7cfd4f7cbd-sj77j" Oct 30 14:16:30.124710 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:30.124679 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b" event={"ID":"52b260c5-122f-4e2a-8344-56d8efbb66f6","Type":"ContainerStarted","Data":"98bc3ae3d4b9fa4ebcd8a3022f9e9a8e99ec3232c5168e72971ce1d7dcb0fb52"} Oct 30 14:16:30.171720 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:30.170334 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-01955278-7cfd4f7cbd-sj77j" podStartSLOduration=2.824870928 podStartE2EDuration="3.17031486s" totalImagesPullingTime="345.443932ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:16:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:16:30.167393302 +0000 UTC m=+1659.911527345" watchObservedRunningTime="2026-10-30 14:16:30.17031486 +0000 UTC m=+1659.914448905" Oct 30 14:16:31.138688 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:31.134167 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b" event={"ID":"52b260c5-122f-4e2a-8344-56d8efbb66f6","Type":"ContainerStarted","Data":"1eb60eea9f4ff8eebbfc71960921badc5a46788f9c55856ed6dea790469f8e6b"} Oct 30 14:16:31.548948 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:31.548868 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b" Oct 30 14:16:31.555604 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:31.555545 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b" Oct 30 14:16:31.608436 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:31.608379 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b" podStartSLOduration=2.608360298 podStartE2EDuration="2.608360298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:16:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:16:31.202870851 +0000 UTC m=+1660.947004895" watchObservedRunningTime="2026-10-30 14:16:31.608360298 +0000 UTC m=+1661.352494342" Oct 30 14:16:31.629010 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:31.628974 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:16:32.141648 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:32.141605 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b" Oct 30 14:16:32.142420 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:32.142399 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b" Oct 30 14:16:33.637745 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:33.637672 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:16:34.406059 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:34.406009 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:16:35.417706 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:35.417669 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:16:36.139595 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:36.139564 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-01955278-7cfd4f7cbd-sj77j" Oct 30 14:16:37.262292 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:37.262252 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:16:39.144782 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:39.144748 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:16:40.966050 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:40.964114 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:16:41.849767 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:41.849732 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:16:42.965456 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:42.965401 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:16:43.468020 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:43.467979 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:16:43.679683 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:43.679605 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:16:43.824333 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:43.824280 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:16:45.156126 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:45.156094 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:16:45.558246 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:45.558205 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:16:45.706155 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:45.706101 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:16:47.424114 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:47.424075 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:16:49.247275 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:49.247228 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:16:51.009206 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:51.009165 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:16:52.854734 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:52.854702 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:16:54.256072 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:54.256036 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:16:54.846492 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:54.846446 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:16:56.910097 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:56.910059 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:16:58.148313 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:16:58.148258 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:16:59.166225 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:16:59.166171 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:01.024853 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:01.024817 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:01.472037 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:01.472000 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:02.377135 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:02.377078 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:02.531169 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:02.531126 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:03.007234 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:03.007197 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:03.159051 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:03.158993 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:05.040623 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:05.040571 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:05.516190 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:05.516155 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:06.007430 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:06.007385 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:06.673493 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:06.673446 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:06.936264 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:06.936157 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:08.769055 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:08.769001 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:10.151427 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:10.151392 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:10.426263 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:10.426190 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:10.671332 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:10.671302 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:10.882676 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:10.876815 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:12.454510 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:12.454478 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:12.604121 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:12.604083 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:13.145382 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:13.145338 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:14.541273 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:14.541236 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:14.964075 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:14.964040 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:15.117957 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:15.117926 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:16.634064 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:16.634008 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:16.797038 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:16.797005 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:18.731574 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:18.731530 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:20.551157 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:20.551121 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:20.706322 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:20.706279 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:20.859779 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:20.859732 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:22.282598 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:22.282567 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:22.409194 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:22.409156 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-972fee-fdbb7989f-mdfj4"] Oct 30 14:17:22.414834 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:22.414809 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-972fee-fdbb7989f-mdfj4" Oct 30 14:17:22.430149 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:22.430121 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-972fee-fdbb7989f-mdfj4"] Oct 30 14:17:22.434604 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:22.434580 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:22.436990 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:22.436956 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-972fee-dockercfg-ssg9l\"" Oct 30 14:17:22.474895 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:22.474862 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-972fee\"" Oct 30 14:17:22.536714 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:22.535805 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw6wp\" (UniqueName: \"kubernetes.io/projected/d9f378f2-91ec-4bab-b862-ae525e311b73-kube-api-access-vw6wp\") pod \"payload-pre-processing-e2e-api-a-972fee-fdbb7989f-mdfj4\" (UID: \"d9f378f2-91ec-4bab-b862-ae525e311b73\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-972fee-fdbb7989f-mdfj4" Oct 30 14:17:22.536714 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:22.535928 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9f378f2-91ec-4bab-b862-ae525e311b73-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-972fee-fdbb7989f-mdfj4\" (UID: \"d9f378f2-91ec-4bab-b862-ae525e311b73\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-972fee-fdbb7989f-mdfj4" Oct 30 14:17:22.670722 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:22.670676 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw6wp\" (UniqueName: \"kubernetes.io/projected/d9f378f2-91ec-4bab-b862-ae525e311b73-kube-api-access-vw6wp\") pod \"payload-pre-processing-e2e-api-a-972fee-fdbb7989f-mdfj4\" (UID: \"d9f378f2-91ec-4bab-b862-ae525e311b73\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-972fee-fdbb7989f-mdfj4" Oct 30 14:17:22.671111 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:22.671073 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9f378f2-91ec-4bab-b862-ae525e311b73-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-972fee-fdbb7989f-mdfj4\" (UID: \"d9f378f2-91ec-4bab-b862-ae525e311b73\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-972fee-fdbb7989f-mdfj4" Oct 30 14:17:22.672403 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:22.672374 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9f378f2-91ec-4bab-b862-ae525e311b73-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-972fee-fdbb7989f-mdfj4\" (UID: \"d9f378f2-91ec-4bab-b862-ae525e311b73\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-972fee-fdbb7989f-mdfj4" Oct 30 14:17:22.715235 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:22.715198 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw6wp\" (UniqueName: \"kubernetes.io/projected/d9f378f2-91ec-4bab-b862-ae525e311b73-kube-api-access-vw6wp\") pod \"payload-pre-processing-e2e-api-a-972fee-fdbb7989f-mdfj4\" (UID: \"d9f378f2-91ec-4bab-b862-ae525e311b73\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-972fee-fdbb7989f-mdfj4" Oct 30 14:17:22.739849 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:22.739813 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-972fee-fdbb7989f-mdfj4" Oct 30 14:17:23.034530 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:23.034481 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:23.069205 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:23.068984 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-972fee-fdbb7989f-mdfj4"] Oct 30 14:17:23.638050 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:23.638014 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-972fee-fdbb7989f-mdfj4" event={"ID":"d9f378f2-91ec-4bab-b862-ae525e311b73","Type":"ContainerStarted","Data":"bcd82b4492fadfcc077ed2a80072ccbf00dde225bacbdcdf4badbe2d930d5f14"} Oct 30 14:17:23.638525 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:23.638065 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-972fee-fdbb7989f-mdfj4" event={"ID":"d9f378f2-91ec-4bab-b862-ae525e311b73","Type":"ContainerStarted","Data":"49ea30aefafe3701704280073cf317ff29461b2cda7b23583deed3bd0ecced7a"} Oct 30 14:17:23.638525 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:23.638094 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-972fee-fdbb7989f-mdfj4" Oct 30 14:17:23.686033 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:23.685974 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-972fee-fdbb7989f-mdfj4" podStartSLOduration=1.685954389 podStartE2EDuration="1.685954389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:17:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:17:23.681277628 +0000 UTC m=+1713.425411671" watchObservedRunningTime="2026-10-30 14:17:23.685954389 +0000 UTC m=+1713.430088431" Oct 30 14:17:24.833242 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:24.833204 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:25.865977 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:25.865937 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:26.015264 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:26.015224 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9dba4af9-6d98fd4964-4fmbr"] Oct 30 14:17:26.088709 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:26.087453 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:26.235537 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:26.235448 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:26.387082 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:26.379997 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:26.742489 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:26.742451 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:27.442427 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:27.442391 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:28.015985 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:17:28.015943 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:17:28.056600 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:28.056565 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:28.207668 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:28.202085 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:28.770711 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:28.770669 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:30.620388 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:30.620347 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:31.017771 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:31.016590 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-9dba4af9-6d98fd4964-4fmbr" podUID="83f22477-fe63-43ef-9752-9cddc4c93627" containerName="payload-processing" containerID="cri-o://06c105066840871b3164e6e308d1e37ca90f661bdd7b66e1994946f339485dd3" gracePeriod=25 Oct 30 14:17:32.519345 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:32.516284 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:33.457606 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:33.457563 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-9dba4af9-6d98fd4964-4fmbr" podUID="83f22477-fe63-43ef-9752-9cddc4c93627" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:17:34.159360 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:34.159325 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2e8b995-lslrj"] Oct 30 14:17:34.164535 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:34.164509 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e8b995-lslrj" Oct 30 14:17:34.197276 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:34.197229 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2e8b995-lslrj"] Oct 30 14:17:34.211688 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:34.207059 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:17:34.211688 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:34.207283 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:17:34.266392 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:34.266346 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7zgt\" (UniqueName: \"kubernetes.io/projected/ce6f63ba-b1b7-4c90-a89c-a8d5e1c1e50a-kube-api-access-r7zgt\") pod \"maas-api-revoke-sub-e2e8b995-lslrj\" (UID: \"ce6f63ba-b1b7-4c90-a89c-a8d5e1c1e50a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e8b995-lslrj" Oct 30 14:17:34.266729 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:34.266707 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce6f63ba-b1b7-4c90-a89c-a8d5e1c1e50a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2e8b995-lslrj\" (UID: \"ce6f63ba-b1b7-4c90-a89c-a8d5e1c1e50a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e8b995-lslrj" Oct 30 14:17:34.266925 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:34.266911 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce6f63ba-b1b7-4c90-a89c-a8d5e1c1e50a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2e8b995-lslrj\" (UID: \"ce6f63ba-b1b7-4c90-a89c-a8d5e1c1e50a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e8b995-lslrj" Oct 30 14:17:34.368347 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:34.368304 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7zgt\" (UniqueName: \"kubernetes.io/projected/ce6f63ba-b1b7-4c90-a89c-a8d5e1c1e50a-kube-api-access-r7zgt\") pod \"maas-api-revoke-sub-e2e8b995-lslrj\" (UID: \"ce6f63ba-b1b7-4c90-a89c-a8d5e1c1e50a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e8b995-lslrj" Oct 30 14:17:34.368524 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:34.368353 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce6f63ba-b1b7-4c90-a89c-a8d5e1c1e50a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2e8b995-lslrj\" (UID: \"ce6f63ba-b1b7-4c90-a89c-a8d5e1c1e50a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e8b995-lslrj" Oct 30 14:17:34.368524 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:34.368383 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce6f63ba-b1b7-4c90-a89c-a8d5e1c1e50a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2e8b995-lslrj\" (UID: \"ce6f63ba-b1b7-4c90-a89c-a8d5e1c1e50a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e8b995-lslrj" Oct 30 14:17:34.369101 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:34.369076 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce6f63ba-b1b7-4c90-a89c-a8d5e1c1e50a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2e8b995-lslrj\" (UID: \"ce6f63ba-b1b7-4c90-a89c-a8d5e1c1e50a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e8b995-lslrj" Oct 30 14:17:34.370902 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:34.370877 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce6f63ba-b1b7-4c90-a89c-a8d5e1c1e50a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2e8b995-lslrj\" (UID: \"ce6f63ba-b1b7-4c90-a89c-a8d5e1c1e50a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e8b995-lslrj" Oct 30 14:17:34.416813 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:34.408527 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7zgt\" (UniqueName: \"kubernetes.io/projected/ce6f63ba-b1b7-4c90-a89c-a8d5e1c1e50a-kube-api-access-r7zgt\") pod \"maas-api-revoke-sub-e2e8b995-lslrj\" (UID: \"ce6f63ba-b1b7-4c90-a89c-a8d5e1c1e50a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e8b995-lslrj" Oct 30 14:17:34.479536 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:34.479508 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e8b995-lslrj" Oct 30 14:17:34.539105 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:34.539016 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:34.649852 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:34.649809 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-972fee-fdbb7989f-mdfj4" Oct 30 14:17:34.919710 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:34.919665 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2e8b995-lslrj"] Oct 30 14:17:34.923614 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:17:34.923582 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce6f63ba_b1b7_4c90_a89c_a8d5e1c1e50a.slice/crio-9c915e46e1875a278607ddd8f195e02251a17e3beac42eb205bdbfe1ae4c71dd WatchSource:0}: Error finding container 9c915e46e1875a278607ddd8f195e02251a17e3beac42eb205bdbfe1ae4c71dd: Status 404 returned error can't find the container with id 9c915e46e1875a278607ddd8f195e02251a17e3beac42eb205bdbfe1ae4c71dd Oct 30 14:17:35.850539 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:35.850501 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e8b995-lslrj" event={"ID":"ce6f63ba-b1b7-4c90-a89c-a8d5e1c1e50a","Type":"ContainerStarted","Data":"d3a8c712cd8a3e327fc14d04987c71e04245084a562c3535f366ce37d4850fbb"} Oct 30 14:17:35.850951 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:35.850560 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e8b995-lslrj" event={"ID":"ce6f63ba-b1b7-4c90-a89c-a8d5e1c1e50a","Type":"ContainerStarted","Data":"9c915e46e1875a278607ddd8f195e02251a17e3beac42eb205bdbfe1ae4c71dd"} Oct 30 14:17:35.905662 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:35.905237 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e8b995-lslrj" podStartSLOduration=1.9052167089999998 podStartE2EDuration="1.905216709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:17:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:17:35.896006641 +0000 UTC m=+1725.640140688" watchObservedRunningTime="2026-10-30 14:17:35.905216709 +0000 UTC m=+1725.649350757" Oct 30 14:17:36.356291 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:36.356254 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:36.858609 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:36.858573 4982 generic.go:356] "Generic (PLEG): container finished" podID="ce6f63ba-b1b7-4c90-a89c-a8d5e1c1e50a" containerID="d3a8c712cd8a3e327fc14d04987c71e04245084a562c3535f366ce37d4850fbb" exitCode=0 Oct 30 14:17:36.859010 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:36.858668 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e8b995-lslrj" event={"ID":"ce6f63ba-b1b7-4c90-a89c-a8d5e1c1e50a","Type":"ContainerDied","Data":"d3a8c712cd8a3e327fc14d04987c71e04245084a562c3535f366ce37d4850fbb"} Oct 30 14:17:38.063318 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:38.063284 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e8b995-lslrj" Oct 30 14:17:38.155500 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:38.155463 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:38.157548 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:38.157516 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce6f63ba-b1b7-4c90-a89c-a8d5e1c1e50a-kube-api-access-r7zgt\" (UniqueName: \"kubernetes.io/projected/ce6f63ba-b1b7-4c90-a89c-a8d5e1c1e50a-kube-api-access-r7zgt\") pod \"ce6f63ba-b1b7-4c90-a89c-a8d5e1c1e50a\" (UID: \"ce6f63ba-b1b7-4c90-a89c-a8d5e1c1e50a\") " Oct 30 14:17:38.157737 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:38.157583 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce6f63ba-b1b7-4c90-a89c-a8d5e1c1e50a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce6f63ba-b1b7-4c90-a89c-a8d5e1c1e50a-maas-api-xks-ca\") pod \"ce6f63ba-b1b7-4c90-a89c-a8d5e1c1e50a\" (UID: \"ce6f63ba-b1b7-4c90-a89c-a8d5e1c1e50a\") " Oct 30 14:17:38.157737 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:38.157661 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce6f63ba-b1b7-4c90-a89c-a8d5e1c1e50a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce6f63ba-b1b7-4c90-a89c-a8d5e1c1e50a-maas-api-openshift-ca\") pod \"ce6f63ba-b1b7-4c90-a89c-a8d5e1c1e50a\" (UID: \"ce6f63ba-b1b7-4c90-a89c-a8d5e1c1e50a\") " Oct 30 14:17:38.158253 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:38.158200 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce6f63ba-b1b7-4c90-a89c-a8d5e1c1e50a-maas-api-openshift-ca" pod "ce6f63ba-b1b7-4c90-a89c-a8d5e1c1e50a" (UID: "ce6f63ba-b1b7-4c90-a89c-a8d5e1c1e50a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:17:38.158617 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:38.158596 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce6f63ba-b1b7-4c90-a89c-a8d5e1c1e50a-maas-api-openshift-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:17:38.159756 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:38.159724 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce6f63ba-b1b7-4c90-a89c-a8d5e1c1e50a-kube-api-access-r7zgt" pod "ce6f63ba-b1b7-4c90-a89c-a8d5e1c1e50a" (UID: "ce6f63ba-b1b7-4c90-a89c-a8d5e1c1e50a"). InnerVolumeSpecName "kube-api-access-r7zgt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:17:38.160156 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:38.160130 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce6f63ba-b1b7-4c90-a89c-a8d5e1c1e50a-maas-api-xks-ca" pod "ce6f63ba-b1b7-4c90-a89c-a8d5e1c1e50a" (UID: "ce6f63ba-b1b7-4c90-a89c-a8d5e1c1e50a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:17:38.260547 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:38.260508 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7zgt\" (UniqueName: \"kubernetes.io/projected/ce6f63ba-b1b7-4c90-a89c-a8d5e1c1e50a-kube-api-access-r7zgt\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:17:38.260547 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:38.260544 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce6f63ba-b1b7-4c90-a89c-a8d5e1c1e50a-maas-api-xks-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:17:38.877723 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:38.877688 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e8b995-lslrj" Oct 30 14:17:38.877905 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:38.877715 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e8b995-lslrj" event={"ID":"ce6f63ba-b1b7-4c90-a89c-a8d5e1c1e50a","Type":"ContainerDied","Data":"9c915e46e1875a278607ddd8f195e02251a17e3beac42eb205bdbfe1ae4c71dd"} Oct 30 14:17:38.877905 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:38.877774 4982 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c915e46e1875a278607ddd8f195e02251a17e3beac42eb205bdbfe1ae4c71dd" Oct 30 14:17:39.948831 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:39.948783 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:41.887575 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:41.887493 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:41.912700 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:41.908297 4982 generic.go:356] "Generic (PLEG): container finished" podID="52b260c5-122f-4e2a-8344-56d8efbb66f6" containerID="1eb60eea9f4ff8eebbfc71960921badc5a46788f9c55856ed6dea790469f8e6b" exitCode=0 Oct 30 14:17:41.912700 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:41.908357 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b" event={"ID":"52b260c5-122f-4e2a-8344-56d8efbb66f6","Type":"ContainerDied","Data":"1eb60eea9f4ff8eebbfc71960921badc5a46788f9c55856ed6dea790469f8e6b"} Oct 30 14:17:41.912700 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:41.909579 4982 scope.go:122] "RemoveContainer" containerID="1eb60eea9f4ff8eebbfc71960921badc5a46788f9c55856ed6dea790469f8e6b" Oct 30 14:17:41.912700 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:41.911949 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:17:41.912700 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:41.912056 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:17:41.912700 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:41.912140 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:17:42.162400 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:42.162308 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:42.326485 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:42.326448 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:42.925540 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:42.925504 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b" event={"ID":"52b260c5-122f-4e2a-8344-56d8efbb66f6","Type":"ContainerStarted","Data":"f581e9dc825db0ec024b40e0f6cf9f589ec9508428476950a821eb17bb42d707"} Oct 30 14:17:43.458845 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:43.458807 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-9dba4af9-6d98fd4964-4fmbr" podUID="83f22477-fe63-43ef-9752-9cddc4c93627" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:17:43.548758 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:43.548712 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b" Oct 30 14:17:43.549635 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:43.549611 4982 patch_prober.go:29] interesting pod/e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 14:17:43.549773 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:43.549671 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b" podUID="52b260c5-122f-4e2a-8344-56d8efbb66f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 14:17:43.610158 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:43.610084 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:44.054777 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:44.039331 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:44.250169 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:44.250125 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:44.456675 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:44.453447 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:44.551203 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:44.551031 4982 patch_prober.go:29] interesting pod/e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 14:17:44.551203 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:44.551106 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b" podUID="52b260c5-122f-4e2a-8344-56d8efbb66f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 14:17:45.274662 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:45.273680 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:45.549969 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:45.549598 4982 patch_prober.go:29] interesting pod/e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 14:17:45.549969 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:45.549665 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b" podUID="52b260c5-122f-4e2a-8344-56d8efbb66f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 14:17:46.544045 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:46.543986 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:46.549300 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:46.549255 4982 patch_prober.go:29] interesting pod/e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 14:17:46.549472 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:46.549325 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b" podUID="52b260c5-122f-4e2a-8344-56d8efbb66f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 14:17:47.142128 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:47.142087 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b" Oct 30 14:17:47.549583 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:47.549492 4982 patch_prober.go:29] interesting pod/e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 14:17:47.549977 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:47.549568 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b" podUID="52b260c5-122f-4e2a-8344-56d8efbb66f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 14:17:48.410030 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:48.409982 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:48.754103 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:48.753985 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b" Oct 30 14:17:48.755657 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:48.755617 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b" Oct 30 14:17:49.770568 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:49.770529 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:50.260778 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:50.260739 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:50.925536 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:50.925504 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:51.074472 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:51.074431 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:51.244043 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:51.244011 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:51.334943 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:51.334921 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9dba4af9-6d98fd4964-4fmbr" Oct 30 14:17:51.404737 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:51.404703 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83f22477-fe63-43ef-9752-9cddc4c93627-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83f22477-fe63-43ef-9752-9cddc4c93627-plugins-config-volume\") pod \"83f22477-fe63-43ef-9752-9cddc4c93627\" (UID: \"83f22477-fe63-43ef-9752-9cddc4c93627\") " Oct 30 14:17:51.404912 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:51.404834 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83f22477-fe63-43ef-9752-9cddc4c93627-kube-api-access-5w7gt\" (UniqueName: \"kubernetes.io/projected/83f22477-fe63-43ef-9752-9cddc4c93627-kube-api-access-5w7gt\") pod \"83f22477-fe63-43ef-9752-9cddc4c93627\" (UID: \"83f22477-fe63-43ef-9752-9cddc4c93627\") " Oct 30 14:17:51.405257 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:51.405233 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83f22477-fe63-43ef-9752-9cddc4c93627-plugins-config-volume" pod "83f22477-fe63-43ef-9752-9cddc4c93627" (UID: "83f22477-fe63-43ef-9752-9cddc4c93627"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:17:51.406208 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:51.406177 4982 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83f22477-fe63-43ef-9752-9cddc4c93627-plugins-config-volume\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:17:51.407319 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:51.407292 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83f22477-fe63-43ef-9752-9cddc4c93627-kube-api-access-5w7gt" pod "83f22477-fe63-43ef-9752-9cddc4c93627" (UID: "83f22477-fe63-43ef-9752-9cddc4c93627"). InnerVolumeSpecName "kube-api-access-5w7gt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:17:51.507849 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:51.507769 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w7gt\" (UniqueName: \"kubernetes.io/projected/83f22477-fe63-43ef-9752-9cddc4c93627-kube-api-access-5w7gt\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:17:51.654299 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:51.654266 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:51.814660 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:51.813047 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:52.146764 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:52.146672 4982 generic.go:356] "Generic (PLEG): container finished" podID="83f22477-fe63-43ef-9752-9cddc4c93627" containerID="06c105066840871b3164e6e308d1e37ca90f661bdd7b66e1994946f339485dd3" exitCode=0 Oct 30 14:17:52.147196 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:52.146759 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9dba4af9-6d98fd4964-4fmbr" Oct 30 14:17:52.147196 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:52.146772 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9dba4af9-6d98fd4964-4fmbr" event={"ID":"83f22477-fe63-43ef-9752-9cddc4c93627","Type":"ContainerDied","Data":"06c105066840871b3164e6e308d1e37ca90f661bdd7b66e1994946f339485dd3"} Oct 30 14:17:52.147196 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:52.146827 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9dba4af9-6d98fd4964-4fmbr" event={"ID":"83f22477-fe63-43ef-9752-9cddc4c93627","Type":"ContainerDied","Data":"3ba46ea7b6865035b0cd8c38bf9f88cfdbfd1a2af83b91c52aeac690886a7592"} Oct 30 14:17:52.147196 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:52.146860 4982 scope.go:122] "RemoveContainer" containerID="06c105066840871b3164e6e308d1e37ca90f661bdd7b66e1994946f339485dd3" Oct 30 14:17:52.159686 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:52.159662 4982 scope.go:122] "RemoveContainer" containerID="06c105066840871b3164e6e308d1e37ca90f661bdd7b66e1994946f339485dd3" Oct 30 14:17:52.160018 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:17:52.159991 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06c105066840871b3164e6e308d1e37ca90f661bdd7b66e1994946f339485dd3\": container with ID starting with 06c105066840871b3164e6e308d1e37ca90f661bdd7b66e1994946f339485dd3 not found: ID does not exist" containerID="06c105066840871b3164e6e308d1e37ca90f661bdd7b66e1994946f339485dd3" Oct 30 14:17:52.160080 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:52.160027 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06c105066840871b3164e6e308d1e37ca90f661bdd7b66e1994946f339485dd3"} err="failed to get container status \"06c105066840871b3164e6e308d1e37ca90f661bdd7b66e1994946f339485dd3\": rpc error: code = NotFound desc = could not find container \"06c105066840871b3164e6e308d1e37ca90f661bdd7b66e1994946f339485dd3\": container with ID starting with 06c105066840871b3164e6e308d1e37ca90f661bdd7b66e1994946f339485dd3 not found: ID does not exist" Oct 30 14:17:52.210254 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:52.210217 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9dba4af9-6d98fd4964-4fmbr"] Oct 30 14:17:52.223664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:52.223584 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9dba4af9-6d98fd4964-4fmbr"] Oct 30 14:17:52.422767 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:52.417629 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83f22477-fe63-43ef-9752-9cddc4c93627" path="/var/lib/kubelet/pods/83f22477-fe63-43ef-9752-9cddc4c93627/volumes" Oct 30 14:17:52.451041 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:52.451007 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:52.621141 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:52.621109 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:52.764597 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:52.764509 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:54.456199 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:54.456153 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:55.663525 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:55.663490 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:55.813038 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:55.813001 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:56.409429 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:56.409398 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk"] Oct 30 14:17:56.409625 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:56.409601 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk" podUID="e8fa477f-0e45-42fa-9941-514ea90d1094" containerName="istio-proxy" containerID="cri-o://5ccc3fa1dfa138b1c81421115baf983c445eca622ac209bdd53cd57bbf935daa" gracePeriod=30 Oct 30 14:17:56.621402 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:56.621281 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:57.356280 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:57.356244 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:57.505040 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:57.504998 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2b206b-5745bc7d79-5s7b7"] Oct 30 14:17:57.505833 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:57.505809 4982 cpu_manager.go:414] "containerMap: removing container" podUID="83f22477-fe63-43ef-9752-9cddc4c93627" containerName="payload-processing" Oct 30 14:17:57.505833 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:57.505830 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83f22477-fe63-43ef-9752-9cddc4c93627" containerName="payload-processing" Oct 30 14:17:57.505991 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:57.505868 4982 cpu_manager.go:414] "containerMap: removing container" podUID="ce6f63ba-b1b7-4c90-a89c-a8d5e1c1e50a" containerName="revoke-keys" Oct 30 14:17:57.505991 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:57.505876 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce6f63ba-b1b7-4c90-a89c-a8d5e1c1e50a" containerName="revoke-keys" Oct 30 14:17:57.506076 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:57.506016 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="83f22477-fe63-43ef-9752-9cddc4c93627" containerName="payload-processing" Oct 30 14:17:57.506076 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:57.506036 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce6f63ba-b1b7-4c90-a89c-a8d5e1c1e50a" containerName="revoke-keys" Oct 30 14:17:57.506278 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:57.506255 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:57.514047 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:57.513931 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2b206b-5745bc7d79-5s7b7" Oct 30 14:17:57.531630 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:57.531567 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2b206b-5745bc7d79-5s7b7"] Oct 30 14:17:57.538488 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:57.538457 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-2b206b\"" Oct 30 14:17:57.538623 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:57.538457 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-2b206b-dockercfg-rl7pf\"" Oct 30 14:17:57.631458 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:57.631369 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrdqp\" (UniqueName: \"kubernetes.io/projected/f40cc27f-7493-4296-9ed2-c2183be0059f-kube-api-access-nrdqp\") pod \"payload-pre-processing-e2e-api-a-2b206b-5745bc7d79-5s7b7\" (UID: \"f40cc27f-7493-4296-9ed2-c2183be0059f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2b206b-5745bc7d79-5s7b7" Oct 30 14:17:57.631783 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:57.631755 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f40cc27f-7493-4296-9ed2-c2183be0059f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2b206b-5745bc7d79-5s7b7\" (UID: \"f40cc27f-7493-4296-9ed2-c2183be0059f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2b206b-5745bc7d79-5s7b7" Oct 30 14:17:57.805029 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:57.804963 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrdqp\" (UniqueName: \"kubernetes.io/projected/f40cc27f-7493-4296-9ed2-c2183be0059f-kube-api-access-nrdqp\") pod \"payload-pre-processing-e2e-api-a-2b206b-5745bc7d79-5s7b7\" (UID: \"f40cc27f-7493-4296-9ed2-c2183be0059f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2b206b-5745bc7d79-5s7b7" Oct 30 14:17:57.820272 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:57.820237 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f40cc27f-7493-4296-9ed2-c2183be0059f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2b206b-5745bc7d79-5s7b7\" (UID: \"f40cc27f-7493-4296-9ed2-c2183be0059f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2b206b-5745bc7d79-5s7b7" Oct 30 14:17:57.821893 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:57.821319 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f40cc27f-7493-4296-9ed2-c2183be0059f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2b206b-5745bc7d79-5s7b7\" (UID: \"f40cc27f-7493-4296-9ed2-c2183be0059f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2b206b-5745bc7d79-5s7b7" Oct 30 14:17:57.857660 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:57.856960 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrdqp\" (UniqueName: \"kubernetes.io/projected/f40cc27f-7493-4296-9ed2-c2183be0059f-kube-api-access-nrdqp\") pod \"payload-pre-processing-e2e-api-a-2b206b-5745bc7d79-5s7b7\" (UID: \"f40cc27f-7493-4296-9ed2-c2183be0059f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2b206b-5745bc7d79-5s7b7" Oct 30 14:17:58.024868 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:17:58.024766 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:17:58.141246 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:58.141215 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2b206b-5745bc7d79-5s7b7" Oct 30 14:17:58.457162 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:17:58.457136 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf40cc27f_7493_4296_9ed2_c2183be0059f.slice/crio-7487df9434d2854031e8e30a8448ab917dd917d0e656500a07f89e555d0ebfe8 WatchSource:0}: Error finding container 7487df9434d2854031e8e30a8448ab917dd917d0e656500a07f89e555d0ebfe8: Status 404 returned error can't find the container with id 7487df9434d2854031e8e30a8448ab917dd917d0e656500a07f89e555d0ebfe8 Oct 30 14:17:58.459545 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:58.459507 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2b206b-5745bc7d79-5s7b7"] Oct 30 14:17:58.739418 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:58.739299 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:17:59.220791 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:59.220754 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2b206b-5745bc7d79-5s7b7" event={"ID":"f40cc27f-7493-4296-9ed2-c2183be0059f","Type":"ContainerStarted","Data":"80bde65d39d326d6889d5fe9b071ab54c98d17bec580bfd0350fc6a6cd4e2365"} Oct 30 14:17:59.220993 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:59.220813 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2b206b-5745bc7d79-5s7b7" event={"ID":"f40cc27f-7493-4296-9ed2-c2183be0059f","Type":"ContainerStarted","Data":"7487df9434d2854031e8e30a8448ab917dd917d0e656500a07f89e555d0ebfe8"} Oct 30 14:17:59.220993 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:59.220969 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2b206b-5745bc7d79-5s7b7" Oct 30 14:17:59.266790 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:17:59.266740 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2b206b-5745bc7d79-5s7b7" podStartSLOduration=2.2667258 podStartE2EDuration="2.2667258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:17:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:17:59.263866906 +0000 UTC m=+1749.008000962" watchObservedRunningTime="2026-10-30 14:17:59.2667258 +0000 UTC m=+1749.010859876" Oct 30 14:18:00.657557 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:00.657509 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:18:01.709872 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:01.709843 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk" Oct 30 14:18:01.796250 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:01.796212 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8fa477f-0e45-42fa-9941-514ea90d1094-kube-api-access-c6gz4\" (UniqueName: \"kubernetes.io/projected/e8fa477f-0e45-42fa-9941-514ea90d1094-kube-api-access-c6gz4\") pod \"e8fa477f-0e45-42fa-9941-514ea90d1094\" (UID: \"e8fa477f-0e45-42fa-9941-514ea90d1094\") " Oct 30 14:18:01.796601 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:01.796310 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8fa477f-0e45-42fa-9941-514ea90d1094-istio-token\" (UniqueName: \"kubernetes.io/projected/e8fa477f-0e45-42fa-9941-514ea90d1094-istio-token\") pod \"e8fa477f-0e45-42fa-9941-514ea90d1094\" (UID: \"e8fa477f-0e45-42fa-9941-514ea90d1094\") " Oct 30 14:18:01.796601 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:01.796342 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8fa477f-0e45-42fa-9941-514ea90d1094-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8fa477f-0e45-42fa-9941-514ea90d1094-istiod-ca-cert\") pod \"e8fa477f-0e45-42fa-9941-514ea90d1094\" (UID: \"e8fa477f-0e45-42fa-9941-514ea90d1094\") " Oct 30 14:18:01.796601 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:01.796375 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8fa477f-0e45-42fa-9941-514ea90d1094-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8fa477f-0e45-42fa-9941-514ea90d1094-credential-socket\") pod \"e8fa477f-0e45-42fa-9941-514ea90d1094\" (UID: \"e8fa477f-0e45-42fa-9941-514ea90d1094\") " Oct 30 14:18:01.796601 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:01.796452 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8fa477f-0e45-42fa-9941-514ea90d1094-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8fa477f-0e45-42fa-9941-514ea90d1094-workload-socket\") pod \"e8fa477f-0e45-42fa-9941-514ea90d1094\" (UID: \"e8fa477f-0e45-42fa-9941-514ea90d1094\") " Oct 30 14:18:01.796601 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:01.796507 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8fa477f-0e45-42fa-9941-514ea90d1094-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8fa477f-0e45-42fa-9941-514ea90d1094-workload-certs\") pod \"e8fa477f-0e45-42fa-9941-514ea90d1094\" (UID: \"e8fa477f-0e45-42fa-9941-514ea90d1094\") " Oct 30 14:18:01.796601 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:01.796558 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e8fa477f-0e45-42fa-9941-514ea90d1094-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8fa477f-0e45-42fa-9941-514ea90d1094-istio-podinfo\") pod \"e8fa477f-0e45-42fa-9941-514ea90d1094\" (UID: \"e8fa477f-0e45-42fa-9941-514ea90d1094\") " Oct 30 14:18:01.797067 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:01.796722 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8fa477f-0e45-42fa-9941-514ea90d1094-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8fa477f-0e45-42fa-9941-514ea90d1094-istio-data\") pod \"e8fa477f-0e45-42fa-9941-514ea90d1094\" (UID: \"e8fa477f-0e45-42fa-9941-514ea90d1094\") " Oct 30 14:18:01.797067 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:01.796748 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8fa477f-0e45-42fa-9941-514ea90d1094-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8fa477f-0e45-42fa-9941-514ea90d1094-istio-envoy\") pod \"e8fa477f-0e45-42fa-9941-514ea90d1094\" (UID: \"e8fa477f-0e45-42fa-9941-514ea90d1094\") " Oct 30 14:18:01.798155 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:01.798126 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8fa477f-0e45-42fa-9941-514ea90d1094-credential-socket" pod "e8fa477f-0e45-42fa-9941-514ea90d1094" (UID: "e8fa477f-0e45-42fa-9941-514ea90d1094"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:18:01.798451 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:01.798429 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8fa477f-0e45-42fa-9941-514ea90d1094-istiod-ca-cert" pod "e8fa477f-0e45-42fa-9941-514ea90d1094" (UID: "e8fa477f-0e45-42fa-9941-514ea90d1094"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:18:01.798720 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:01.798527 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8fa477f-0e45-42fa-9941-514ea90d1094-workload-certs" pod "e8fa477f-0e45-42fa-9941-514ea90d1094" (UID: "e8fa477f-0e45-42fa-9941-514ea90d1094"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:18:01.798988 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:01.798961 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8fa477f-0e45-42fa-9941-514ea90d1094-workload-socket" pod "e8fa477f-0e45-42fa-9941-514ea90d1094" (UID: "e8fa477f-0e45-42fa-9941-514ea90d1094"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:18:01.799053 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:01.799028 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8fa477f-0e45-42fa-9941-514ea90d1094-istio-data" pod "e8fa477f-0e45-42fa-9941-514ea90d1094" (UID: "e8fa477f-0e45-42fa-9941-514ea90d1094"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:18:01.807720 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:01.803434 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8fa477f-0e45-42fa-9941-514ea90d1094-istio-envoy" pod "e8fa477f-0e45-42fa-9941-514ea90d1094" (UID: "e8fa477f-0e45-42fa-9941-514ea90d1094"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:18:01.807720 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:01.803578 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8fa477f-0e45-42fa-9941-514ea90d1094-kube-api-access-c6gz4" pod "e8fa477f-0e45-42fa-9941-514ea90d1094" (UID: "e8fa477f-0e45-42fa-9941-514ea90d1094"). InnerVolumeSpecName "kube-api-access-c6gz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:18:01.807720 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:01.803684 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8fa477f-0e45-42fa-9941-514ea90d1094-istio-token" pod "e8fa477f-0e45-42fa-9941-514ea90d1094" (UID: "e8fa477f-0e45-42fa-9941-514ea90d1094"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:18:01.807720 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:01.804756 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8fa477f-0e45-42fa-9941-514ea90d1094-istio-podinfo" pod "e8fa477f-0e45-42fa-9941-514ea90d1094" (UID: "e8fa477f-0e45-42fa-9941-514ea90d1094"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:18:01.900080 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:01.899765 4982 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8fa477f-0e45-42fa-9941-514ea90d1094-istio-data\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:18:01.900080 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:01.899845 4982 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8fa477f-0e45-42fa-9941-514ea90d1094-istio-envoy\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:18:01.900080 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:01.899862 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6gz4\" (UniqueName: \"kubernetes.io/projected/e8fa477f-0e45-42fa-9941-514ea90d1094-kube-api-access-c6gz4\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:18:01.900080 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:01.899879 4982 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8fa477f-0e45-42fa-9941-514ea90d1094-istio-token\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:18:01.900080 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:01.899893 4982 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8fa477f-0e45-42fa-9941-514ea90d1094-istiod-ca-cert\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:18:01.900080 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:01.899906 4982 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8fa477f-0e45-42fa-9941-514ea90d1094-credential-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:18:01.900080 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:01.899919 4982 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8fa477f-0e45-42fa-9941-514ea90d1094-workload-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:18:01.900080 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:01.899934 4982 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8fa477f-0e45-42fa-9941-514ea90d1094-workload-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:18:01.900080 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:01.899979 4982 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8fa477f-0e45-42fa-9941-514ea90d1094-istio-podinfo\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:18:02.258721 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:02.258684 4982 generic.go:356] "Generic (PLEG): container finished" podID="e8fa477f-0e45-42fa-9941-514ea90d1094" containerID="5ccc3fa1dfa138b1c81421115baf983c445eca622ac209bdd53cd57bbf935daa" exitCode=0 Oct 30 14:18:02.258911 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:02.258749 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk" Oct 30 14:18:02.258911 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:02.258779 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk" event={"ID":"e8fa477f-0e45-42fa-9941-514ea90d1094","Type":"ContainerDied","Data":"5ccc3fa1dfa138b1c81421115baf983c445eca622ac209bdd53cd57bbf935daa"} Oct 30 14:18:02.258911 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:02.258874 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk" event={"ID":"e8fa477f-0e45-42fa-9941-514ea90d1094","Type":"ContainerDied","Data":"385b43eb2169b20ea3142273f63625efa674b2b904eb69ea3e077192b12d055e"} Oct 30 14:18:02.258911 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:02.258906 4982 scope.go:122] "RemoveContainer" containerID="5ccc3fa1dfa138b1c81421115baf983c445eca622ac209bdd53cd57bbf935daa" Oct 30 14:18:02.271494 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:02.271472 4982 scope.go:122] "RemoveContainer" containerID="5ccc3fa1dfa138b1c81421115baf983c445eca622ac209bdd53cd57bbf935daa" Oct 30 14:18:02.271824 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:18:02.271803 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ccc3fa1dfa138b1c81421115baf983c445eca622ac209bdd53cd57bbf935daa\": container with ID starting with 5ccc3fa1dfa138b1c81421115baf983c445eca622ac209bdd53cd57bbf935daa not found: ID does not exist" containerID="5ccc3fa1dfa138b1c81421115baf983c445eca622ac209bdd53cd57bbf935daa" Oct 30 14:18:02.271911 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:02.271831 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ccc3fa1dfa138b1c81421115baf983c445eca622ac209bdd53cd57bbf935daa"} err="failed to get container status \"5ccc3fa1dfa138b1c81421115baf983c445eca622ac209bdd53cd57bbf935daa\": rpc error: code = NotFound desc = could not find container \"5ccc3fa1dfa138b1c81421115baf983c445eca622ac209bdd53cd57bbf935daa\": container with ID starting with 5ccc3fa1dfa138b1c81421115baf983c445eca622ac209bdd53cd57bbf935daa not found: ID does not exist" Oct 30 14:18:02.318387 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:02.318354 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk"] Oct 30 14:18:02.338700 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:02.338543 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zqcnk"] Oct 30 14:18:02.404545 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:02.404506 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8fa477f-0e45-42fa-9941-514ea90d1094" path="/var/lib/kubelet/pods/e8fa477f-0e45-42fa-9941-514ea90d1094/volumes" Oct 30 14:18:02.450768 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:02.450730 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:18:04.525973 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:04.525941 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:18:06.315801 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:06.315754 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:18:08.368804 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:08.368773 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:18:10.230929 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:10.230897 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2b206b-5745bc7d79-5s7b7" Oct 30 14:18:10.439098 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:10.439059 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:18:12.229071 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:12.229025 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:18:13.806188 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:13.806148 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:18:14.371873 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:14.371829 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:18:14.524263 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:14.524222 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:18:14.677421 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:14.677327 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:18:15.399077 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:15.399037 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:18:16.541083 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:16.541040 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:18:16.688927 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:16.688889 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:18:17.613067 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:17.613026 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:18:18.644899 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:18.644847 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:18:20.441432 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:20.441391 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:18:22.303877 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:22.303844 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:18:22.810941 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:22.810910 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:18:24.479929 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:24.479890 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:18:26.423946 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:26.423912 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf"] Oct 30 14:18:26.424440 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:26.424131 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf" podUID="b616a96d-9d08-4171-89a5-d84cd2039e4a" containerName="istio-proxy" containerID="cri-o://733e0862c3083fff16ee42062bb8acc9c5aeed7936283a0f45e45e08f3f49653" gracePeriod=30 Oct 30 14:18:26.440449 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:26.440422 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:18:26.587505 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:26.587455 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:18:27.193966 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:27.193927 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:18:27.712116 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:27.712077 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:18:27.959548 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:27.959508 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:18:28.091343 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:18:28.091303 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:18:28.433387 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:28.433298 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:18:28.787087 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:28.786996 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:18:30.306233 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:30.306179 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:18:30.467071 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:30.467025 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:18:30.636991 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:30.636958 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:18:31.734200 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:31.733860 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf" Oct 30 14:18:31.734997 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:31.734972 4982 generic.go:356] "Generic (PLEG): container finished" podID="b616a96d-9d08-4171-89a5-d84cd2039e4a" containerID="733e0862c3083fff16ee42062bb8acc9c5aeed7936283a0f45e45e08f3f49653" exitCode=0 Oct 30 14:18:31.735108 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:31.735051 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf" event={"ID":"b616a96d-9d08-4171-89a5-d84cd2039e4a","Type":"ContainerDied","Data":"733e0862c3083fff16ee42062bb8acc9c5aeed7936283a0f45e45e08f3f49653"} Oct 30 14:18:31.735152 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:31.735116 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf" event={"ID":"b616a96d-9d08-4171-89a5-d84cd2039e4a","Type":"ContainerDied","Data":"efbd281cefff27db5e04d6462d1e8a0bdd21d30e7e80c25ecc6739f8a4f60bd2"} Oct 30 14:18:31.735186 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:31.735148 4982 scope.go:122] "RemoveContainer" containerID="733e0862c3083fff16ee42062bb8acc9c5aeed7936283a0f45e45e08f3f49653" Oct 30 14:18:31.778318 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:31.776871 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b616a96d-9d08-4171-89a5-d84cd2039e4a-istio-token\" (UniqueName: \"kubernetes.io/projected/b616a96d-9d08-4171-89a5-d84cd2039e4a-istio-token\") pod \"b616a96d-9d08-4171-89a5-d84cd2039e4a\" (UID: \"b616a96d-9d08-4171-89a5-d84cd2039e4a\") " Oct 30 14:18:31.778728 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:31.778612 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b616a96d-9d08-4171-89a5-d84cd2039e4a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b616a96d-9d08-4171-89a5-d84cd2039e4a-workload-socket\") pod \"b616a96d-9d08-4171-89a5-d84cd2039e4a\" (UID: \"b616a96d-9d08-4171-89a5-d84cd2039e4a\") " Oct 30 14:18:31.778962 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:31.778904 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b616a96d-9d08-4171-89a5-d84cd2039e4a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b616a96d-9d08-4171-89a5-d84cd2039e4a-workload-certs\") pod \"b616a96d-9d08-4171-89a5-d84cd2039e4a\" (UID: \"b616a96d-9d08-4171-89a5-d84cd2039e4a\") " Oct 30 14:18:31.779128 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:31.779093 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b616a96d-9d08-4171-89a5-d84cd2039e4a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b616a96d-9d08-4171-89a5-d84cd2039e4a-istio-podinfo\") pod \"b616a96d-9d08-4171-89a5-d84cd2039e4a\" (UID: \"b616a96d-9d08-4171-89a5-d84cd2039e4a\") " Oct 30 14:18:31.779445 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:31.779367 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b616a96d-9d08-4171-89a5-d84cd2039e4a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b616a96d-9d08-4171-89a5-d84cd2039e4a-istio-data\") pod \"b616a96d-9d08-4171-89a5-d84cd2039e4a\" (UID: \"b616a96d-9d08-4171-89a5-d84cd2039e4a\") " Oct 30 14:18:31.779445 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:31.779416 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b616a96d-9d08-4171-89a5-d84cd2039e4a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b616a96d-9d08-4171-89a5-d84cd2039e4a-credential-socket\") pod \"b616a96d-9d08-4171-89a5-d84cd2039e4a\" (UID: \"b616a96d-9d08-4171-89a5-d84cd2039e4a\") " Oct 30 14:18:31.779905 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:31.779735 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b616a96d-9d08-4171-89a5-d84cd2039e4a-kube-api-access-dxl6w\" (UniqueName: \"kubernetes.io/projected/b616a96d-9d08-4171-89a5-d84cd2039e4a-kube-api-access-dxl6w\") pod \"b616a96d-9d08-4171-89a5-d84cd2039e4a\" (UID: \"b616a96d-9d08-4171-89a5-d84cd2039e4a\") " Oct 30 14:18:31.780499 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:31.780083 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b616a96d-9d08-4171-89a5-d84cd2039e4a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b616a96d-9d08-4171-89a5-d84cd2039e4a-istiod-ca-cert\") pod \"b616a96d-9d08-4171-89a5-d84cd2039e4a\" (UID: \"b616a96d-9d08-4171-89a5-d84cd2039e4a\") " Oct 30 14:18:31.780499 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:31.780164 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b616a96d-9d08-4171-89a5-d84cd2039e4a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b616a96d-9d08-4171-89a5-d84cd2039e4a-istio-envoy\") pod \"b616a96d-9d08-4171-89a5-d84cd2039e4a\" (UID: \"b616a96d-9d08-4171-89a5-d84cd2039e4a\") " Oct 30 14:18:31.783078 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:31.783049 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b616a96d-9d08-4171-89a5-d84cd2039e4a-istio-data" pod "b616a96d-9d08-4171-89a5-d84cd2039e4a" (UID: "b616a96d-9d08-4171-89a5-d84cd2039e4a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:18:31.785253 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:31.784985 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b616a96d-9d08-4171-89a5-d84cd2039e4a-workload-socket" pod "b616a96d-9d08-4171-89a5-d84cd2039e4a" (UID: "b616a96d-9d08-4171-89a5-d84cd2039e4a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:18:31.788282 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:31.788018 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b616a96d-9d08-4171-89a5-d84cd2039e4a-istiod-ca-cert" pod "b616a96d-9d08-4171-89a5-d84cd2039e4a" (UID: "b616a96d-9d08-4171-89a5-d84cd2039e4a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:18:31.788813 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:31.788540 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b616a96d-9d08-4171-89a5-d84cd2039e4a-istio-envoy" pod "b616a96d-9d08-4171-89a5-d84cd2039e4a" (UID: "b616a96d-9d08-4171-89a5-d84cd2039e4a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:18:31.788973 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:31.788940 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b616a96d-9d08-4171-89a5-d84cd2039e4a-credential-socket" pod "b616a96d-9d08-4171-89a5-d84cd2039e4a" (UID: "b616a96d-9d08-4171-89a5-d84cd2039e4a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:18:31.790792 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:31.790687 4982 scope.go:122] "RemoveContainer" containerID="733e0862c3083fff16ee42062bb8acc9c5aeed7936283a0f45e45e08f3f49653" Oct 30 14:18:31.791216 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:18:31.791186 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"733e0862c3083fff16ee42062bb8acc9c5aeed7936283a0f45e45e08f3f49653\": container with ID starting with 733e0862c3083fff16ee42062bb8acc9c5aeed7936283a0f45e45e08f3f49653 not found: ID does not exist" containerID="733e0862c3083fff16ee42062bb8acc9c5aeed7936283a0f45e45e08f3f49653" Oct 30 14:18:31.791301 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:31.791227 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"733e0862c3083fff16ee42062bb8acc9c5aeed7936283a0f45e45e08f3f49653"} err="failed to get container status \"733e0862c3083fff16ee42062bb8acc9c5aeed7936283a0f45e45e08f3f49653\": rpc error: code = NotFound desc = could not find container \"733e0862c3083fff16ee42062bb8acc9c5aeed7936283a0f45e45e08f3f49653\": container with ID starting with 733e0862c3083fff16ee42062bb8acc9c5aeed7936283a0f45e45e08f3f49653 not found: ID does not exist" Oct 30 14:18:31.794246 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:31.794222 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b616a96d-9d08-4171-89a5-d84cd2039e4a-kube-api-access-dxl6w" pod "b616a96d-9d08-4171-89a5-d84cd2039e4a" (UID: "b616a96d-9d08-4171-89a5-d84cd2039e4a"). InnerVolumeSpecName "kube-api-access-dxl6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:18:31.795321 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:31.794261 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b616a96d-9d08-4171-89a5-d84cd2039e4a-istio-token" pod "b616a96d-9d08-4171-89a5-d84cd2039e4a" (UID: "b616a96d-9d08-4171-89a5-d84cd2039e4a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:18:31.799950 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:31.795991 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b616a96d-9d08-4171-89a5-d84cd2039e4a-istio-podinfo" pod "b616a96d-9d08-4171-89a5-d84cd2039e4a" (UID: "b616a96d-9d08-4171-89a5-d84cd2039e4a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:18:31.800064 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:31.800042 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b616a96d-9d08-4171-89a5-d84cd2039e4a-workload-certs" pod "b616a96d-9d08-4171-89a5-d84cd2039e4a" (UID: "b616a96d-9d08-4171-89a5-d84cd2039e4a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:18:31.887079 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:31.887038 4982 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b616a96d-9d08-4171-89a5-d84cd2039e4a-istio-token\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:18:31.887079 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:31.887080 4982 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b616a96d-9d08-4171-89a5-d84cd2039e4a-workload-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:18:31.887339 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:31.887096 4982 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b616a96d-9d08-4171-89a5-d84cd2039e4a-workload-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:18:31.887339 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:31.887110 4982 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b616a96d-9d08-4171-89a5-d84cd2039e4a-istio-podinfo\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:18:31.887339 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:31.887122 4982 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b616a96d-9d08-4171-89a5-d84cd2039e4a-istio-data\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:18:31.887339 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:31.887136 4982 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b616a96d-9d08-4171-89a5-d84cd2039e4a-credential-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:18:31.887339 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:31.887153 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxl6w\" (UniqueName: \"kubernetes.io/projected/b616a96d-9d08-4171-89a5-d84cd2039e4a-kube-api-access-dxl6w\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:18:31.887339 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:31.887169 4982 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b616a96d-9d08-4171-89a5-d84cd2039e4a-istiod-ca-cert\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:18:31.887339 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:31.887182 4982 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b616a96d-9d08-4171-89a5-d84cd2039e4a-istio-envoy\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:18:31.973503 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:31.973435 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:18:32.132857 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:32.132821 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:18:32.357664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:32.354227 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:18:32.745250 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:32.745173 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf" Oct 30 14:18:32.746581 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:32.746551 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:18:32.824969 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:32.824929 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf"] Oct 30 14:18:32.854940 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:32.854892 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-28ksf"] Oct 30 14:18:32.902287 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:32.902251 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:18:33.466170 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:33.466137 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:18:33.668662 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:33.665793 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:18:33.890903 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:33.890601 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:18:34.049737 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:34.049700 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:18:34.473317 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:34.467349 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b616a96d-9d08-4171-89a5-d84cd2039e4a" path="/var/lib/kubelet/pods/b616a96d-9d08-4171-89a5-d84cd2039e4a/volumes" Oct 30 14:18:35.218242 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:35.218199 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:18:37.222980 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:37.222940 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:18:37.537723 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:37.533291 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:18:38.858245 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:38.858204 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:18:39.017050 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:39.017011 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:18:39.418820 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:39.418769 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:18:41.726447 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:41.726412 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:18:42.275553 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:42.275515 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc"] Oct 30 14:18:42.275786 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:42.275764 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc" podUID="1ed4b142-fae5-4489-818c-8b53e3fea272" containerName="istio-proxy" containerID="cri-o://7bae0c3d4a401a4cb6b00e460fbb02f8e318ccac576fe114d38cd7f58d21a233" gracePeriod=30 Oct 30 14:18:42.357952 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:42.357921 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:18:42.562690 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:42.562626 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:18:43.051264 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:43.051168 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:18:43.294625 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:43.294595 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:18:43.499222 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:43.499187 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-01955278-hjfp4"] Oct 30 14:18:43.500324 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:43.500286 4982 cpu_manager.go:414] "containerMap: removing container" podUID="b616a96d-9d08-4171-89a5-d84cd2039e4a" containerName="istio-proxy" Oct 30 14:18:43.500324 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:43.500320 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b616a96d-9d08-4171-89a5-d84cd2039e4a" containerName="istio-proxy" Oct 30 14:18:43.500519 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:43.500337 4982 cpu_manager.go:414] "containerMap: removing container" podUID="e8fa477f-0e45-42fa-9941-514ea90d1094" containerName="istio-proxy" Oct 30 14:18:43.500519 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:43.500343 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8fa477f-0e45-42fa-9941-514ea90d1094" containerName="istio-proxy" Oct 30 14:18:43.500519 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:43.500475 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="b616a96d-9d08-4171-89a5-d84cd2039e4a" containerName="istio-proxy" Oct 30 14:18:43.500519 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:43.500486 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8fa477f-0e45-42fa-9941-514ea90d1094" containerName="istio-proxy" Oct 30 14:18:43.506792 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:43.506767 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-01955278-hjfp4" Oct 30 14:18:43.528332 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:43.528302 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-01955278-hjfp4"] Oct 30 14:18:43.535353 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:43.535330 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:18:43.535707 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:43.535689 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:18:43.568549 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:43.566469 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfnt6\" (UniqueName: \"kubernetes.io/projected/de490cc2-45ff-4fe8-894f-46273f27e3d4-kube-api-access-hfnt6\") pod \"maas-api-revoke-keys-e2e-mt-01955278-hjfp4\" (UID: \"de490cc2-45ff-4fe8-894f-46273f27e3d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-01955278-hjfp4" Oct 30 14:18:43.568549 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:43.566520 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de490cc2-45ff-4fe8-894f-46273f27e3d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-01955278-hjfp4\" (UID: \"de490cc2-45ff-4fe8-894f-46273f27e3d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-01955278-hjfp4" Oct 30 14:18:43.568549 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:43.566582 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de490cc2-45ff-4fe8-894f-46273f27e3d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-01955278-hjfp4\" (UID: \"de490cc2-45ff-4fe8-894f-46273f27e3d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-01955278-hjfp4" Oct 30 14:18:43.672260 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:43.670717 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de490cc2-45ff-4fe8-894f-46273f27e3d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-01955278-hjfp4\" (UID: \"de490cc2-45ff-4fe8-894f-46273f27e3d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-01955278-hjfp4" Oct 30 14:18:43.672260 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:43.671096 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfnt6\" (UniqueName: \"kubernetes.io/projected/de490cc2-45ff-4fe8-894f-46273f27e3d4-kube-api-access-hfnt6\") pod \"maas-api-revoke-keys-e2e-mt-01955278-hjfp4\" (UID: \"de490cc2-45ff-4fe8-894f-46273f27e3d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-01955278-hjfp4" Oct 30 14:18:43.672260 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:43.671161 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de490cc2-45ff-4fe8-894f-46273f27e3d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-01955278-hjfp4\" (UID: \"de490cc2-45ff-4fe8-894f-46273f27e3d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-01955278-hjfp4" Oct 30 14:18:43.688780 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:43.672716 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de490cc2-45ff-4fe8-894f-46273f27e3d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-01955278-hjfp4\" (UID: \"de490cc2-45ff-4fe8-894f-46273f27e3d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-01955278-hjfp4" Oct 30 14:18:43.716306 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:43.699088 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de490cc2-45ff-4fe8-894f-46273f27e3d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-01955278-hjfp4\" (UID: \"de490cc2-45ff-4fe8-894f-46273f27e3d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-01955278-hjfp4" Oct 30 14:18:43.738751 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:43.727542 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfnt6\" (UniqueName: \"kubernetes.io/projected/de490cc2-45ff-4fe8-894f-46273f27e3d4-kube-api-access-hfnt6\") pod \"maas-api-revoke-keys-e2e-mt-01955278-hjfp4\" (UID: \"de490cc2-45ff-4fe8-894f-46273f27e3d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-01955278-hjfp4" Oct 30 14:18:43.815752 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:43.815720 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:18:43.824699 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:43.824671 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-01955278-hjfp4" Oct 30 14:18:44.118695 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:44.118578 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:18:44.137512 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:18:44.137479 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde490cc2_45ff_4fe8_894f_46273f27e3d4.slice/crio-025c3ff65566271212475fce09d4a25d5c78fe75d6bbde4ef2851af3d130a987 WatchSource:0}: Error finding container 025c3ff65566271212475fce09d4a25d5c78fe75d6bbde4ef2851af3d130a987: Status 404 returned error can't find the container with id 025c3ff65566271212475fce09d4a25d5c78fe75d6bbde4ef2851af3d130a987 Oct 30 14:18:44.141589 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:44.141492 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-01955278-hjfp4"] Oct 30 14:18:44.941794 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:44.941757 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-01955278-hjfp4" event={"ID":"de490cc2-45ff-4fe8-894f-46273f27e3d4","Type":"ContainerStarted","Data":"28c94a2f3d776edd7687532a918b289a2bb2781ef0347f66683229b2745aa27b"} Oct 30 14:18:44.941965 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:44.941815 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-01955278-hjfp4" event={"ID":"de490cc2-45ff-4fe8-894f-46273f27e3d4","Type":"ContainerStarted","Data":"025c3ff65566271212475fce09d4a25d5c78fe75d6bbde4ef2851af3d130a987"} Oct 30 14:18:44.985703 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:44.985629 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-01955278-hjfp4" podStartSLOduration=1.9856120860000002 podStartE2EDuration="1.985612086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:18:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:18:44.984656379 +0000 UTC m=+1794.728790418" watchObservedRunningTime="2026-10-30 14:18:44.985612086 +0000 UTC m=+1794.729746129" Oct 30 14:18:45.949655 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:45.949599 4982 generic.go:356] "Generic (PLEG): container finished" podID="de490cc2-45ff-4fe8-894f-46273f27e3d4" containerID="28c94a2f3d776edd7687532a918b289a2bb2781ef0347f66683229b2745aa27b" exitCode=0 Oct 30 14:18:45.950029 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:45.949683 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-01955278-hjfp4" event={"ID":"de490cc2-45ff-4fe8-894f-46273f27e3d4","Type":"ContainerDied","Data":"28c94a2f3d776edd7687532a918b289a2bb2781ef0347f66683229b2745aa27b"} Oct 30 14:18:46.170217 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:46.170178 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:18:47.197201 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:47.197177 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-01955278-hjfp4" Oct 30 14:18:47.252598 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:47.252531 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de490cc2-45ff-4fe8-894f-46273f27e3d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de490cc2-45ff-4fe8-894f-46273f27e3d4-maas-api-openshift-ca\") pod \"de490cc2-45ff-4fe8-894f-46273f27e3d4\" (UID: \"de490cc2-45ff-4fe8-894f-46273f27e3d4\") " Oct 30 14:18:47.252598 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:47.252570 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de490cc2-45ff-4fe8-894f-46273f27e3d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de490cc2-45ff-4fe8-894f-46273f27e3d4-maas-api-xks-ca\") pod \"de490cc2-45ff-4fe8-894f-46273f27e3d4\" (UID: \"de490cc2-45ff-4fe8-894f-46273f27e3d4\") " Oct 30 14:18:47.252823 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:47.252603 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de490cc2-45ff-4fe8-894f-46273f27e3d4-kube-api-access-hfnt6\" (UniqueName: \"kubernetes.io/projected/de490cc2-45ff-4fe8-894f-46273f27e3d4-kube-api-access-hfnt6\") pod \"de490cc2-45ff-4fe8-894f-46273f27e3d4\" (UID: \"de490cc2-45ff-4fe8-894f-46273f27e3d4\") " Oct 30 14:18:47.252932 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:47.252908 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de490cc2-45ff-4fe8-894f-46273f27e3d4-maas-api-openshift-ca" pod "de490cc2-45ff-4fe8-894f-46273f27e3d4" (UID: "de490cc2-45ff-4fe8-894f-46273f27e3d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:18:47.253212 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:47.253192 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de490cc2-45ff-4fe8-894f-46273f27e3d4-maas-api-openshift-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:18:47.254661 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:47.254630 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de490cc2-45ff-4fe8-894f-46273f27e3d4-maas-api-xks-ca" pod "de490cc2-45ff-4fe8-894f-46273f27e3d4" (UID: "de490cc2-45ff-4fe8-894f-46273f27e3d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:18:47.254813 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:47.254792 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de490cc2-45ff-4fe8-894f-46273f27e3d4-kube-api-access-hfnt6" pod "de490cc2-45ff-4fe8-894f-46273f27e3d4" (UID: "de490cc2-45ff-4fe8-894f-46273f27e3d4"). InnerVolumeSpecName "kube-api-access-hfnt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:18:47.369660 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:47.361279 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de490cc2-45ff-4fe8-894f-46273f27e3d4-maas-api-xks-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:18:47.369660 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:47.361319 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfnt6\" (UniqueName: \"kubernetes.io/projected/de490cc2-45ff-4fe8-894f-46273f27e3d4-kube-api-access-hfnt6\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:18:47.699424 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:47.699397 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc" Oct 30 14:18:47.783821 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:47.782057 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ed4b142-fae5-4489-818c-8b53e3fea272-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ed4b142-fae5-4489-818c-8b53e3fea272-istiod-ca-cert\") pod \"1ed4b142-fae5-4489-818c-8b53e3fea272\" (UID: \"1ed4b142-fae5-4489-818c-8b53e3fea272\") " Oct 30 14:18:47.783821 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:47.782225 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ed4b142-fae5-4489-818c-8b53e3fea272-istio-token\" (UniqueName: \"kubernetes.io/projected/1ed4b142-fae5-4489-818c-8b53e3fea272-istio-token\") pod \"1ed4b142-fae5-4489-818c-8b53e3fea272\" (UID: \"1ed4b142-fae5-4489-818c-8b53e3fea272\") " Oct 30 14:18:47.783821 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:47.782282 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ed4b142-fae5-4489-818c-8b53e3fea272-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed4b142-fae5-4489-818c-8b53e3fea272-workload-socket\") pod \"1ed4b142-fae5-4489-818c-8b53e3fea272\" (UID: \"1ed4b142-fae5-4489-818c-8b53e3fea272\") " Oct 30 14:18:47.783821 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:47.782403 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ed4b142-fae5-4489-818c-8b53e3fea272-kube-api-access-pw6hf\" (UniqueName: \"kubernetes.io/projected/1ed4b142-fae5-4489-818c-8b53e3fea272-kube-api-access-pw6hf\") pod \"1ed4b142-fae5-4489-818c-8b53e3fea272\" (UID: \"1ed4b142-fae5-4489-818c-8b53e3fea272\") " Oct 30 14:18:47.783821 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:47.782466 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ed4b142-fae5-4489-818c-8b53e3fea272-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed4b142-fae5-4489-818c-8b53e3fea272-credential-socket\") pod \"1ed4b142-fae5-4489-818c-8b53e3fea272\" (UID: \"1ed4b142-fae5-4489-818c-8b53e3fea272\") " Oct 30 14:18:47.783821 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:47.782513 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ed4b142-fae5-4489-818c-8b53e3fea272-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ed4b142-fae5-4489-818c-8b53e3fea272-istio-envoy\") pod \"1ed4b142-fae5-4489-818c-8b53e3fea272\" (UID: \"1ed4b142-fae5-4489-818c-8b53e3fea272\") " Oct 30 14:18:47.783821 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:47.782545 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ed4b142-fae5-4489-818c-8b53e3fea272-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ed4b142-fae5-4489-818c-8b53e3fea272-istio-data\") pod \"1ed4b142-fae5-4489-818c-8b53e3fea272\" (UID: \"1ed4b142-fae5-4489-818c-8b53e3fea272\") " Oct 30 14:18:47.783821 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:47.782621 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ed4b142-fae5-4489-818c-8b53e3fea272-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ed4b142-fae5-4489-818c-8b53e3fea272-workload-certs\") pod \"1ed4b142-fae5-4489-818c-8b53e3fea272\" (UID: \"1ed4b142-fae5-4489-818c-8b53e3fea272\") " Oct 30 14:18:47.783821 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:47.782685 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ed4b142-fae5-4489-818c-8b53e3fea272-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ed4b142-fae5-4489-818c-8b53e3fea272-istio-podinfo\") pod \"1ed4b142-fae5-4489-818c-8b53e3fea272\" (UID: \"1ed4b142-fae5-4489-818c-8b53e3fea272\") " Oct 30 14:18:47.786777 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:47.786098 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ed4b142-fae5-4489-818c-8b53e3fea272-credential-socket" pod "1ed4b142-fae5-4489-818c-8b53e3fea272" (UID: "1ed4b142-fae5-4489-818c-8b53e3fea272"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:18:47.786777 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:47.786327 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ed4b142-fae5-4489-818c-8b53e3fea272-workload-certs" pod "1ed4b142-fae5-4489-818c-8b53e3fea272" (UID: "1ed4b142-fae5-4489-818c-8b53e3fea272"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:18:47.786777 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:47.786518 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ed4b142-fae5-4489-818c-8b53e3fea272-workload-socket" pod "1ed4b142-fae5-4489-818c-8b53e3fea272" (UID: "1ed4b142-fae5-4489-818c-8b53e3fea272"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:18:47.786777 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:47.786744 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ed4b142-fae5-4489-818c-8b53e3fea272-istiod-ca-cert" pod "1ed4b142-fae5-4489-818c-8b53e3fea272" (UID: "1ed4b142-fae5-4489-818c-8b53e3fea272"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:18:47.787319 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:47.787285 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ed4b142-fae5-4489-818c-8b53e3fea272-istio-data" pod "1ed4b142-fae5-4489-818c-8b53e3fea272" (UID: "1ed4b142-fae5-4489-818c-8b53e3fea272"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:18:47.798360 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:47.798316 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ed4b142-fae5-4489-818c-8b53e3fea272-istio-envoy" pod "1ed4b142-fae5-4489-818c-8b53e3fea272" (UID: "1ed4b142-fae5-4489-818c-8b53e3fea272"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:18:47.798889 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:47.798859 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ed4b142-fae5-4489-818c-8b53e3fea272-istio-token" pod "1ed4b142-fae5-4489-818c-8b53e3fea272" (UID: "1ed4b142-fae5-4489-818c-8b53e3fea272"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:18:47.799379 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:47.799345 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ed4b142-fae5-4489-818c-8b53e3fea272-istio-podinfo" pod "1ed4b142-fae5-4489-818c-8b53e3fea272" (UID: "1ed4b142-fae5-4489-818c-8b53e3fea272"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:18:47.799499 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:47.799472 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ed4b142-fae5-4489-818c-8b53e3fea272-kube-api-access-pw6hf" pod "1ed4b142-fae5-4489-818c-8b53e3fea272" (UID: "1ed4b142-fae5-4489-818c-8b53e3fea272"). InnerVolumeSpecName "kube-api-access-pw6hf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:18:47.886321 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:47.886277 4982 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ed4b142-fae5-4489-818c-8b53e3fea272-istio-podinfo\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:18:47.886321 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:47.886321 4982 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ed4b142-fae5-4489-818c-8b53e3fea272-istiod-ca-cert\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:18:47.886612 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:47.886337 4982 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ed4b142-fae5-4489-818c-8b53e3fea272-istio-token\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:18:47.886612 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:47.886351 4982 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed4b142-fae5-4489-818c-8b53e3fea272-workload-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:18:47.886612 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:47.886363 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pw6hf\" (UniqueName: \"kubernetes.io/projected/1ed4b142-fae5-4489-818c-8b53e3fea272-kube-api-access-pw6hf\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:18:47.886612 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:47.886375 4982 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed4b142-fae5-4489-818c-8b53e3fea272-credential-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:18:47.886612 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:47.886389 4982 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ed4b142-fae5-4489-818c-8b53e3fea272-istio-envoy\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:18:47.886612 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:47.886400 4982 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ed4b142-fae5-4489-818c-8b53e3fea272-istio-data\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:18:47.886612 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:47.886418 4982 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ed4b142-fae5-4489-818c-8b53e3fea272-workload-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:18:47.971209 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:47.971180 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-01955278-hjfp4" Oct 30 14:18:47.971980 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:47.971182 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-01955278-hjfp4" event={"ID":"de490cc2-45ff-4fe8-894f-46273f27e3d4","Type":"ContainerDied","Data":"025c3ff65566271212475fce09d4a25d5c78fe75d6bbde4ef2851af3d130a987"} Oct 30 14:18:47.972177 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:47.972163 4982 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="025c3ff65566271212475fce09d4a25d5c78fe75d6bbde4ef2851af3d130a987" Oct 30 14:18:47.973423 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:47.973396 4982 generic.go:356] "Generic (PLEG): container finished" podID="1ed4b142-fae5-4489-818c-8b53e3fea272" containerID="7bae0c3d4a401a4cb6b00e460fbb02f8e318ccac576fe114d38cd7f58d21a233" exitCode=0 Oct 30 14:18:47.973537 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:47.973493 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc" event={"ID":"1ed4b142-fae5-4489-818c-8b53e3fea272","Type":"ContainerDied","Data":"7bae0c3d4a401a4cb6b00e460fbb02f8e318ccac576fe114d38cd7f58d21a233"} Oct 30 14:18:47.973599 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:47.973534 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc" event={"ID":"1ed4b142-fae5-4489-818c-8b53e3fea272","Type":"ContainerDied","Data":"9bc8f4cbc4ff6e507bfa49314a66d05b9d46a7034600604662ea9bbb0c84d5bf"} Oct 30 14:18:47.973599 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:47.973565 4982 scope.go:122] "RemoveContainer" containerID="7bae0c3d4a401a4cb6b00e460fbb02f8e318ccac576fe114d38cd7f58d21a233" Oct 30 14:18:47.973955 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:47.973760 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc" Oct 30 14:18:47.988618 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:47.988597 4982 scope.go:122] "RemoveContainer" containerID="7bae0c3d4a401a4cb6b00e460fbb02f8e318ccac576fe114d38cd7f58d21a233" Oct 30 14:18:47.992713 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:18:47.989404 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bae0c3d4a401a4cb6b00e460fbb02f8e318ccac576fe114d38cd7f58d21a233\": container with ID starting with 7bae0c3d4a401a4cb6b00e460fbb02f8e318ccac576fe114d38cd7f58d21a233 not found: ID does not exist" containerID="7bae0c3d4a401a4cb6b00e460fbb02f8e318ccac576fe114d38cd7f58d21a233" Oct 30 14:18:47.992713 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:47.989441 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bae0c3d4a401a4cb6b00e460fbb02f8e318ccac576fe114d38cd7f58d21a233"} err="failed to get container status \"7bae0c3d4a401a4cb6b00e460fbb02f8e318ccac576fe114d38cd7f58d21a233\": rpc error: code = NotFound desc = could not find container \"7bae0c3d4a401a4cb6b00e460fbb02f8e318ccac576fe114d38cd7f58d21a233\": container with ID starting with 7bae0c3d4a401a4cb6b00e460fbb02f8e318ccac576fe114d38cd7f58d21a233 not found: ID does not exist" Oct 30 14:18:48.043922 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:48.043838 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc"] Oct 30 14:18:48.049081 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:48.049051 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-d67bc"] Oct 30 14:18:48.499464 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:48.499429 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ed4b142-fae5-4489-818c-8b53e3fea272" path="/var/lib/kubelet/pods/1ed4b142-fae5-4489-818c-8b53e3fea272/volumes" Oct 30 14:18:48.809609 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:48.809574 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:18:50.633851 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:50.633820 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ntlqq_dc967875-1546-40bb-8376-fde09416c204/console-operator/2.log" Oct 30 14:18:50.636610 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:50.636571 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ntlqq_dc967875-1546-40bb-8376-fde09416c204/console-operator/2.log" Oct 30 14:18:50.639962 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:50.639941 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhxjf_aca47fe8-7068-4743-b6ab-283ee0874f66/ovn-acl-logging/0.log" Oct 30 14:18:50.642564 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:50.642539 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhxjf_aca47fe8-7068-4743-b6ab-283ee0874f66/ovn-acl-logging/0.log" Oct 30 14:18:50.845174 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:50.845137 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:18:52.694169 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:52.694129 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:18:54.185364 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:54.185330 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:18:54.353470 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:54.353438 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:18:54.667342 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:54.667305 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:18:54.846664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:54.838508 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:18:56.813537 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:56.813501 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:18:58.018451 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:18:58.018414 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:18:58.673170 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:18:58.673128 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:00.720589 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:00.720550 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:02.739809 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:02.739772 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:04.744653 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:04.744547 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:06.616448 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:06.616407 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:08.455020 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:08.454984 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:08.606847 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:08.606810 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:09.279199 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:09.279169 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:09.838147 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:09.838115 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:10.004073 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:10.004034 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:10.514595 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:10.510406 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:12.336013 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:12.335982 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:14.315461 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:14.315400 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:14.456397 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:14.456362 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:14.746372 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:14.746278 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:16.582277 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:16.582238 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:17.255530 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:17.255478 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:17.902087 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:17.902041 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:18.472253 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:18.472215 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:18.633204 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:18.633167 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:20.421673 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:20.416111 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:20.576776 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:20.576693 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:20.744933 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:20.744850 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:21.941577 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:21.941542 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:22.216308 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:22.216199 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:22.514213 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:22.514020 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:23.015591 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:23.015547 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:24.166097 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:24.166054 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:24.485548 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:24.484935 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:25.765300 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:25.765260 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:26.245938 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:26.245897 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:26.482041 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:26.482007 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:27.714256 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:27.714189 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-01955278-7cfd4f7cbd-sj77j"] Oct 30 14:19:27.715096 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:27.715061 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-01955278-7cfd4f7cbd-sj77j" podUID="708748da-ecb9-4028-bb6e-c4ed4b889921" containerName="maas-api" containerID="cri-o://95e9079256043fc8e4a7d94ad5241e9ead21b6c8c9f3d7b2cdde024352704931" gracePeriod=30 Oct 30 14:19:27.811270 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:27.811235 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:28.206812 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:19:28.206599 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:19:28.252991 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:28.252964 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-01955278-7cfd4f7cbd-sj77j" Oct 30 14:19:28.371678 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:28.371128 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/708748da-ecb9-4028-bb6e-c4ed4b889921-kube-api-access-hsvg9\" (UniqueName: \"kubernetes.io/projected/708748da-ecb9-4028-bb6e-c4ed4b889921-kube-api-access-hsvg9\") pod \"708748da-ecb9-4028-bb6e-c4ed4b889921\" (UID: \"708748da-ecb9-4028-bb6e-c4ed4b889921\") " Oct 30 14:19:28.371678 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:28.371223 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/708748da-ecb9-4028-bb6e-c4ed4b889921-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/708748da-ecb9-4028-bb6e-c4ed4b889921-maas-api-tls\") pod \"708748da-ecb9-4028-bb6e-c4ed4b889921\" (UID: \"708748da-ecb9-4028-bb6e-c4ed4b889921\") " Oct 30 14:19:28.371678 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:28.371432 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/708748da-ecb9-4028-bb6e-c4ed4b889921-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/708748da-ecb9-4028-bb6e-c4ed4b889921-openshift-service-ca\") pod \"708748da-ecb9-4028-bb6e-c4ed4b889921\" (UID: \"708748da-ecb9-4028-bb6e-c4ed4b889921\") " Oct 30 14:19:28.371678 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:28.371633 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/708748da-ecb9-4028-bb6e-c4ed4b889921-metrics-certs\" (UniqueName: \"kubernetes.io/secret/708748da-ecb9-4028-bb6e-c4ed4b889921-metrics-certs\") pod \"708748da-ecb9-4028-bb6e-c4ed4b889921\" (UID: \"708748da-ecb9-4028-bb6e-c4ed4b889921\") " Oct 30 14:19:28.394692 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:28.385039 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/708748da-ecb9-4028-bb6e-c4ed4b889921-openshift-service-ca" pod "708748da-ecb9-4028-bb6e-c4ed4b889921" (UID: "708748da-ecb9-4028-bb6e-c4ed4b889921"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:19:28.394692 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:28.389174 4982 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/708748da-ecb9-4028-bb6e-c4ed4b889921-openshift-service-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:19:28.422668 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:28.419811 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/708748da-ecb9-4028-bb6e-c4ed4b889921-metrics-certs" pod "708748da-ecb9-4028-bb6e-c4ed4b889921" (UID: "708748da-ecb9-4028-bb6e-c4ed4b889921"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:19:28.422668 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:28.420731 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/708748da-ecb9-4028-bb6e-c4ed4b889921-kube-api-access-hsvg9" pod "708748da-ecb9-4028-bb6e-c4ed4b889921" (UID: "708748da-ecb9-4028-bb6e-c4ed4b889921"). InnerVolumeSpecName "kube-api-access-hsvg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:19:28.436691 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:28.426400 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/708748da-ecb9-4028-bb6e-c4ed4b889921-maas-api-tls" pod "708748da-ecb9-4028-bb6e-c4ed4b889921" (UID: "708748da-ecb9-4028-bb6e-c4ed4b889921"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:19:28.499721 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:28.493131 4982 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/708748da-ecb9-4028-bb6e-c4ed4b889921-metrics-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:19:28.499721 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:28.493174 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsvg9\" (UniqueName: \"kubernetes.io/projected/708748da-ecb9-4028-bb6e-c4ed4b889921-kube-api-access-hsvg9\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:19:28.499721 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:28.493193 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/708748da-ecb9-4028-bb6e-c4ed4b889921-maas-api-tls\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:19:28.499721 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:28.497659 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:28.688752 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:28.688712 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-01955278-hjfp4"] Oct 30 14:19:28.697602 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:28.697562 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-01955278-7cfd4f7cbd-sj77j" Oct 30 14:19:28.711374 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:28.704191 4982 generic.go:356] "Generic (PLEG): container finished" podID="708748da-ecb9-4028-bb6e-c4ed4b889921" containerID="95e9079256043fc8e4a7d94ad5241e9ead21b6c8c9f3d7b2cdde024352704931" exitCode=0 Oct 30 14:19:28.711374 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:28.704466 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-01955278-7cfd4f7cbd-sj77j" event={"ID":"708748da-ecb9-4028-bb6e-c4ed4b889921","Type":"ContainerDied","Data":"95e9079256043fc8e4a7d94ad5241e9ead21b6c8c9f3d7b2cdde024352704931"} Oct 30 14:19:28.711374 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:28.704530 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-01955278-7cfd4f7cbd-sj77j" event={"ID":"708748da-ecb9-4028-bb6e-c4ed4b889921","Type":"ContainerDied","Data":"4d6171ca7dd2d82c5f54b0698a6b4c4ac4fee55b07f9809a8e00339d3b1a0aa1"} Oct 30 14:19:28.711374 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:28.704560 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-01955278-hjfp4"] Oct 30 14:19:28.711374 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:28.704595 4982 scope.go:122] "RemoveContainer" containerID="95e9079256043fc8e4a7d94ad5241e9ead21b6c8c9f3d7b2cdde024352704931" Oct 30 14:19:28.739596 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:28.739564 4982 scope.go:122] "RemoveContainer" containerID="95e9079256043fc8e4a7d94ad5241e9ead21b6c8c9f3d7b2cdde024352704931" Oct 30 14:19:28.740003 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:19:28.739969 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95e9079256043fc8e4a7d94ad5241e9ead21b6c8c9f3d7b2cdde024352704931\": container with ID starting with 95e9079256043fc8e4a7d94ad5241e9ead21b6c8c9f3d7b2cdde024352704931 not found: ID does not exist" containerID="95e9079256043fc8e4a7d94ad5241e9ead21b6c8c9f3d7b2cdde024352704931" Oct 30 14:19:28.740049 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:28.740018 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95e9079256043fc8e4a7d94ad5241e9ead21b6c8c9f3d7b2cdde024352704931"} err="failed to get container status \"95e9079256043fc8e4a7d94ad5241e9ead21b6c8c9f3d7b2cdde024352704931\": rpc error: code = NotFound desc = could not find container \"95e9079256043fc8e4a7d94ad5241e9ead21b6c8c9f3d7b2cdde024352704931\": container with ID starting with 95e9079256043fc8e4a7d94ad5241e9ead21b6c8c9f3d7b2cdde024352704931 not found: ID does not exist" Oct 30 14:19:28.762583 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:28.762526 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:28.784385 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:28.784361 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-01955278-7cfd4f7cbd-sj77j"] Oct 30 14:19:28.791228 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:28.791202 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-01955278-7cfd4f7cbd-sj77j"] Oct 30 14:19:28.975942 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:28.975908 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:29.187108 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:29.187079 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:30.410436 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:30.410400 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="708748da-ecb9-4028-bb6e-c4ed4b889921" path="/var/lib/kubelet/pods/708748da-ecb9-4028-bb6e-c4ed4b889921/volumes" Oct 30 14:19:30.411040 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:30.411019 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de490cc2-45ff-4fe8-894f-46273f27e3d4" path="/var/lib/kubelet/pods/de490cc2-45ff-4fe8-894f-46273f27e3d4/volumes" Oct 30 14:19:30.736150 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:30.736065 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:32.841320 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:32.841284 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:34.895753 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:34.895715 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:36.377169 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:36.377123 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:36.539269 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:36.539225 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:36.748298 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:36.748203 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:36.919140 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:36.916477 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:38.677746 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:38.677683 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:38.829957 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:38.829898 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:40.727595 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:40.727563 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:42.751403 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:42.751369 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:43.936457 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:43.936427 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:44.830913 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:44.830849 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:45.868189 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:45.868149 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:46.392447 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:46.392408 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:46.552208 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:46.552175 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:47.042088 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:47.042049 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:47.577603 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:47.577567 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:48.045508 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:48.045436 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:48.269361 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:48.269325 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:49.134850 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:49.134814 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:50.906393 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:50.906356 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:51.118983 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:51.118944 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:51.350970 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:51.350935 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:52.723386 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:52.723349 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:52.854621 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:52.854583 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5a9d7b-9bb744df9-2qrjh"] Oct 30 14:19:52.855561 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:52.855536 4982 cpu_manager.go:414] "containerMap: removing container" podUID="de490cc2-45ff-4fe8-894f-46273f27e3d4" containerName="revoke-keys" Oct 30 14:19:52.855630 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:52.855564 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de490cc2-45ff-4fe8-894f-46273f27e3d4" containerName="revoke-keys" Oct 30 14:19:52.855691 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:52.855633 4982 cpu_manager.go:414] "containerMap: removing container" podUID="1ed4b142-fae5-4489-818c-8b53e3fea272" containerName="istio-proxy" Oct 30 14:19:52.855691 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:52.855658 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ed4b142-fae5-4489-818c-8b53e3fea272" containerName="istio-proxy" Oct 30 14:19:52.855691 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:52.855675 4982 cpu_manager.go:414] "containerMap: removing container" podUID="708748da-ecb9-4028-bb6e-c4ed4b889921" containerName="maas-api" Oct 30 14:19:52.855691 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:52.855683 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="708748da-ecb9-4028-bb6e-c4ed4b889921" containerName="maas-api" Oct 30 14:19:52.855859 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:52.855845 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="de490cc2-45ff-4fe8-894f-46273f27e3d4" containerName="revoke-keys" Oct 30 14:19:52.855898 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:52.855876 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ed4b142-fae5-4489-818c-8b53e3fea272" containerName="istio-proxy" Oct 30 14:19:52.855898 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:52.855885 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="708748da-ecb9-4028-bb6e-c4ed4b889921" containerName="maas-api" Oct 30 14:19:52.861385 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:52.861361 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5a9d7b-9bb744df9-2qrjh" Oct 30 14:19:52.866265 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:52.866228 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:52.878600 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:52.878566 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5a9d7b-9bb744df9-2qrjh"] Oct 30 14:19:52.946701 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:52.940338 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-5a9d7b\"" Oct 30 14:19:52.948600 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:52.947119 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-5a9d7b-dockercfg-9hrh8\"" Oct 30 14:19:52.949957 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:52.949904 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw87g\" (UniqueName: \"kubernetes.io/projected/241e6f6a-6c36-4487-b3ba-92941c524787-kube-api-access-rw87g\") pod \"payload-pre-processing-e2e-inf-a-5a9d7b-9bb744df9-2qrjh\" (UID: \"241e6f6a-6c36-4487-b3ba-92941c524787\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5a9d7b-9bb744df9-2qrjh" Oct 30 14:19:52.950132 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:52.950112 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/241e6f6a-6c36-4487-b3ba-92941c524787-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-5a9d7b-9bb744df9-2qrjh\" (UID: \"241e6f6a-6c36-4487-b3ba-92941c524787\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5a9d7b-9bb744df9-2qrjh" Oct 30 14:19:53.052730 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:53.051706 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw87g\" (UniqueName: \"kubernetes.io/projected/241e6f6a-6c36-4487-b3ba-92941c524787-kube-api-access-rw87g\") pod \"payload-pre-processing-e2e-inf-a-5a9d7b-9bb744df9-2qrjh\" (UID: \"241e6f6a-6c36-4487-b3ba-92941c524787\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5a9d7b-9bb744df9-2qrjh" Oct 30 14:19:53.052730 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:53.051761 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/241e6f6a-6c36-4487-b3ba-92941c524787-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-5a9d7b-9bb744df9-2qrjh\" (UID: \"241e6f6a-6c36-4487-b3ba-92941c524787\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5a9d7b-9bb744df9-2qrjh" Oct 30 14:19:53.060579 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:53.053019 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/241e6f6a-6c36-4487-b3ba-92941c524787-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-5a9d7b-9bb744df9-2qrjh\" (UID: \"241e6f6a-6c36-4487-b3ba-92941c524787\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5a9d7b-9bb744df9-2qrjh" Oct 30 14:19:53.238570 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:53.238532 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:53.256347 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:53.256316 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw87g\" (UniqueName: \"kubernetes.io/projected/241e6f6a-6c36-4487-b3ba-92941c524787-kube-api-access-rw87g\") pod \"payload-pre-processing-e2e-inf-a-5a9d7b-9bb744df9-2qrjh\" (UID: \"241e6f6a-6c36-4487-b3ba-92941c524787\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5a9d7b-9bb744df9-2qrjh" Oct 30 14:19:53.432227 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:53.432187 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:53.476141 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:53.476118 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5a9d7b-9bb744df9-2qrjh" Oct 30 14:19:53.647859 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:53.647818 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:53.789999 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:53.789157 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5a9d7b-9bb744df9-2qrjh"] Oct 30 14:19:53.954178 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:53.954149 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5a9d7b-9bb744df9-2qrjh" event={"ID":"241e6f6a-6c36-4487-b3ba-92941c524787","Type":"ContainerStarted","Data":"c0a244784c6e81328d58cd41e4ec2cddda2d79e0758be3a4cee0363107e3d249"} Oct 30 14:19:53.954316 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:53.954207 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5a9d7b-9bb744df9-2qrjh" event={"ID":"241e6f6a-6c36-4487-b3ba-92941c524787","Type":"ContainerStarted","Data":"e7fced250406957ab624861619257af1bd4e64ae17058484a26f5a76ca632269"} Oct 30 14:19:53.954316 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:53.954238 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5a9d7b-9bb744df9-2qrjh" Oct 30 14:19:54.006152 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:54.006092 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5a9d7b-9bb744df9-2qrjh" podStartSLOduration=2.006074854 podStartE2EDuration="2.006074854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:19:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:19:54.000543451 +0000 UTC m=+1863.744677510" watchObservedRunningTime="2026-10-30 14:19:54.006074854 +0000 UTC m=+1863.750208895" Oct 30 14:19:54.815521 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:54.815471 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:54.899052 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:54.899017 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4bf33a-6bb59459b-4qdp6"] Oct 30 14:19:54.902952 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:54.902930 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4bf33a-6bb59459b-4qdp6" Oct 30 14:19:54.927618 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:54.927577 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4bf33a-6bb59459b-4qdp6"] Oct 30 14:19:54.934221 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:54.934187 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-4bf33a\"" Oct 30 14:19:54.963689 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:54.962101 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:54.974902 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:54.974868 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4bf33a-7f5fb57b55-9mjfq"] Oct 30 14:19:54.981577 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:54.981553 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4bf33a-7f5fb57b55-9mjfq" Oct 30 14:19:55.001881 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:55.001846 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsn7t\" (UniqueName: \"kubernetes.io/projected/d34de819-45d5-4dd0-a4a2-b9f742983fad-kube-api-access-fsn7t\") pod \"payload-processing-e2e-api-a-4bf33a-6bb59459b-4qdp6\" (UID: \"d34de819-45d5-4dd0-a4a2-b9f742983fad\") " pod="openshift-ingress/payload-processing-e2e-api-a-4bf33a-6bb59459b-4qdp6" Oct 30 14:19:55.002337 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:55.002309 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wwfh\" (UniqueName: \"kubernetes.io/projected/65c30204-f1f4-4017-9ecf-71bbe81dbff6-kube-api-access-5wwfh\") pod \"payload-pre-processing-e2e-api-a-4bf33a-7f5fb57b55-9mjfq\" (UID: \"65c30204-f1f4-4017-9ecf-71bbe81dbff6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4bf33a-7f5fb57b55-9mjfq" Oct 30 14:19:55.012111 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:55.002407 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d34de819-45d5-4dd0-a4a2-b9f742983fad-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4bf33a-6bb59459b-4qdp6\" (UID: \"d34de819-45d5-4dd0-a4a2-b9f742983fad\") " pod="openshift-ingress/payload-processing-e2e-api-a-4bf33a-6bb59459b-4qdp6" Oct 30 14:19:55.012111 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:55.002513 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65c30204-f1f4-4017-9ecf-71bbe81dbff6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-4bf33a-7f5fb57b55-9mjfq\" (UID: \"65c30204-f1f4-4017-9ecf-71bbe81dbff6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4bf33a-7f5fb57b55-9mjfq" Oct 30 14:19:55.012111 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:55.002940 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-4bf33a-dockercfg-p2cf8\"" Oct 30 14:19:55.012111 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:55.009688 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4bf33a-7f5fb57b55-9mjfq"] Oct 30 14:19:55.105649 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:55.105276 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsn7t\" (UniqueName: \"kubernetes.io/projected/d34de819-45d5-4dd0-a4a2-b9f742983fad-kube-api-access-fsn7t\") pod \"payload-processing-e2e-api-a-4bf33a-6bb59459b-4qdp6\" (UID: \"d34de819-45d5-4dd0-a4a2-b9f742983fad\") " pod="openshift-ingress/payload-processing-e2e-api-a-4bf33a-6bb59459b-4qdp6" Oct 30 14:19:55.105649 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:55.105346 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wwfh\" (UniqueName: \"kubernetes.io/projected/65c30204-f1f4-4017-9ecf-71bbe81dbff6-kube-api-access-5wwfh\") pod \"payload-pre-processing-e2e-api-a-4bf33a-7f5fb57b55-9mjfq\" (UID: \"65c30204-f1f4-4017-9ecf-71bbe81dbff6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4bf33a-7f5fb57b55-9mjfq" Oct 30 14:19:55.105649 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:55.105428 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d34de819-45d5-4dd0-a4a2-b9f742983fad-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4bf33a-6bb59459b-4qdp6\" (UID: \"d34de819-45d5-4dd0-a4a2-b9f742983fad\") " pod="openshift-ingress/payload-processing-e2e-api-a-4bf33a-6bb59459b-4qdp6" Oct 30 14:19:55.105649 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:55.105526 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65c30204-f1f4-4017-9ecf-71bbe81dbff6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-4bf33a-7f5fb57b55-9mjfq\" (UID: \"65c30204-f1f4-4017-9ecf-71bbe81dbff6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4bf33a-7f5fb57b55-9mjfq" Oct 30 14:19:55.123103 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:55.120279 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d34de819-45d5-4dd0-a4a2-b9f742983fad-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4bf33a-6bb59459b-4qdp6\" (UID: \"d34de819-45d5-4dd0-a4a2-b9f742983fad\") " pod="openshift-ingress/payload-processing-e2e-api-a-4bf33a-6bb59459b-4qdp6" Oct 30 14:19:55.144663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:55.141500 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65c30204-f1f4-4017-9ecf-71bbe81dbff6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-4bf33a-7f5fb57b55-9mjfq\" (UID: \"65c30204-f1f4-4017-9ecf-71bbe81dbff6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4bf33a-7f5fb57b55-9mjfq" Oct 30 14:19:55.276664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:55.262473 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsn7t\" (UniqueName: \"kubernetes.io/projected/d34de819-45d5-4dd0-a4a2-b9f742983fad-kube-api-access-fsn7t\") pod \"payload-processing-e2e-api-a-4bf33a-6bb59459b-4qdp6\" (UID: \"d34de819-45d5-4dd0-a4a2-b9f742983fad\") " pod="openshift-ingress/payload-processing-e2e-api-a-4bf33a-6bb59459b-4qdp6" Oct 30 14:19:55.285661 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:55.284333 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wwfh\" (UniqueName: \"kubernetes.io/projected/65c30204-f1f4-4017-9ecf-71bbe81dbff6-kube-api-access-5wwfh\") pod \"payload-pre-processing-e2e-api-a-4bf33a-7f5fb57b55-9mjfq\" (UID: \"65c30204-f1f4-4017-9ecf-71bbe81dbff6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4bf33a-7f5fb57b55-9mjfq" Oct 30 14:19:55.330170 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:55.329971 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4bf33a-7f5fb57b55-9mjfq" Oct 30 14:19:55.517823 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:55.517782 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4bf33a-6bb59459b-4qdp6" Oct 30 14:19:55.655726 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:55.655625 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4bf33a-7f5fb57b55-9mjfq"] Oct 30 14:19:55.658060 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:19:55.657956 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65c30204_f1f4_4017_9ecf_71bbe81dbff6.slice/crio-c32bc904e8b0d320e1b0c51ae219106b9d5ba09d686c7c44747afbcea955fca2 WatchSource:0}: Error finding container c32bc904e8b0d320e1b0c51ae219106b9d5ba09d686c7c44747afbcea955fca2: Status 404 returned error can't find the container with id c32bc904e8b0d320e1b0c51ae219106b9d5ba09d686c7c44747afbcea955fca2 Oct 30 14:19:55.807232 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:55.807196 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:55.848237 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:19:55.848204 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd34de819_45d5_4dd0_a4a2_b9f742983fad.slice/crio-d81943e6365ebba4223791c23dca147e01173b9c8eb3fcf00b9e990fdf0099c1 WatchSource:0}: Error finding container d81943e6365ebba4223791c23dca147e01173b9c8eb3fcf00b9e990fdf0099c1: Status 404 returned error can't find the container with id d81943e6365ebba4223791c23dca147e01173b9c8eb3fcf00b9e990fdf0099c1 Oct 30 14:19:55.851313 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:55.851282 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4bf33a-6bb59459b-4qdp6"] Oct 30 14:19:55.987582 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:55.987549 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4bf33a-7f5fb57b55-9mjfq" event={"ID":"65c30204-f1f4-4017-9ecf-71bbe81dbff6","Type":"ContainerStarted","Data":"72038def3f5a26a62400c2ba2bc6c708975393f7f4097cb75b46902e0cd7daa8"} Oct 30 14:19:55.987758 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:55.987619 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4bf33a-7f5fb57b55-9mjfq" Oct 30 14:19:55.987758 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:55.987667 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4bf33a-7f5fb57b55-9mjfq" event={"ID":"65c30204-f1f4-4017-9ecf-71bbe81dbff6","Type":"ContainerStarted","Data":"c32bc904e8b0d320e1b0c51ae219106b9d5ba09d686c7c44747afbcea955fca2"} Oct 30 14:19:55.989552 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:55.989519 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4bf33a-6bb59459b-4qdp6" event={"ID":"d34de819-45d5-4dd0-a4a2-b9f742983fad","Type":"ContainerStarted","Data":"a0625d95c03484145dd21b7791009b469d9ad144007860859c7ddbb5a6a545e6"} Oct 30 14:19:55.989702 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:55.989600 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4bf33a-6bb59459b-4qdp6" event={"ID":"d34de819-45d5-4dd0-a4a2-b9f742983fad","Type":"ContainerStarted","Data":"d81943e6365ebba4223791c23dca147e01173b9c8eb3fcf00b9e990fdf0099c1"} Oct 30 14:19:55.990341 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:55.990320 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-4bf33a-6bb59459b-4qdp6" Oct 30 14:19:56.033914 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:56.033864 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4bf33a-7f5fb57b55-9mjfq" podStartSLOduration=2.033848748 podStartE2EDuration="2.033848748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:19:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:19:56.032491 +0000 UTC m=+1865.776625043" watchObservedRunningTime="2026-10-30 14:19:56.033848748 +0000 UTC m=+1865.777982791" Oct 30 14:19:56.078064 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:56.078003 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-4bf33a-6bb59459b-4qdp6" podStartSLOduration=2.077988971 podStartE2EDuration="2.077988971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:19:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:19:56.074996773 +0000 UTC m=+1865.819130811" watchObservedRunningTime="2026-10-30 14:19:56.077988971 +0000 UTC m=+1865.822123010" Oct 30 14:19:56.713064 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:56.713029 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8"] Oct 30 14:19:56.713323 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:56.713262 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8" podUID="00eefe61-57f5-4d1c-b9a5-8dd54db2922e" containerName="istio-proxy" containerID="cri-o://797831e639462afe18671a43b4d40589e0afcb8b3fe4208cb8d075134fa2a194" gracePeriod=30 Oct 30 14:19:57.874250 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:57.874206 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:19:58.019286 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:19:58.019249 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:19:58.566373 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:58.566339 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9dba4af9-c7zvt"] Oct 30 14:19:58.570480 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:58.570463 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9dba4af9-c7zvt" Oct 30 14:19:58.610660 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:58.606135 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:19:58.610660 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:58.606177 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:19:58.634945 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:58.634902 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9dba4af9-c7zvt"] Oct 30 14:19:58.686486 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:58.686451 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84cee745-3ab3-4075-a9b8-9ab78e6c0d01-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-9dba4af9-c7zvt\" (UID: \"84cee745-3ab3-4075-a9b8-9ab78e6c0d01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9dba4af9-c7zvt" Oct 30 14:19:58.686486 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:58.686493 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89fqn\" (UniqueName: \"kubernetes.io/projected/84cee745-3ab3-4075-a9b8-9ab78e6c0d01-kube-api-access-89fqn\") pod \"maas-api-revoke-keys-e2e-mt-9dba4af9-c7zvt\" (UID: \"84cee745-3ab3-4075-a9b8-9ab78e6c0d01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9dba4af9-c7zvt" Oct 30 14:19:58.686716 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:58.686697 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84cee745-3ab3-4075-a9b8-9ab78e6c0d01-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-9dba4af9-c7zvt\" (UID: \"84cee745-3ab3-4075-a9b8-9ab78e6c0d01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9dba4af9-c7zvt" Oct 30 14:19:58.788053 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:58.788017 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84cee745-3ab3-4075-a9b8-9ab78e6c0d01-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-9dba4af9-c7zvt\" (UID: \"84cee745-3ab3-4075-a9b8-9ab78e6c0d01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9dba4af9-c7zvt" Oct 30 14:19:58.788053 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:58.788061 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89fqn\" (UniqueName: \"kubernetes.io/projected/84cee745-3ab3-4075-a9b8-9ab78e6c0d01-kube-api-access-89fqn\") pod \"maas-api-revoke-keys-e2e-mt-9dba4af9-c7zvt\" (UID: \"84cee745-3ab3-4075-a9b8-9ab78e6c0d01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9dba4af9-c7zvt" Oct 30 14:19:58.788264 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:58.788198 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84cee745-3ab3-4075-a9b8-9ab78e6c0d01-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-9dba4af9-c7zvt\" (UID: \"84cee745-3ab3-4075-a9b8-9ab78e6c0d01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9dba4af9-c7zvt" Oct 30 14:19:58.788930 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:58.788893 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84cee745-3ab3-4075-a9b8-9ab78e6c0d01-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-9dba4af9-c7zvt\" (UID: \"84cee745-3ab3-4075-a9b8-9ab78e6c0d01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9dba4af9-c7zvt" Oct 30 14:19:58.790667 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:58.790606 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84cee745-3ab3-4075-a9b8-9ab78e6c0d01-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-9dba4af9-c7zvt\" (UID: \"84cee745-3ab3-4075-a9b8-9ab78e6c0d01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9dba4af9-c7zvt" Oct 30 14:19:58.814621 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:58.814577 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89fqn\" (UniqueName: \"kubernetes.io/projected/84cee745-3ab3-4075-a9b8-9ab78e6c0d01-kube-api-access-89fqn\") pod \"maas-api-revoke-keys-e2e-mt-9dba4af9-c7zvt\" (UID: \"84cee745-3ab3-4075-a9b8-9ab78e6c0d01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9dba4af9-c7zvt" Oct 30 14:19:58.885767 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:58.885684 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9dba4af9-c7zvt" Oct 30 14:19:59.200196 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:19:59.200155 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84cee745_3ab3_4075_a9b8_9ab78e6c0d01.slice/crio-7fa03efceedc3a92f3d8fc21d191de7ee2f5c3ed48390bf1625337837e5a041f WatchSource:0}: Error finding container 7fa03efceedc3a92f3d8fc21d191de7ee2f5c3ed48390bf1625337837e5a041f: Status 404 returned error can't find the container with id 7fa03efceedc3a92f3d8fc21d191de7ee2f5c3ed48390bf1625337837e5a041f Oct 30 14:19:59.201403 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:19:59.201378 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9dba4af9-c7zvt"] Oct 30 14:20:00.026155 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:00.026119 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9dba4af9-c7zvt" event={"ID":"84cee745-3ab3-4075-a9b8-9ab78e6c0d01","Type":"ContainerStarted","Data":"f74cdb1385614e779f8d24ec813c24d43d481a53b994a46a5e5a94344c3909ba"} Oct 30 14:20:00.026661 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:00.026182 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9dba4af9-c7zvt" event={"ID":"84cee745-3ab3-4075-a9b8-9ab78e6c0d01","Type":"ContainerStarted","Data":"7fa03efceedc3a92f3d8fc21d191de7ee2f5c3ed48390bf1625337837e5a041f"} Oct 30 14:20:00.026661 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:00.026582 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:20:00.074830 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:00.074778 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9dba4af9-c7zvt" podStartSLOduration=2.074759157 podStartE2EDuration="2.074759157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:19:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:20:00.072203023 +0000 UTC m=+1869.816337063" watchObservedRunningTime="2026-10-30 14:20:00.074759157 +0000 UTC m=+1869.818893201" Oct 30 14:20:01.036278 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:01.036248 4982 generic.go:356] "Generic (PLEG): container finished" podID="84cee745-3ab3-4075-a9b8-9ab78e6c0d01" containerID="f74cdb1385614e779f8d24ec813c24d43d481a53b994a46a5e5a94344c3909ba" exitCode=0 Oct 30 14:20:01.036692 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:01.036333 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9dba4af9-c7zvt" event={"ID":"84cee745-3ab3-4075-a9b8-9ab78e6c0d01","Type":"ContainerDied","Data":"f74cdb1385614e779f8d24ec813c24d43d481a53b994a46a5e5a94344c3909ba"} Oct 30 14:20:01.986192 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:01.986155 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:20:02.027413 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:02.027345 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8" Oct 30 14:20:02.046826 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:02.046795 4982 generic.go:356] "Generic (PLEG): container finished" podID="00eefe61-57f5-4d1c-b9a5-8dd54db2922e" containerID="797831e639462afe18671a43b4d40589e0afcb8b3fe4208cb8d075134fa2a194" exitCode=0 Oct 30 14:20:02.047270 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:02.046836 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8" event={"ID":"00eefe61-57f5-4d1c-b9a5-8dd54db2922e","Type":"ContainerDied","Data":"797831e639462afe18671a43b4d40589e0afcb8b3fe4208cb8d075134fa2a194"} Oct 30 14:20:02.047270 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:02.046859 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8" Oct 30 14:20:02.047270 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:02.046914 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8" event={"ID":"00eefe61-57f5-4d1c-b9a5-8dd54db2922e","Type":"ContainerDied","Data":"c4fe3274541b6f22597a5219e5bfc984e20b32f3174e644a1b5f2516078ce3c0"} Oct 30 14:20:02.047270 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:02.046954 4982 scope.go:122] "RemoveContainer" containerID="797831e639462afe18671a43b4d40589e0afcb8b3fe4208cb8d075134fa2a194" Oct 30 14:20:02.062373 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:02.062341 4982 scope.go:122] "RemoveContainer" containerID="797831e639462afe18671a43b4d40589e0afcb8b3fe4208cb8d075134fa2a194" Oct 30 14:20:02.062810 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:20:02.062786 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"797831e639462afe18671a43b4d40589e0afcb8b3fe4208cb8d075134fa2a194\": container with ID starting with 797831e639462afe18671a43b4d40589e0afcb8b3fe4208cb8d075134fa2a194 not found: ID does not exist" containerID="797831e639462afe18671a43b4d40589e0afcb8b3fe4208cb8d075134fa2a194" Oct 30 14:20:02.062896 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:02.062821 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"797831e639462afe18671a43b4d40589e0afcb8b3fe4208cb8d075134fa2a194"} err="failed to get container status \"797831e639462afe18671a43b4d40589e0afcb8b3fe4208cb8d075134fa2a194\": rpc error: code = NotFound desc = could not find container \"797831e639462afe18671a43b4d40589e0afcb8b3fe4208cb8d075134fa2a194\": container with ID starting with 797831e639462afe18671a43b4d40589e0afcb8b3fe4208cb8d075134fa2a194 not found: ID does not exist" Oct 30 14:20:02.148318 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:02.146579 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00eefe61-57f5-4d1c-b9a5-8dd54db2922e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00eefe61-57f5-4d1c-b9a5-8dd54db2922e-workload-certs\") pod \"00eefe61-57f5-4d1c-b9a5-8dd54db2922e\" (UID: \"00eefe61-57f5-4d1c-b9a5-8dd54db2922e\") " Oct 30 14:20:02.148318 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:02.147426 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/00eefe61-57f5-4d1c-b9a5-8dd54db2922e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00eefe61-57f5-4d1c-b9a5-8dd54db2922e-istio-podinfo\") pod \"00eefe61-57f5-4d1c-b9a5-8dd54db2922e\" (UID: \"00eefe61-57f5-4d1c-b9a5-8dd54db2922e\") " Oct 30 14:20:02.148318 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:02.147457 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00eefe61-57f5-4d1c-b9a5-8dd54db2922e-workload-certs" pod "00eefe61-57f5-4d1c-b9a5-8dd54db2922e" (UID: "00eefe61-57f5-4d1c-b9a5-8dd54db2922e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:20:02.148318 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:02.147530 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00eefe61-57f5-4d1c-b9a5-8dd54db2922e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00eefe61-57f5-4d1c-b9a5-8dd54db2922e-istio-data\") pod \"00eefe61-57f5-4d1c-b9a5-8dd54db2922e\" (UID: \"00eefe61-57f5-4d1c-b9a5-8dd54db2922e\") " Oct 30 14:20:02.148318 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:02.147566 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00eefe61-57f5-4d1c-b9a5-8dd54db2922e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00eefe61-57f5-4d1c-b9a5-8dd54db2922e-istio-envoy\") pod \"00eefe61-57f5-4d1c-b9a5-8dd54db2922e\" (UID: \"00eefe61-57f5-4d1c-b9a5-8dd54db2922e\") " Oct 30 14:20:02.148318 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:02.147672 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00eefe61-57f5-4d1c-b9a5-8dd54db2922e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00eefe61-57f5-4d1c-b9a5-8dd54db2922e-workload-socket\") pod \"00eefe61-57f5-4d1c-b9a5-8dd54db2922e\" (UID: \"00eefe61-57f5-4d1c-b9a5-8dd54db2922e\") " Oct 30 14:20:02.148318 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:02.147854 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00eefe61-57f5-4d1c-b9a5-8dd54db2922e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00eefe61-57f5-4d1c-b9a5-8dd54db2922e-credential-socket\") pod \"00eefe61-57f5-4d1c-b9a5-8dd54db2922e\" (UID: \"00eefe61-57f5-4d1c-b9a5-8dd54db2922e\") " Oct 30 14:20:02.148318 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:02.147891 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00eefe61-57f5-4d1c-b9a5-8dd54db2922e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00eefe61-57f5-4d1c-b9a5-8dd54db2922e-istiod-ca-cert\") pod \"00eefe61-57f5-4d1c-b9a5-8dd54db2922e\" (UID: \"00eefe61-57f5-4d1c-b9a5-8dd54db2922e\") " Oct 30 14:20:02.148318 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:02.147971 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00eefe61-57f5-4d1c-b9a5-8dd54db2922e-kube-api-access-8vvkl\" (UniqueName: \"kubernetes.io/projected/00eefe61-57f5-4d1c-b9a5-8dd54db2922e-kube-api-access-8vvkl\") pod \"00eefe61-57f5-4d1c-b9a5-8dd54db2922e\" (UID: \"00eefe61-57f5-4d1c-b9a5-8dd54db2922e\") " Oct 30 14:20:02.148318 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:02.148107 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00eefe61-57f5-4d1c-b9a5-8dd54db2922e-istio-token\" (UniqueName: \"kubernetes.io/projected/00eefe61-57f5-4d1c-b9a5-8dd54db2922e-istio-token\") pod \"00eefe61-57f5-4d1c-b9a5-8dd54db2922e\" (UID: \"00eefe61-57f5-4d1c-b9a5-8dd54db2922e\") " Oct 30 14:20:02.149061 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:02.148336 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00eefe61-57f5-4d1c-b9a5-8dd54db2922e-workload-socket" pod "00eefe61-57f5-4d1c-b9a5-8dd54db2922e" (UID: "00eefe61-57f5-4d1c-b9a5-8dd54db2922e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:20:02.149061 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:02.148505 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00eefe61-57f5-4d1c-b9a5-8dd54db2922e-istio-data" pod "00eefe61-57f5-4d1c-b9a5-8dd54db2922e" (UID: "00eefe61-57f5-4d1c-b9a5-8dd54db2922e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:20:02.149061 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:02.148957 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00eefe61-57f5-4d1c-b9a5-8dd54db2922e-credential-socket" pod "00eefe61-57f5-4d1c-b9a5-8dd54db2922e" (UID: "00eefe61-57f5-4d1c-b9a5-8dd54db2922e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:20:02.149835 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:02.149814 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00eefe61-57f5-4d1c-b9a5-8dd54db2922e-istiod-ca-cert" pod "00eefe61-57f5-4d1c-b9a5-8dd54db2922e" (UID: "00eefe61-57f5-4d1c-b9a5-8dd54db2922e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:20:02.150109 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:02.150085 4982 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00eefe61-57f5-4d1c-b9a5-8dd54db2922e-credential-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:20:02.150211 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:02.150116 4982 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00eefe61-57f5-4d1c-b9a5-8dd54db2922e-istiod-ca-cert\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:20:02.150211 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:02.150130 4982 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00eefe61-57f5-4d1c-b9a5-8dd54db2922e-workload-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:20:02.150211 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:02.150142 4982 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00eefe61-57f5-4d1c-b9a5-8dd54db2922e-istio-data\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:20:02.150211 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:02.150155 4982 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00eefe61-57f5-4d1c-b9a5-8dd54db2922e-workload-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:20:02.153062 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:02.153035 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00eefe61-57f5-4d1c-b9a5-8dd54db2922e-istio-token" pod "00eefe61-57f5-4d1c-b9a5-8dd54db2922e" (UID: "00eefe61-57f5-4d1c-b9a5-8dd54db2922e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:20:02.160008 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:02.159977 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/00eefe61-57f5-4d1c-b9a5-8dd54db2922e-istio-podinfo" pod "00eefe61-57f5-4d1c-b9a5-8dd54db2922e" (UID: "00eefe61-57f5-4d1c-b9a5-8dd54db2922e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:20:02.161623 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:02.161599 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00eefe61-57f5-4d1c-b9a5-8dd54db2922e-kube-api-access-8vvkl" pod "00eefe61-57f5-4d1c-b9a5-8dd54db2922e" (UID: "00eefe61-57f5-4d1c-b9a5-8dd54db2922e"). InnerVolumeSpecName "kube-api-access-8vvkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:20:02.162299 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:02.162231 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00eefe61-57f5-4d1c-b9a5-8dd54db2922e-istio-envoy" pod "00eefe61-57f5-4d1c-b9a5-8dd54db2922e" (UID: "00eefe61-57f5-4d1c-b9a5-8dd54db2922e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:20:02.252076 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:02.252044 4982 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00eefe61-57f5-4d1c-b9a5-8dd54db2922e-istio-podinfo\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:20:02.252076 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:02.252075 4982 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00eefe61-57f5-4d1c-b9a5-8dd54db2922e-istio-envoy\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:20:02.252276 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:02.252086 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vvkl\" (UniqueName: \"kubernetes.io/projected/00eefe61-57f5-4d1c-b9a5-8dd54db2922e-kube-api-access-8vvkl\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:20:02.252276 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:02.252098 4982 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00eefe61-57f5-4d1c-b9a5-8dd54db2922e-istio-token\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:20:02.269187 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:02.269160 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9dba4af9-c7zvt" Oct 30 14:20:02.352591 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:02.352556 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84cee745-3ab3-4075-a9b8-9ab78e6c0d01-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84cee745-3ab3-4075-a9b8-9ab78e6c0d01-maas-api-openshift-ca\") pod \"84cee745-3ab3-4075-a9b8-9ab78e6c0d01\" (UID: \"84cee745-3ab3-4075-a9b8-9ab78e6c0d01\") " Oct 30 14:20:02.352816 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:02.352712 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84cee745-3ab3-4075-a9b8-9ab78e6c0d01-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84cee745-3ab3-4075-a9b8-9ab78e6c0d01-maas-api-xks-ca\") pod \"84cee745-3ab3-4075-a9b8-9ab78e6c0d01\" (UID: \"84cee745-3ab3-4075-a9b8-9ab78e6c0d01\") " Oct 30 14:20:02.352992 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:02.352964 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84cee745-3ab3-4075-a9b8-9ab78e6c0d01-maas-api-openshift-ca" pod "84cee745-3ab3-4075-a9b8-9ab78e6c0d01" (UID: "84cee745-3ab3-4075-a9b8-9ab78e6c0d01"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:20:02.353072 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:02.353050 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84cee745-3ab3-4075-a9b8-9ab78e6c0d01-kube-api-access-89fqn\" (UniqueName: \"kubernetes.io/projected/84cee745-3ab3-4075-a9b8-9ab78e6c0d01-kube-api-access-89fqn\") pod \"84cee745-3ab3-4075-a9b8-9ab78e6c0d01\" (UID: \"84cee745-3ab3-4075-a9b8-9ab78e6c0d01\") " Oct 30 14:20:02.353705 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:02.353630 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84cee745-3ab3-4075-a9b8-9ab78e6c0d01-maas-api-openshift-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:20:02.355556 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:02.355451 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84cee745-3ab3-4075-a9b8-9ab78e6c0d01-kube-api-access-89fqn" pod "84cee745-3ab3-4075-a9b8-9ab78e6c0d01" (UID: "84cee745-3ab3-4075-a9b8-9ab78e6c0d01"). InnerVolumeSpecName "kube-api-access-89fqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:20:02.355556 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:02.355519 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84cee745-3ab3-4075-a9b8-9ab78e6c0d01-maas-api-xks-ca" pod "84cee745-3ab3-4075-a9b8-9ab78e6c0d01" (UID: "84cee745-3ab3-4075-a9b8-9ab78e6c0d01"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:20:02.412260 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:02.412210 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8"] Oct 30 14:20:02.415079 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:02.415048 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-7zkr8"] Oct 30 14:20:02.455613 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:02.455588 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89fqn\" (UniqueName: \"kubernetes.io/projected/84cee745-3ab3-4075-a9b8-9ab78e6c0d01-kube-api-access-89fqn\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:20:02.455613 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:02.455614 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84cee745-3ab3-4075-a9b8-9ab78e6c0d01-maas-api-xks-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:20:03.056311 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:03.056216 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9dba4af9-c7zvt" Oct 30 14:20:03.056311 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:03.056234 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9dba4af9-c7zvt" event={"ID":"84cee745-3ab3-4075-a9b8-9ab78e6c0d01","Type":"ContainerDied","Data":"7fa03efceedc3a92f3d8fc21d191de7ee2f5c3ed48390bf1625337837e5a041f"} Oct 30 14:20:03.056311 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:03.056289 4982 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7fa03efceedc3a92f3d8fc21d191de7ee2f5c3ed48390bf1625337837e5a041f" Oct 30 14:20:03.936997 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:03.936674 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:20:04.096333 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:04.096280 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:20:04.402292 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:04.402258 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00eefe61-57f5-4d1c-b9a5-8dd54db2922e" path="/var/lib/kubelet/pods/00eefe61-57f5-4d1c-b9a5-8dd54db2922e/volumes" Oct 30 14:20:04.789694 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:04.789589 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:20:04.891155 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:04.891122 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d727dc-plmx7"] Oct 30 14:20:04.891803 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:04.891786 4982 cpu_manager.go:414] "containerMap: removing container" podUID="84cee745-3ab3-4075-a9b8-9ab78e6c0d01" containerName="revoke-keys" Oct 30 14:20:04.891887 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:04.891806 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84cee745-3ab3-4075-a9b8-9ab78e6c0d01" containerName="revoke-keys" Oct 30 14:20:04.891887 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:04.891851 4982 cpu_manager.go:414] "containerMap: removing container" podUID="00eefe61-57f5-4d1c-b9a5-8dd54db2922e" containerName="istio-proxy" Oct 30 14:20:04.891887 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:04.891864 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00eefe61-57f5-4d1c-b9a5-8dd54db2922e" containerName="istio-proxy" Oct 30 14:20:04.891985 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:04.891951 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="00eefe61-57f5-4d1c-b9a5-8dd54db2922e" containerName="istio-proxy" Oct 30 14:20:04.891985 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:04.891962 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="84cee745-3ab3-4075-a9b8-9ab78e6c0d01" containerName="revoke-keys" Oct 30 14:20:04.895963 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:04.895944 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d727dc-plmx7" Oct 30 14:20:04.902263 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:04.902237 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aebc69a-cfda-4a73-99f8-4a9ecbdcafb3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d727dc-plmx7\" (UID: \"3aebc69a-cfda-4a73-99f8-4a9ecbdcafb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d727dc-plmx7" Oct 30 14:20:04.902401 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:04.902268 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb87x\" (UniqueName: \"kubernetes.io/projected/3aebc69a-cfda-4a73-99f8-4a9ecbdcafb3-kube-api-access-pb87x\") pod \"maas-api-revoke-keys-e2e-api-a-d727dc-plmx7\" (UID: \"3aebc69a-cfda-4a73-99f8-4a9ecbdcafb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d727dc-plmx7" Oct 30 14:20:04.902528 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:04.902503 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aebc69a-cfda-4a73-99f8-4a9ecbdcafb3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d727dc-plmx7\" (UID: \"3aebc69a-cfda-4a73-99f8-4a9ecbdcafb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d727dc-plmx7" Oct 30 14:20:04.918049 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:04.918022 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d727dc-plmx7"] Oct 30 14:20:04.945764 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:04.945728 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:20:04.950882 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:04.950858 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:20:04.950992 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:04.950977 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:20:04.980240 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:04.980211 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5a9d7b-9bb744df9-2qrjh" Oct 30 14:20:05.003684 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:05.003625 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aebc69a-cfda-4a73-99f8-4a9ecbdcafb3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d727dc-plmx7\" (UID: \"3aebc69a-cfda-4a73-99f8-4a9ecbdcafb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d727dc-plmx7" Oct 30 14:20:05.003813 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:05.003793 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aebc69a-cfda-4a73-99f8-4a9ecbdcafb3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d727dc-plmx7\" (UID: \"3aebc69a-cfda-4a73-99f8-4a9ecbdcafb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d727dc-plmx7" Oct 30 14:20:05.003882 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:05.003826 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb87x\" (UniqueName: \"kubernetes.io/projected/3aebc69a-cfda-4a73-99f8-4a9ecbdcafb3-kube-api-access-pb87x\") pod \"maas-api-revoke-keys-e2e-api-a-d727dc-plmx7\" (UID: \"3aebc69a-cfda-4a73-99f8-4a9ecbdcafb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d727dc-plmx7" Oct 30 14:20:05.004534 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:05.004511 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aebc69a-cfda-4a73-99f8-4a9ecbdcafb3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d727dc-plmx7\" (UID: \"3aebc69a-cfda-4a73-99f8-4a9ecbdcafb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d727dc-plmx7" Oct 30 14:20:05.006067 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:05.006050 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aebc69a-cfda-4a73-99f8-4a9ecbdcafb3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d727dc-plmx7\" (UID: \"3aebc69a-cfda-4a73-99f8-4a9ecbdcafb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d727dc-plmx7" Oct 30 14:20:05.035892 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:05.035858 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb87x\" (UniqueName: \"kubernetes.io/projected/3aebc69a-cfda-4a73-99f8-4a9ecbdcafb3-kube-api-access-pb87x\") pod \"maas-api-revoke-keys-e2e-api-a-d727dc-plmx7\" (UID: \"3aebc69a-cfda-4a73-99f8-4a9ecbdcafb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d727dc-plmx7" Oct 30 14:20:05.209732 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:05.209694 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d727dc-plmx7" Oct 30 14:20:05.513975 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:20:05.513937 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aebc69a_cfda_4a73_99f8_4a9ecbdcafb3.slice/crio-5a0ef291aa6a2c9c6f5f99940debe956af5bee3d9020b9ef10bd788b764d2c59 WatchSource:0}: Error finding container 5a0ef291aa6a2c9c6f5f99940debe956af5bee3d9020b9ef10bd788b764d2c59: Status 404 returned error can't find the container with id 5a0ef291aa6a2c9c6f5f99940debe956af5bee3d9020b9ef10bd788b764d2c59 Oct 30 14:20:05.514444 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:05.514411 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d727dc-plmx7"] Oct 30 14:20:06.030887 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:06.030852 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:20:06.109984 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:06.109940 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d727dc-plmx7" event={"ID":"3aebc69a-cfda-4a73-99f8-4a9ecbdcafb3","Type":"ContainerStarted","Data":"f65febe65e870197362445c4cbbe79159f4fb58d539c6c3efb7e4d1e5d8893c5"} Oct 30 14:20:06.110267 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:06.110245 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d727dc-plmx7" event={"ID":"3aebc69a-cfda-4a73-99f8-4a9ecbdcafb3","Type":"ContainerStarted","Data":"5a0ef291aa6a2c9c6f5f99940debe956af5bee3d9020b9ef10bd788b764d2c59"} Oct 30 14:20:06.156800 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:06.156741 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d727dc-plmx7" podStartSLOduration=2.156722673 podStartE2EDuration="2.156722673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:20:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:20:06.15370211 +0000 UTC m=+1875.897836152" watchObservedRunningTime="2026-10-30 14:20:06.156722673 +0000 UTC m=+1875.900856716" Oct 30 14:20:07.000735 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:07.000702 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4bf33a-7f5fb57b55-9mjfq" Oct 30 14:20:07.118318 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:07.118288 4982 generic.go:356] "Generic (PLEG): container finished" podID="3aebc69a-cfda-4a73-99f8-4a9ecbdcafb3" containerID="f65febe65e870197362445c4cbbe79159f4fb58d539c6c3efb7e4d1e5d8893c5" exitCode=0 Oct 30 14:20:07.118486 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:07.118365 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d727dc-plmx7" event={"ID":"3aebc69a-cfda-4a73-99f8-4a9ecbdcafb3","Type":"ContainerDied","Data":"f65febe65e870197362445c4cbbe79159f4fb58d539c6c3efb7e4d1e5d8893c5"} Oct 30 14:20:08.009168 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:08.009134 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-4bf33a-6bb59459b-4qdp6" Oct 30 14:20:08.013210 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:08.013183 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:20:08.163035 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:08.162998 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:20:08.357849 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:08.357825 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d727dc-plmx7" Oct 30 14:20:08.448213 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:08.448176 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:20:08.475585 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:08.475547 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3aebc69a-cfda-4a73-99f8-4a9ecbdcafb3-kube-api-access-pb87x\" (UniqueName: \"kubernetes.io/projected/3aebc69a-cfda-4a73-99f8-4a9ecbdcafb3-kube-api-access-pb87x\") pod \"3aebc69a-cfda-4a73-99f8-4a9ecbdcafb3\" (UID: \"3aebc69a-cfda-4a73-99f8-4a9ecbdcafb3\") " Oct 30 14:20:08.475781 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:08.475737 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3aebc69a-cfda-4a73-99f8-4a9ecbdcafb3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aebc69a-cfda-4a73-99f8-4a9ecbdcafb3-maas-api-xks-ca\") pod \"3aebc69a-cfda-4a73-99f8-4a9ecbdcafb3\" (UID: \"3aebc69a-cfda-4a73-99f8-4a9ecbdcafb3\") " Oct 30 14:20:08.475866 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:08.475832 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3aebc69a-cfda-4a73-99f8-4a9ecbdcafb3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aebc69a-cfda-4a73-99f8-4a9ecbdcafb3-maas-api-openshift-ca\") pod \"3aebc69a-cfda-4a73-99f8-4a9ecbdcafb3\" (UID: \"3aebc69a-cfda-4a73-99f8-4a9ecbdcafb3\") " Oct 30 14:20:08.476267 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:08.476215 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3aebc69a-cfda-4a73-99f8-4a9ecbdcafb3-maas-api-openshift-ca" pod "3aebc69a-cfda-4a73-99f8-4a9ecbdcafb3" (UID: "3aebc69a-cfda-4a73-99f8-4a9ecbdcafb3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:20:08.476726 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:08.476685 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aebc69a-cfda-4a73-99f8-4a9ecbdcafb3-maas-api-openshift-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:20:08.477900 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:08.477878 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3aebc69a-cfda-4a73-99f8-4a9ecbdcafb3-maas-api-xks-ca" pod "3aebc69a-cfda-4a73-99f8-4a9ecbdcafb3" (UID: "3aebc69a-cfda-4a73-99f8-4a9ecbdcafb3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:20:08.478042 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:08.478023 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aebc69a-cfda-4a73-99f8-4a9ecbdcafb3-kube-api-access-pb87x" pod "3aebc69a-cfda-4a73-99f8-4a9ecbdcafb3" (UID: "3aebc69a-cfda-4a73-99f8-4a9ecbdcafb3"). InnerVolumeSpecName "kube-api-access-pb87x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:20:08.577828 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:08.577751 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pb87x\" (UniqueName: \"kubernetes.io/projected/3aebc69a-cfda-4a73-99f8-4a9ecbdcafb3-kube-api-access-pb87x\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:20:08.577828 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:08.577780 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aebc69a-cfda-4a73-99f8-4a9ecbdcafb3-maas-api-xks-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:20:09.092586 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:09.092539 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:20:09.136066 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:09.136036 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d727dc-plmx7" Oct 30 14:20:09.136066 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:09.136057 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d727dc-plmx7" event={"ID":"3aebc69a-cfda-4a73-99f8-4a9ecbdcafb3","Type":"ContainerDied","Data":"5a0ef291aa6a2c9c6f5f99940debe956af5bee3d9020b9ef10bd788b764d2c59"} Oct 30 14:20:09.136339 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:09.136114 4982 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a0ef291aa6a2c9c6f5f99940debe956af5bee3d9020b9ef10bd788b764d2c59" Oct 30 14:20:09.633969 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:09.633928 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:20:09.856501 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:09.856418 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:20:10.142432 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:10.142396 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:20:10.305663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:10.298308 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:20:12.144345 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:12.144313 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:20:14.282092 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:14.279204 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:20:15.554682 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:15.554632 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:20:15.705782 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:15.705746 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:20:16.172727 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:16.171836 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:20:18.011697 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:18.011626 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:20:19.929352 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:19.929310 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:20:21.271596 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:21.271563 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-972fee-4k5ct"] Oct 30 14:20:21.272274 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:21.272257 4982 cpu_manager.go:414] "containerMap: removing container" podUID="3aebc69a-cfda-4a73-99f8-4a9ecbdcafb3" containerName="revoke-keys" Oct 30 14:20:21.272335 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:21.272276 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aebc69a-cfda-4a73-99f8-4a9ecbdcafb3" containerName="revoke-keys" Oct 30 14:20:21.272399 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:21.272390 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aebc69a-cfda-4a73-99f8-4a9ecbdcafb3" containerName="revoke-keys" Oct 30 14:20:21.275972 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:21.275954 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-972fee-4k5ct" Oct 30 14:20:21.304179 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:21.304142 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-972fee-4k5ct"] Oct 30 14:20:21.319930 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:21.306602 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:20:21.320098 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:21.306767 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:20:21.335334 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:21.335306 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq6kx\" (UniqueName: \"kubernetes.io/projected/c518067a-74aa-4008-992b-83608da423f7-kube-api-access-dq6kx\") pod \"maas-api-revoke-keys-e2e-api-a-972fee-4k5ct\" (UID: \"c518067a-74aa-4008-992b-83608da423f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-972fee-4k5ct" Oct 30 14:20:21.335448 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:21.335340 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c518067a-74aa-4008-992b-83608da423f7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-972fee-4k5ct\" (UID: \"c518067a-74aa-4008-992b-83608da423f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-972fee-4k5ct" Oct 30 14:20:21.335604 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:21.335577 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c518067a-74aa-4008-992b-83608da423f7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-972fee-4k5ct\" (UID: \"c518067a-74aa-4008-992b-83608da423f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-972fee-4k5ct" Oct 30 14:20:21.437405 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:21.437370 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq6kx\" (UniqueName: \"kubernetes.io/projected/c518067a-74aa-4008-992b-83608da423f7-kube-api-access-dq6kx\") pod \"maas-api-revoke-keys-e2e-api-a-972fee-4k5ct\" (UID: \"c518067a-74aa-4008-992b-83608da423f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-972fee-4k5ct" Oct 30 14:20:21.437405 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:21.437407 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c518067a-74aa-4008-992b-83608da423f7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-972fee-4k5ct\" (UID: \"c518067a-74aa-4008-992b-83608da423f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-972fee-4k5ct" Oct 30 14:20:21.437620 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:21.437509 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c518067a-74aa-4008-992b-83608da423f7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-972fee-4k5ct\" (UID: \"c518067a-74aa-4008-992b-83608da423f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-972fee-4k5ct" Oct 30 14:20:21.438143 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:21.438122 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c518067a-74aa-4008-992b-83608da423f7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-972fee-4k5ct\" (UID: \"c518067a-74aa-4008-992b-83608da423f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-972fee-4k5ct" Oct 30 14:20:21.439795 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:21.439771 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c518067a-74aa-4008-992b-83608da423f7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-972fee-4k5ct\" (UID: \"c518067a-74aa-4008-992b-83608da423f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-972fee-4k5ct" Oct 30 14:20:21.464023 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:21.463999 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq6kx\" (UniqueName: \"kubernetes.io/projected/c518067a-74aa-4008-992b-83608da423f7-kube-api-access-dq6kx\") pod \"maas-api-revoke-keys-e2e-api-a-972fee-4k5ct\" (UID: \"c518067a-74aa-4008-992b-83608da423f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-972fee-4k5ct" Oct 30 14:20:21.590058 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:21.590014 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-972fee-4k5ct" Oct 30 14:20:21.909914 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:20:21.909871 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc518067a_74aa_4008_992b_83608da423f7.slice/crio-b84c1222751bbff11e5c747cf163423087f591dae1e426598192fbb35be6edf5 WatchSource:0}: Error finding container b84c1222751bbff11e5c747cf163423087f591dae1e426598192fbb35be6edf5: Status 404 returned error can't find the container with id b84c1222751bbff11e5c747cf163423087f591dae1e426598192fbb35be6edf5 Oct 30 14:20:21.912554 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:21.912520 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-972fee-4k5ct"] Oct 30 14:20:21.915427 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:21.915393 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:20:22.262177 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:22.262096 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-972fee-4k5ct" event={"ID":"c518067a-74aa-4008-992b-83608da423f7","Type":"ContainerStarted","Data":"b84c1222751bbff11e5c747cf163423087f591dae1e426598192fbb35be6edf5"} Oct 30 14:20:23.334090 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:23.333842 4982 generic.go:356] "Generic (PLEG): container finished" podID="c518067a-74aa-4008-992b-83608da423f7" containerID="62db31b6e442c83835af08227ef58d9af15774b093c8b5730cdd98172dd4d50b" exitCode=0 Oct 30 14:20:23.334090 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:23.333998 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-972fee-4k5ct" event={"ID":"c518067a-74aa-4008-992b-83608da423f7","Type":"ContainerDied","Data":"62db31b6e442c83835af08227ef58d9af15774b093c8b5730cdd98172dd4d50b"} Oct 30 14:20:24.133122 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:24.131579 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:20:24.716675 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:24.716598 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-972fee-4k5ct" Oct 30 14:20:24.766596 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:24.766556 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c518067a-74aa-4008-992b-83608da423f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c518067a-74aa-4008-992b-83608da423f7-maas-api-xks-ca\") pod \"c518067a-74aa-4008-992b-83608da423f7\" (UID: \"c518067a-74aa-4008-992b-83608da423f7\") " Oct 30 14:20:24.766797 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:24.766776 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c518067a-74aa-4008-992b-83608da423f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c518067a-74aa-4008-992b-83608da423f7-maas-api-openshift-ca\") pod \"c518067a-74aa-4008-992b-83608da423f7\" (UID: \"c518067a-74aa-4008-992b-83608da423f7\") " Oct 30 14:20:24.767519 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:24.767492 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c518067a-74aa-4008-992b-83608da423f7-maas-api-openshift-ca" pod "c518067a-74aa-4008-992b-83608da423f7" (UID: "c518067a-74aa-4008-992b-83608da423f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:20:24.768889 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:24.768864 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c518067a-74aa-4008-992b-83608da423f7-maas-api-xks-ca" pod "c518067a-74aa-4008-992b-83608da423f7" (UID: "c518067a-74aa-4008-992b-83608da423f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:20:24.891663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:24.868183 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c518067a-74aa-4008-992b-83608da423f7-kube-api-access-dq6kx\" (UniqueName: \"kubernetes.io/projected/c518067a-74aa-4008-992b-83608da423f7-kube-api-access-dq6kx\") pod \"c518067a-74aa-4008-992b-83608da423f7\" (UID: \"c518067a-74aa-4008-992b-83608da423f7\") " Oct 30 14:20:24.891663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:24.869112 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c518067a-74aa-4008-992b-83608da423f7-maas-api-openshift-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:20:24.891663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:24.869141 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c518067a-74aa-4008-992b-83608da423f7-maas-api-xks-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:20:24.935710 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:24.927301 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c518067a-74aa-4008-992b-83608da423f7-kube-api-access-dq6kx" pod "c518067a-74aa-4008-992b-83608da423f7" (UID: "c518067a-74aa-4008-992b-83608da423f7"). InnerVolumeSpecName "kube-api-access-dq6kx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:20:24.976663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:24.970960 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq6kx\" (UniqueName: \"kubernetes.io/projected/c518067a-74aa-4008-992b-83608da423f7-kube-api-access-dq6kx\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:20:25.349161 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:25.349134 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-972fee-4k5ct" Oct 30 14:20:25.349161 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:25.349151 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-972fee-4k5ct" event={"ID":"c518067a-74aa-4008-992b-83608da423f7","Type":"ContainerDied","Data":"b84c1222751bbff11e5c747cf163423087f591dae1e426598192fbb35be6edf5"} Oct 30 14:20:25.349387 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:25.349199 4982 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b84c1222751bbff11e5c747cf163423087f591dae1e426598192fbb35be6edf5" Oct 30 14:20:25.755407 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:25.755310 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:20:26.414758 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:26.412369 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:20:26.538561 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:26.538525 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9dba4af9-c7zvt"] Oct 30 14:20:26.549597 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:26.549567 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9dba4af9-c7zvt"] Oct 30 14:20:26.599994 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:26.599959 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:20:28.011328 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:20:28.011293 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:20:28.402699 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:28.402666 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84cee745-3ab3-4075-a9b8-9ab78e6c0d01" path="/var/lib/kubelet/pods/84cee745-3ab3-4075-a9b8-9ab78e6c0d01/volumes" Oct 30 14:20:28.588908 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:28.588863 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:20:30.615036 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:30.612733 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:20:31.698901 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:31.698866 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-972fee-fdbb7989f-mdfj4"] Oct 30 14:20:31.836187 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:31.836150 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:20:32.007268 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:32.007173 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:20:32.683222 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:32.683184 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:20:34.477049 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:34.477015 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:20:34.626446 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:34.626401 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:20:36.433798 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:36.433758 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:20:36.571740 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:36.571709 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:20:36.699789 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:36.699684 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-972fee-fdbb7989f-mdfj4" podUID="d9f378f2-91ec-4bab-b862-ae525e311b73" containerName="payload-pre-processing" containerID="cri-o://bcd82b4492fadfcc077ed2a80072ccbf00dde225bacbdcdf4badbe2d930d5f14" gracePeriod=25 Oct 30 14:20:37.005016 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:37.004990 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-972fee-fdbb7989f-mdfj4" Oct 30 14:20:37.112976 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:37.112944 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9f378f2-91ec-4bab-b862-ae525e311b73-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9f378f2-91ec-4bab-b862-ae525e311b73-plugins-config-volume\") pod \"d9f378f2-91ec-4bab-b862-ae525e311b73\" (UID: \"d9f378f2-91ec-4bab-b862-ae525e311b73\") " Oct 30 14:20:37.113164 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:37.113152 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9f378f2-91ec-4bab-b862-ae525e311b73-kube-api-access-vw6wp\" (UniqueName: \"kubernetes.io/projected/d9f378f2-91ec-4bab-b862-ae525e311b73-kube-api-access-vw6wp\") pod \"d9f378f2-91ec-4bab-b862-ae525e311b73\" (UID: \"d9f378f2-91ec-4bab-b862-ae525e311b73\") " Oct 30 14:20:37.113400 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:37.113369 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9f378f2-91ec-4bab-b862-ae525e311b73-plugins-config-volume" pod "d9f378f2-91ec-4bab-b862-ae525e311b73" (UID: "d9f378f2-91ec-4bab-b862-ae525e311b73"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:20:37.113578 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:37.113560 4982 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9f378f2-91ec-4bab-b862-ae525e311b73-plugins-config-volume\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:20:37.115265 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:37.115234 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9f378f2-91ec-4bab-b862-ae525e311b73-kube-api-access-vw6wp" pod "d9f378f2-91ec-4bab-b862-ae525e311b73" (UID: "d9f378f2-91ec-4bab-b862-ae525e311b73"). InnerVolumeSpecName "kube-api-access-vw6wp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:20:37.131546 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:37.131515 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:20:37.217774 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:37.216362 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vw6wp\" (UniqueName: \"kubernetes.io/projected/d9f378f2-91ec-4bab-b862-ae525e311b73-kube-api-access-vw6wp\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:20:37.475790 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:37.475754 4982 generic.go:356] "Generic (PLEG): container finished" podID="d9f378f2-91ec-4bab-b862-ae525e311b73" containerID="bcd82b4492fadfcc077ed2a80072ccbf00dde225bacbdcdf4badbe2d930d5f14" exitCode=0 Oct 30 14:20:37.476277 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:37.475856 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-972fee-fdbb7989f-mdfj4" event={"ID":"d9f378f2-91ec-4bab-b862-ae525e311b73","Type":"ContainerDied","Data":"bcd82b4492fadfcc077ed2a80072ccbf00dde225bacbdcdf4badbe2d930d5f14"} Oct 30 14:20:37.476277 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:37.475900 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-972fee-fdbb7989f-mdfj4" Oct 30 14:20:37.476277 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:37.475914 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-972fee-fdbb7989f-mdfj4" event={"ID":"d9f378f2-91ec-4bab-b862-ae525e311b73","Type":"ContainerDied","Data":"49ea30aefafe3701704280073cf317ff29461b2cda7b23583deed3bd0ecced7a"} Oct 30 14:20:37.476277 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:37.475938 4982 scope.go:122] "RemoveContainer" containerID="bcd82b4492fadfcc077ed2a80072ccbf00dde225bacbdcdf4badbe2d930d5f14" Oct 30 14:20:37.489613 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:37.489589 4982 scope.go:122] "RemoveContainer" containerID="bcd82b4492fadfcc077ed2a80072ccbf00dde225bacbdcdf4badbe2d930d5f14" Oct 30 14:20:37.490207 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:20:37.490180 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcd82b4492fadfcc077ed2a80072ccbf00dde225bacbdcdf4badbe2d930d5f14\": container with ID starting with bcd82b4492fadfcc077ed2a80072ccbf00dde225bacbdcdf4badbe2d930d5f14 not found: ID does not exist" containerID="bcd82b4492fadfcc077ed2a80072ccbf00dde225bacbdcdf4badbe2d930d5f14" Oct 30 14:20:37.490306 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:37.490219 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcd82b4492fadfcc077ed2a80072ccbf00dde225bacbdcdf4badbe2d930d5f14"} err="failed to get container status \"bcd82b4492fadfcc077ed2a80072ccbf00dde225bacbdcdf4badbe2d930d5f14\": rpc error: code = NotFound desc = could not find container \"bcd82b4492fadfcc077ed2a80072ccbf00dde225bacbdcdf4badbe2d930d5f14\": container with ID starting with bcd82b4492fadfcc077ed2a80072ccbf00dde225bacbdcdf4badbe2d930d5f14 not found: ID does not exist" Oct 30 14:20:37.537152 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:37.537124 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-972fee-fdbb7989f-mdfj4"] Oct 30 14:20:37.548407 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:37.548380 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-972fee-fdbb7989f-mdfj4"] Oct 30 14:20:38.407196 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:38.407157 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9f378f2-91ec-4bab-b862-ae525e311b73" path="/var/lib/kubelet/pods/d9f378f2-91ec-4bab-b862-ae525e311b73/volumes" Oct 30 14:20:38.632335 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:38.632305 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:20:40.484753 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:40.484714 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:20:40.957274 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:40.957236 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:20:41.103919 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:41.103885 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:20:42.153049 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:42.153010 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:20:42.352170 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:42.351459 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:20:44.061591 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:44.061556 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:20:44.388542 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:44.388428 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:20:46.192854 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:46.192816 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:20:48.030000 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:48.029951 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:20:49.872971 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:49.872932 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:20:51.693277 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:51.693238 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:20:53.567112 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:53.567067 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:20:55.827852 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:55.827819 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:20:56.957408 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:56.957370 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw"] Oct 30 14:20:56.957818 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:56.957604 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw" podUID="2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28" containerName="istio-proxy" containerID="cri-o://9e508c66aa266c2278f1dd5abb2387d3e3f4901a55d519df8f3ffceabfdd225b" gracePeriod=30 Oct 30 14:20:57.009091 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:57.009058 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4"] Oct 30 14:20:57.009312 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:57.009288 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4" podUID="b14bb682-071a-4c22-ad2d-cab7434a4192" containerName="istio-proxy" containerID="cri-o://31e08d6c89814d4efd0c371849a1e652cd204be7f967ab5a584e85e3d4e9ecd5" gracePeriod=30 Oct 30 14:20:57.242075 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:57.241992 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk"] Oct 30 14:20:57.242229 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:57.242207 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk" podUID="99cbf845-7193-4e4d-8315-b5b0614cfd33" containerName="istio-proxy" containerID="cri-o://c2e6c1f8067dbeaa782c8a4627a999a6a47e555d4161a0f79cd9ee848e98e031" gracePeriod=30 Oct 30 14:20:57.442007 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:57.441971 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls"] Oct 30 14:20:57.442519 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:57.442483 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls" podUID="2a20886d-6388-4460-ae47-12a5a28e0eb6" containerName="istio-proxy" containerID="cri-o://ba609330d711f7655ff170ef31402eaac26bea8c9f5f1ecc33e14d6ba9d81a0a" gracePeriod=30 Oct 30 14:20:57.653043 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:57.653008 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:20:58.012669 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:20:58.012577 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:20:58.898692 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:58.898321 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:20:59.170248 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:59.170144 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:20:59.913100 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:59.913000 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:20:59.985883 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:59.985846 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2b206b-t2tz7"] Oct 30 14:20:59.986537 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:59.986519 4982 cpu_manager.go:414] "containerMap: removing container" podUID="d9f378f2-91ec-4bab-b862-ae525e311b73" containerName="payload-pre-processing" Oct 30 14:20:59.986588 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:59.986540 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9f378f2-91ec-4bab-b862-ae525e311b73" containerName="payload-pre-processing" Oct 30 14:20:59.986588 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:59.986575 4982 cpu_manager.go:414] "containerMap: removing container" podUID="c518067a-74aa-4008-992b-83608da423f7" containerName="revoke-keys" Oct 30 14:20:59.986588 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:59.986580 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c518067a-74aa-4008-992b-83608da423f7" containerName="revoke-keys" Oct 30 14:20:59.986728 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:59.986684 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9f378f2-91ec-4bab-b862-ae525e311b73" containerName="payload-pre-processing" Oct 30 14:20:59.986728 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:59.986700 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="c518067a-74aa-4008-992b-83608da423f7" containerName="revoke-keys" Oct 30 14:20:59.991765 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:20:59.991745 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2b206b-t2tz7" Oct 30 14:21:00.011689 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:00.011635 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2b206b-t2tz7"] Oct 30 14:21:00.048874 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:00.048723 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d74808-c8z6d"] Oct 30 14:21:00.073590 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:00.073558 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:21:00.073801 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:00.073777 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:21:00.074618 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:00.074589 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c897e27-a3b6-4221-b7b2-7ae5cb159d36-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2b206b-t2tz7\" (UID: \"6c897e27-a3b6-4221-b7b2-7ae5cb159d36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2b206b-t2tz7" Oct 30 14:21:00.075973 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:00.075942 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktssl\" (UniqueName: \"kubernetes.io/projected/6c897e27-a3b6-4221-b7b2-7ae5cb159d36-kube-api-access-ktssl\") pod \"maas-api-revoke-keys-e2e-api-a-2b206b-t2tz7\" (UID: \"6c897e27-a3b6-4221-b7b2-7ae5cb159d36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2b206b-t2tz7" Oct 30 14:21:00.076168 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:00.076149 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c897e27-a3b6-4221-b7b2-7ae5cb159d36-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2b206b-t2tz7\" (UID: \"6c897e27-a3b6-4221-b7b2-7ae5cb159d36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2b206b-t2tz7" Oct 30 14:21:00.087082 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:00.087056 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d74808-c8z6d"] Oct 30 14:21:00.087459 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:00.087391 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d74808-c8z6d" Oct 30 14:21:00.182499 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:00.182393 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bff13918-6bc7-42b8-86ca-216d6baeb4f6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d74808-c8z6d\" (UID: \"bff13918-6bc7-42b8-86ca-216d6baeb4f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d74808-c8z6d" Oct 30 14:21:00.183018 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:00.182572 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktssl\" (UniqueName: \"kubernetes.io/projected/6c897e27-a3b6-4221-b7b2-7ae5cb159d36-kube-api-access-ktssl\") pod \"maas-api-revoke-keys-e2e-api-a-2b206b-t2tz7\" (UID: \"6c897e27-a3b6-4221-b7b2-7ae5cb159d36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2b206b-t2tz7" Oct 30 14:21:00.191312 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:00.182668 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c897e27-a3b6-4221-b7b2-7ae5cb159d36-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2b206b-t2tz7\" (UID: \"6c897e27-a3b6-4221-b7b2-7ae5cb159d36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2b206b-t2tz7" Oct 30 14:21:00.191716 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:00.191688 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bff13918-6bc7-42b8-86ca-216d6baeb4f6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d74808-c8z6d\" (UID: \"bff13918-6bc7-42b8-86ca-216d6baeb4f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d74808-c8z6d" Oct 30 14:21:00.191972 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:00.191937 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c897e27-a3b6-4221-b7b2-7ae5cb159d36-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2b206b-t2tz7\" (UID: \"6c897e27-a3b6-4221-b7b2-7ae5cb159d36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2b206b-t2tz7" Oct 30 14:21:00.192091 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:00.192025 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zbvp\" (UniqueName: \"kubernetes.io/projected/bff13918-6bc7-42b8-86ca-216d6baeb4f6-kube-api-access-6zbvp\") pod \"maas-api-revoke-keys-e2e-api-a-d74808-c8z6d\" (UID: \"bff13918-6bc7-42b8-86ca-216d6baeb4f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d74808-c8z6d" Oct 30 14:21:00.193703 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:00.193677 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c897e27-a3b6-4221-b7b2-7ae5cb159d36-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2b206b-t2tz7\" (UID: \"6c897e27-a3b6-4221-b7b2-7ae5cb159d36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2b206b-t2tz7" Oct 30 14:21:00.199143 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:00.199105 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c897e27-a3b6-4221-b7b2-7ae5cb159d36-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2b206b-t2tz7\" (UID: \"6c897e27-a3b6-4221-b7b2-7ae5cb159d36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2b206b-t2tz7" Oct 30 14:21:00.208818 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:00.208787 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktssl\" (UniqueName: \"kubernetes.io/projected/6c897e27-a3b6-4221-b7b2-7ae5cb159d36-kube-api-access-ktssl\") pod \"maas-api-revoke-keys-e2e-api-a-2b206b-t2tz7\" (UID: \"6c897e27-a3b6-4221-b7b2-7ae5cb159d36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2b206b-t2tz7" Oct 30 14:21:00.293576 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:00.293543 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zbvp\" (UniqueName: \"kubernetes.io/projected/bff13918-6bc7-42b8-86ca-216d6baeb4f6-kube-api-access-6zbvp\") pod \"maas-api-revoke-keys-e2e-api-a-d74808-c8z6d\" (UID: \"bff13918-6bc7-42b8-86ca-216d6baeb4f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d74808-c8z6d" Oct 30 14:21:00.293757 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:00.293609 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bff13918-6bc7-42b8-86ca-216d6baeb4f6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d74808-c8z6d\" (UID: \"bff13918-6bc7-42b8-86ca-216d6baeb4f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d74808-c8z6d" Oct 30 14:21:00.293978 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:00.293957 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bff13918-6bc7-42b8-86ca-216d6baeb4f6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d74808-c8z6d\" (UID: \"bff13918-6bc7-42b8-86ca-216d6baeb4f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d74808-c8z6d" Oct 30 14:21:00.294577 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:00.294546 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bff13918-6bc7-42b8-86ca-216d6baeb4f6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d74808-c8z6d\" (UID: \"bff13918-6bc7-42b8-86ca-216d6baeb4f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d74808-c8z6d" Oct 30 14:21:00.296084 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:00.296056 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bff13918-6bc7-42b8-86ca-216d6baeb4f6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d74808-c8z6d\" (UID: \"bff13918-6bc7-42b8-86ca-216d6baeb4f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d74808-c8z6d" Oct 30 14:21:00.306031 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:00.306008 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2b206b-t2tz7" Oct 30 14:21:00.319497 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:00.319478 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zbvp\" (UniqueName: \"kubernetes.io/projected/bff13918-6bc7-42b8-86ca-216d6baeb4f6-kube-api-access-6zbvp\") pod \"maas-api-revoke-keys-e2e-api-a-d74808-c8z6d\" (UID: \"bff13918-6bc7-42b8-86ca-216d6baeb4f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d74808-c8z6d" Oct 30 14:21:00.404570 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:00.404383 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d74808-c8z6d" Oct 30 14:21:00.615130 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:00.615093 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2b206b-t2tz7"] Oct 30 14:21:00.615549 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:21:00.615500 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c897e27_a3b6_4221_b7b2_7ae5cb159d36.slice/crio-23a2db58fb0a31e57fe403ba771144e528a26654dc919687dd739587767a6106 WatchSource:0}: Error finding container 23a2db58fb0a31e57fe403ba771144e528a26654dc919687dd739587767a6106: Status 404 returned error can't find the container with id 23a2db58fb0a31e57fe403ba771144e528a26654dc919687dd739587767a6106 Oct 30 14:21:00.683470 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:00.683442 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2b206b-t2tz7" event={"ID":"6c897e27-a3b6-4221-b7b2-7ae5cb159d36","Type":"ContainerStarted","Data":"23a2db58fb0a31e57fe403ba771144e528a26654dc919687dd739587767a6106"} Oct 30 14:21:00.717684 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:00.717621 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d74808-c8z6d"] Oct 30 14:21:00.783077 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:21:00.783040 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbff13918_6bc7_42b8_86ca_216d6baeb4f6.slice/crio-2d8da68392f6d096d7df82af7d75e49e12cba30a2bb742a2d266cb8fe75f43c0 WatchSource:0}: Error finding container 2d8da68392f6d096d7df82af7d75e49e12cba30a2bb742a2d266cb8fe75f43c0: Status 404 returned error can't find the container with id 2d8da68392f6d096d7df82af7d75e49e12cba30a2bb742a2d266cb8fe75f43c0 Oct 30 14:21:01.693748 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:01.693694 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d74808-c8z6d" event={"ID":"bff13918-6bc7-42b8-86ca-216d6baeb4f6","Type":"ContainerStarted","Data":"3dfde50d83a37bb9cac2b7f9ca55ea6070830d3d29abe559464966f1394c554a"} Oct 30 14:21:01.694141 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:01.693806 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d74808-c8z6d" event={"ID":"bff13918-6bc7-42b8-86ca-216d6baeb4f6","Type":"ContainerStarted","Data":"2d8da68392f6d096d7df82af7d75e49e12cba30a2bb742a2d266cb8fe75f43c0"} Oct 30 14:21:01.695486 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:01.695456 4982 generic.go:356] "Generic (PLEG): container finished" podID="6c897e27-a3b6-4221-b7b2-7ae5cb159d36" containerID="c697967614cff5094a283125ee9ab25b56553bdb6e99e3af9e3b280703297328" exitCode=0 Oct 30 14:21:01.695597 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:01.695533 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2b206b-t2tz7" event={"ID":"6c897e27-a3b6-4221-b7b2-7ae5cb159d36","Type":"ContainerDied","Data":"c697967614cff5094a283125ee9ab25b56553bdb6e99e3af9e3b280703297328"} Oct 30 14:21:01.739066 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:01.739021 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d74808-c8z6d" podStartSLOduration=1.739009029 podStartE2EDuration="1.739009029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:21:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:21:01.737871525 +0000 UTC m=+1931.482005565" watchObservedRunningTime="2026-10-30 14:21:01.739009029 +0000 UTC m=+1931.483143069" Oct 30 14:21:01.913652 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:01.913594 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:02.380795 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.380768 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4" Oct 30 14:21:02.385353 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.385328 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw" Oct 30 14:21:02.533261 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.533229 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28-credential-socket\") pod \"2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28\" (UID: \"2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28\") " Oct 30 14:21:02.533439 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.533319 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28-istio-token\" (UniqueName: \"kubernetes.io/projected/2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28-istio-token\") pod \"2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28\" (UID: \"2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28\") " Oct 30 14:21:02.533439 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.533355 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28-workload-certs\") pod \"2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28\" (UID: \"2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28\") " Oct 30 14:21:02.533439 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.533380 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b14bb682-071a-4c22-ad2d-cab7434a4192-istio-token\" (UniqueName: \"kubernetes.io/projected/b14bb682-071a-4c22-ad2d-cab7434a4192-istio-token\") pod \"b14bb682-071a-4c22-ad2d-cab7434a4192\" (UID: \"b14bb682-071a-4c22-ad2d-cab7434a4192\") " Oct 30 14:21:02.533439 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.533437 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28-workload-socket\") pod \"2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28\" (UID: \"2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28\") " Oct 30 14:21:02.533671 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.533467 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b14bb682-071a-4c22-ad2d-cab7434a4192-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b14bb682-071a-4c22-ad2d-cab7434a4192-istio-podinfo\") pod \"b14bb682-071a-4c22-ad2d-cab7434a4192\" (UID: \"b14bb682-071a-4c22-ad2d-cab7434a4192\") " Oct 30 14:21:02.533671 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.533497 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28-istio-data\") pod \"2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28\" (UID: \"2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28\") " Oct 30 14:21:02.533671 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.533606 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b14bb682-071a-4c22-ad2d-cab7434a4192-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b14bb682-071a-4c22-ad2d-cab7434a4192-istio-envoy\") pod \"b14bb682-071a-4c22-ad2d-cab7434a4192\" (UID: \"b14bb682-071a-4c22-ad2d-cab7434a4192\") " Oct 30 14:21:02.533671 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.533608 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28-credential-socket" pod "2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28" (UID: "2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:21:02.533866 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.533705 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28-istio-podinfo\") pod \"2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28\" (UID: \"2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28\") " Oct 30 14:21:02.533866 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.533770 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b14bb682-071a-4c22-ad2d-cab7434a4192-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b14bb682-071a-4c22-ad2d-cab7434a4192-credential-socket\") pod \"b14bb682-071a-4c22-ad2d-cab7434a4192\" (UID: \"b14bb682-071a-4c22-ad2d-cab7434a4192\") " Oct 30 14:21:02.533866 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.533799 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b14bb682-071a-4c22-ad2d-cab7434a4192-kube-api-access-9jk4t\" (UniqueName: \"kubernetes.io/projected/b14bb682-071a-4c22-ad2d-cab7434a4192-kube-api-access-9jk4t\") pod \"b14bb682-071a-4c22-ad2d-cab7434a4192\" (UID: \"b14bb682-071a-4c22-ad2d-cab7434a4192\") " Oct 30 14:21:02.533866 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.533827 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28-workload-socket" pod "2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28" (UID: "2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:21:02.533866 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.533836 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b14bb682-071a-4c22-ad2d-cab7434a4192-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b14bb682-071a-4c22-ad2d-cab7434a4192-workload-certs\") pod \"b14bb682-071a-4c22-ad2d-cab7434a4192\" (UID: \"b14bb682-071a-4c22-ad2d-cab7434a4192\") " Oct 30 14:21:02.534113 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.533918 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b14bb682-071a-4c22-ad2d-cab7434a4192-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b14bb682-071a-4c22-ad2d-cab7434a4192-workload-socket\") pod \"b14bb682-071a-4c22-ad2d-cab7434a4192\" (UID: \"b14bb682-071a-4c22-ad2d-cab7434a4192\") " Oct 30 14:21:02.534113 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.533962 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b14bb682-071a-4c22-ad2d-cab7434a4192-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b14bb682-071a-4c22-ad2d-cab7434a4192-istiod-ca-cert\") pod \"b14bb682-071a-4c22-ad2d-cab7434a4192\" (UID: \"b14bb682-071a-4c22-ad2d-cab7434a4192\") " Oct 30 14:21:02.534113 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.533994 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b14bb682-071a-4c22-ad2d-cab7434a4192-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b14bb682-071a-4c22-ad2d-cab7434a4192-istio-data\") pod \"b14bb682-071a-4c22-ad2d-cab7434a4192\" (UID: \"b14bb682-071a-4c22-ad2d-cab7434a4192\") " Oct 30 14:21:02.534113 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.534024 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28-workload-certs" pod "2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28" (UID: "2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:21:02.534113 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.534072 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28-istio-envoy\") pod \"2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28\" (UID: \"2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28\") " Oct 30 14:21:02.534348 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.534115 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28-kube-api-access-j94cr\" (UniqueName: \"kubernetes.io/projected/2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28-kube-api-access-j94cr\") pod \"2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28\" (UID: \"2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28\") " Oct 30 14:21:02.534348 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.534147 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28-istiod-ca-cert\") pod \"2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28\" (UID: \"2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28\") " Oct 30 14:21:02.535324 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.535061 4982 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28-workload-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:21:02.535324 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.535089 4982 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28-workload-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:21:02.535324 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.535105 4982 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28-credential-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:21:02.535627 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.535528 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28-istio-data" pod "2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28" (UID: "2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:21:02.536027 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.535924 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b14bb682-071a-4c22-ad2d-cab7434a4192-credential-socket" pod "b14bb682-071a-4c22-ad2d-cab7434a4192" (UID: "b14bb682-071a-4c22-ad2d-cab7434a4192"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:21:02.536146 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.535986 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28-istiod-ca-cert" pod "2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28" (UID: "2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:21:02.536381 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.536357 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b14bb682-071a-4c22-ad2d-cab7434a4192-workload-socket" pod "b14bb682-071a-4c22-ad2d-cab7434a4192" (UID: "b14bb682-071a-4c22-ad2d-cab7434a4192"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:21:02.536542 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.536430 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b14bb682-071a-4c22-ad2d-cab7434a4192-istio-data" pod "b14bb682-071a-4c22-ad2d-cab7434a4192" (UID: "b14bb682-071a-4c22-ad2d-cab7434a4192"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:21:02.537378 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.537037 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b14bb682-071a-4c22-ad2d-cab7434a4192-workload-certs" pod "b14bb682-071a-4c22-ad2d-cab7434a4192" (UID: "b14bb682-071a-4c22-ad2d-cab7434a4192"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:21:02.537378 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.537221 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b14bb682-071a-4c22-ad2d-cab7434a4192-istio-envoy" pod "b14bb682-071a-4c22-ad2d-cab7434a4192" (UID: "b14bb682-071a-4c22-ad2d-cab7434a4192"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:21:02.537378 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.537319 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28-istio-token" pod "2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28" (UID: "2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:21:02.537378 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.537348 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b14bb682-071a-4c22-ad2d-cab7434a4192-istio-token" pod "b14bb682-071a-4c22-ad2d-cab7434a4192" (UID: "b14bb682-071a-4c22-ad2d-cab7434a4192"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:21:02.537764 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.537738 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b14bb682-071a-4c22-ad2d-cab7434a4192-istiod-ca-cert" pod "b14bb682-071a-4c22-ad2d-cab7434a4192" (UID: "b14bb682-071a-4c22-ad2d-cab7434a4192"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:21:02.538147 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.538080 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28-istio-podinfo" pod "2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28" (UID: "2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:21:02.539151 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.539122 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28-istio-envoy" pod "2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28" (UID: "2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:21:02.539404 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.539375 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28-kube-api-access-j94cr" pod "2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28" (UID: "2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28"). InnerVolumeSpecName "kube-api-access-j94cr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:21:02.539721 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.539699 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b14bb682-071a-4c22-ad2d-cab7434a4192-kube-api-access-9jk4t" pod "b14bb682-071a-4c22-ad2d-cab7434a4192" (UID: "b14bb682-071a-4c22-ad2d-cab7434a4192"). InnerVolumeSpecName "kube-api-access-9jk4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:21:02.539808 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.539768 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b14bb682-071a-4c22-ad2d-cab7434a4192-istio-podinfo" pod "b14bb682-071a-4c22-ad2d-cab7434a4192" (UID: "b14bb682-071a-4c22-ad2d-cab7434a4192"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:21:02.560560 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.560534 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk" Oct 30 14:21:02.643876 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.638891 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j94cr\" (UniqueName: \"kubernetes.io/projected/2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28-kube-api-access-j94cr\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:21:02.643876 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.638914 4982 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28-istiod-ca-cert\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:21:02.643876 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.638924 4982 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28-istio-token\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:21:02.643876 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.638933 4982 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b14bb682-071a-4c22-ad2d-cab7434a4192-istio-token\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:21:02.643876 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.638944 4982 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b14bb682-071a-4c22-ad2d-cab7434a4192-istio-podinfo\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:21:02.643876 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.638956 4982 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28-istio-data\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:21:02.643876 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.638964 4982 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b14bb682-071a-4c22-ad2d-cab7434a4192-istio-envoy\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:21:02.643876 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.638972 4982 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28-istio-podinfo\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:21:02.643876 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.638979 4982 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b14bb682-071a-4c22-ad2d-cab7434a4192-credential-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:21:02.643876 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.638988 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jk4t\" (UniqueName: \"kubernetes.io/projected/b14bb682-071a-4c22-ad2d-cab7434a4192-kube-api-access-9jk4t\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:21:02.643876 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.638997 4982 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b14bb682-071a-4c22-ad2d-cab7434a4192-workload-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:21:02.643876 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.639005 4982 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b14bb682-071a-4c22-ad2d-cab7434a4192-workload-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:21:02.643876 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.639015 4982 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b14bb682-071a-4c22-ad2d-cab7434a4192-istiod-ca-cert\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:21:02.643876 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.639026 4982 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b14bb682-071a-4c22-ad2d-cab7434a4192-istio-data\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:21:02.643876 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.639039 4982 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28-istio-envoy\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:21:02.710156 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.708165 4982 generic.go:356] "Generic (PLEG): container finished" podID="b14bb682-071a-4c22-ad2d-cab7434a4192" containerID="31e08d6c89814d4efd0c371849a1e652cd204be7f967ab5a584e85e3d4e9ecd5" exitCode=0 Oct 30 14:21:02.710156 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.708308 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4" event={"ID":"b14bb682-071a-4c22-ad2d-cab7434a4192","Type":"ContainerDied","Data":"31e08d6c89814d4efd0c371849a1e652cd204be7f967ab5a584e85e3d4e9ecd5"} Oct 30 14:21:02.710156 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.708366 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4" event={"ID":"b14bb682-071a-4c22-ad2d-cab7434a4192","Type":"ContainerDied","Data":"66448dafa9336847254cbac81cfebb15676c158f26fe8d7b542cd300f017184c"} Oct 30 14:21:02.710156 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.708395 4982 scope.go:122] "RemoveContainer" containerID="31e08d6c89814d4efd0c371849a1e652cd204be7f967ab5a584e85e3d4e9ecd5" Oct 30 14:21:02.710156 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.708570 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4" Oct 30 14:21:02.713990 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.713959 4982 generic.go:356] "Generic (PLEG): container finished" podID="bff13918-6bc7-42b8-86ca-216d6baeb4f6" containerID="3dfde50d83a37bb9cac2b7f9ca55ea6070830d3d29abe559464966f1394c554a" exitCode=0 Oct 30 14:21:02.714112 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.714092 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d74808-c8z6d" event={"ID":"bff13918-6bc7-42b8-86ca-216d6baeb4f6","Type":"ContainerDied","Data":"3dfde50d83a37bb9cac2b7f9ca55ea6070830d3d29abe559464966f1394c554a"} Oct 30 14:21:02.716527 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.716503 4982 generic.go:356] "Generic (PLEG): container finished" podID="2a20886d-6388-4460-ae47-12a5a28e0eb6" containerID="ba609330d711f7655ff170ef31402eaac26bea8c9f5f1ecc33e14d6ba9d81a0a" exitCode=0 Oct 30 14:21:02.716632 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.716594 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls" event={"ID":"2a20886d-6388-4460-ae47-12a5a28e0eb6","Type":"ContainerDied","Data":"ba609330d711f7655ff170ef31402eaac26bea8c9f5f1ecc33e14d6ba9d81a0a"} Oct 30 14:21:02.719028 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.718171 4982 generic.go:356] "Generic (PLEG): container finished" podID="2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28" containerID="9e508c66aa266c2278f1dd5abb2387d3e3f4901a55d519df8f3ffceabfdd225b" exitCode=0 Oct 30 14:21:02.719028 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.718224 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw" Oct 30 14:21:02.719028 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.718323 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw" event={"ID":"2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28","Type":"ContainerDied","Data":"9e508c66aa266c2278f1dd5abb2387d3e3f4901a55d519df8f3ffceabfdd225b"} Oct 30 14:21:02.719028 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.718359 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw" event={"ID":"2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28","Type":"ContainerDied","Data":"0d1f36a956c23e7b13f7364daf3ada1d7148c12da09c49fedf4f3c614512ff90"} Oct 30 14:21:02.721560 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.720754 4982 generic.go:356] "Generic (PLEG): container finished" podID="99cbf845-7193-4e4d-8315-b5b0614cfd33" containerID="c2e6c1f8067dbeaa782c8a4627a999a6a47e555d4161a0f79cd9ee848e98e031" exitCode=0 Oct 30 14:21:02.721560 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.721024 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk" Oct 30 14:21:02.721560 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.721166 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk" event={"ID":"99cbf845-7193-4e4d-8315-b5b0614cfd33","Type":"ContainerDied","Data":"c2e6c1f8067dbeaa782c8a4627a999a6a47e555d4161a0f79cd9ee848e98e031"} Oct 30 14:21:02.721560 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.721198 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk" event={"ID":"99cbf845-7193-4e4d-8315-b5b0614cfd33","Type":"ContainerDied","Data":"f4ca797ff3c768528fc0a7a9163bc7b42968753f6b6258312dfa376e7093693e"} Oct 30 14:21:02.737764 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.737744 4982 scope.go:122] "RemoveContainer" containerID="31e08d6c89814d4efd0c371849a1e652cd204be7f967ab5a584e85e3d4e9ecd5" Oct 30 14:21:02.738081 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:21:02.738062 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31e08d6c89814d4efd0c371849a1e652cd204be7f967ab5a584e85e3d4e9ecd5\": container with ID starting with 31e08d6c89814d4efd0c371849a1e652cd204be7f967ab5a584e85e3d4e9ecd5 not found: ID does not exist" containerID="31e08d6c89814d4efd0c371849a1e652cd204be7f967ab5a584e85e3d4e9ecd5" Oct 30 14:21:02.738140 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.738089 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31e08d6c89814d4efd0c371849a1e652cd204be7f967ab5a584e85e3d4e9ecd5"} err="failed to get container status \"31e08d6c89814d4efd0c371849a1e652cd204be7f967ab5a584e85e3d4e9ecd5\": rpc error: code = NotFound desc = could not find container \"31e08d6c89814d4efd0c371849a1e652cd204be7f967ab5a584e85e3d4e9ecd5\": container with ID starting with 31e08d6c89814d4efd0c371849a1e652cd204be7f967ab5a584e85e3d4e9ecd5 not found: ID does not exist" Oct 30 14:21:02.738140 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.738105 4982 scope.go:122] "RemoveContainer" containerID="9e508c66aa266c2278f1dd5abb2387d3e3f4901a55d519df8f3ffceabfdd225b" Oct 30 14:21:02.740453 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.740431 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99cbf845-7193-4e4d-8315-b5b0614cfd33-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99cbf845-7193-4e4d-8315-b5b0614cfd33-credential-socket\") pod \"99cbf845-7193-4e4d-8315-b5b0614cfd33\" (UID: \"99cbf845-7193-4e4d-8315-b5b0614cfd33\") " Oct 30 14:21:02.740547 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.740488 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99cbf845-7193-4e4d-8315-b5b0614cfd33-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99cbf845-7193-4e4d-8315-b5b0614cfd33-workload-socket\") pod \"99cbf845-7193-4e4d-8315-b5b0614cfd33\" (UID: \"99cbf845-7193-4e4d-8315-b5b0614cfd33\") " Oct 30 14:21:02.740547 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.740536 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99cbf845-7193-4e4d-8315-b5b0614cfd33-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99cbf845-7193-4e4d-8315-b5b0614cfd33-istio-envoy\") pod \"99cbf845-7193-4e4d-8315-b5b0614cfd33\" (UID: \"99cbf845-7193-4e4d-8315-b5b0614cfd33\") " Oct 30 14:21:02.740619 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.740566 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/99cbf845-7193-4e4d-8315-b5b0614cfd33-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99cbf845-7193-4e4d-8315-b5b0614cfd33-istio-podinfo\") pod \"99cbf845-7193-4e4d-8315-b5b0614cfd33\" (UID: \"99cbf845-7193-4e4d-8315-b5b0614cfd33\") " Oct 30 14:21:02.740619 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.740596 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99cbf845-7193-4e4d-8315-b5b0614cfd33-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99cbf845-7193-4e4d-8315-b5b0614cfd33-workload-certs\") pod \"99cbf845-7193-4e4d-8315-b5b0614cfd33\" (UID: \"99cbf845-7193-4e4d-8315-b5b0614cfd33\") " Oct 30 14:21:02.740774 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.740625 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99cbf845-7193-4e4d-8315-b5b0614cfd33-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99cbf845-7193-4e4d-8315-b5b0614cfd33-istio-data\") pod \"99cbf845-7193-4e4d-8315-b5b0614cfd33\" (UID: \"99cbf845-7193-4e4d-8315-b5b0614cfd33\") " Oct 30 14:21:02.740811 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.740780 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99cbf845-7193-4e4d-8315-b5b0614cfd33-kube-api-access-htlnw\" (UniqueName: \"kubernetes.io/projected/99cbf845-7193-4e4d-8315-b5b0614cfd33-kube-api-access-htlnw\") pod \"99cbf845-7193-4e4d-8315-b5b0614cfd33\" (UID: \"99cbf845-7193-4e4d-8315-b5b0614cfd33\") " Oct 30 14:21:02.740884 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.740851 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99cbf845-7193-4e4d-8315-b5b0614cfd33-istio-token\" (UniqueName: \"kubernetes.io/projected/99cbf845-7193-4e4d-8315-b5b0614cfd33-istio-token\") pod \"99cbf845-7193-4e4d-8315-b5b0614cfd33\" (UID: \"99cbf845-7193-4e4d-8315-b5b0614cfd33\") " Oct 30 14:21:02.740931 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.740904 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99cbf845-7193-4e4d-8315-b5b0614cfd33-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99cbf845-7193-4e4d-8315-b5b0614cfd33-istiod-ca-cert\") pod \"99cbf845-7193-4e4d-8315-b5b0614cfd33\" (UID: \"99cbf845-7193-4e4d-8315-b5b0614cfd33\") " Oct 30 14:21:02.740973 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.740929 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99cbf845-7193-4e4d-8315-b5b0614cfd33-workload-socket" pod "99cbf845-7193-4e4d-8315-b5b0614cfd33" (UID: "99cbf845-7193-4e4d-8315-b5b0614cfd33"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:21:02.741032 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.741000 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99cbf845-7193-4e4d-8315-b5b0614cfd33-credential-socket" pod "99cbf845-7193-4e4d-8315-b5b0614cfd33" (UID: "99cbf845-7193-4e4d-8315-b5b0614cfd33"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:21:02.741124 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.741099 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99cbf845-7193-4e4d-8315-b5b0614cfd33-istio-data" pod "99cbf845-7193-4e4d-8315-b5b0614cfd33" (UID: "99cbf845-7193-4e4d-8315-b5b0614cfd33"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:21:02.741232 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.741213 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99cbf845-7193-4e4d-8315-b5b0614cfd33-workload-certs" pod "99cbf845-7193-4e4d-8315-b5b0614cfd33" (UID: "99cbf845-7193-4e4d-8315-b5b0614cfd33"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:21:02.741693 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.741669 4982 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99cbf845-7193-4e4d-8315-b5b0614cfd33-credential-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:21:02.741793 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.741695 4982 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99cbf845-7193-4e4d-8315-b5b0614cfd33-workload-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:21:02.741793 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.741712 4982 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99cbf845-7193-4e4d-8315-b5b0614cfd33-workload-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:21:02.741793 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.741726 4982 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99cbf845-7193-4e4d-8315-b5b0614cfd33-istio-data\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:21:02.741793 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.741749 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99cbf845-7193-4e4d-8315-b5b0614cfd33-istiod-ca-cert" pod "99cbf845-7193-4e4d-8315-b5b0614cfd33" (UID: "99cbf845-7193-4e4d-8315-b5b0614cfd33"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:21:02.755177 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.755156 4982 scope.go:122] "RemoveContainer" containerID="9e508c66aa266c2278f1dd5abb2387d3e3f4901a55d519df8f3ffceabfdd225b" Oct 30 14:21:02.755401 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.755364 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99cbf845-7193-4e4d-8315-b5b0614cfd33-istio-envoy" pod "99cbf845-7193-4e4d-8315-b5b0614cfd33" (UID: "99cbf845-7193-4e4d-8315-b5b0614cfd33"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:21:02.755476 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.755401 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99cbf845-7193-4e4d-8315-b5b0614cfd33-istio-token" pod "99cbf845-7193-4e4d-8315-b5b0614cfd33" (UID: "99cbf845-7193-4e4d-8315-b5b0614cfd33"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:21:02.755514 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.755469 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/99cbf845-7193-4e4d-8315-b5b0614cfd33-istio-podinfo" pod "99cbf845-7193-4e4d-8315-b5b0614cfd33" (UID: "99cbf845-7193-4e4d-8315-b5b0614cfd33"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:21:02.755665 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:21:02.755621 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e508c66aa266c2278f1dd5abb2387d3e3f4901a55d519df8f3ffceabfdd225b\": container with ID starting with 9e508c66aa266c2278f1dd5abb2387d3e3f4901a55d519df8f3ffceabfdd225b not found: ID does not exist" containerID="9e508c66aa266c2278f1dd5abb2387d3e3f4901a55d519df8f3ffceabfdd225b" Oct 30 14:21:02.755727 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.755662 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e508c66aa266c2278f1dd5abb2387d3e3f4901a55d519df8f3ffceabfdd225b"} err="failed to get container status \"9e508c66aa266c2278f1dd5abb2387d3e3f4901a55d519df8f3ffceabfdd225b\": rpc error: code = NotFound desc = could not find container \"9e508c66aa266c2278f1dd5abb2387d3e3f4901a55d519df8f3ffceabfdd225b\": container with ID starting with 9e508c66aa266c2278f1dd5abb2387d3e3f4901a55d519df8f3ffceabfdd225b not found: ID does not exist" Oct 30 14:21:02.755727 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.755680 4982 scope.go:122] "RemoveContainer" containerID="c2e6c1f8067dbeaa782c8a4627a999a6a47e555d4161a0f79cd9ee848e98e031" Oct 30 14:21:02.756080 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.756043 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99cbf845-7193-4e4d-8315-b5b0614cfd33-kube-api-access-htlnw" pod "99cbf845-7193-4e4d-8315-b5b0614cfd33" (UID: "99cbf845-7193-4e4d-8315-b5b0614cfd33"). InnerVolumeSpecName "kube-api-access-htlnw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:21:02.771154 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.771132 4982 scope.go:122] "RemoveContainer" containerID="c2e6c1f8067dbeaa782c8a4627a999a6a47e555d4161a0f79cd9ee848e98e031" Oct 30 14:21:02.771538 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:21:02.771509 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2e6c1f8067dbeaa782c8a4627a999a6a47e555d4161a0f79cd9ee848e98e031\": container with ID starting with c2e6c1f8067dbeaa782c8a4627a999a6a47e555d4161a0f79cd9ee848e98e031 not found: ID does not exist" containerID="c2e6c1f8067dbeaa782c8a4627a999a6a47e555d4161a0f79cd9ee848e98e031" Oct 30 14:21:02.771596 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.771554 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2e6c1f8067dbeaa782c8a4627a999a6a47e555d4161a0f79cd9ee848e98e031"} err="failed to get container status \"c2e6c1f8067dbeaa782c8a4627a999a6a47e555d4161a0f79cd9ee848e98e031\": rpc error: code = NotFound desc = could not find container \"c2e6c1f8067dbeaa782c8a4627a999a6a47e555d4161a0f79cd9ee848e98e031\": container with ID starting with c2e6c1f8067dbeaa782c8a4627a999a6a47e555d4161a0f79cd9ee848e98e031 not found: ID does not exist" Oct 30 14:21:02.784632 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.784603 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls" Oct 30 14:21:02.836960 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.836925 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4"] Oct 30 14:21:02.843615 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.843582 4982 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99cbf845-7193-4e4d-8315-b5b0614cfd33-istio-envoy\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:21:02.843615 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.843621 4982 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99cbf845-7193-4e4d-8315-b5b0614cfd33-istio-podinfo\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:21:02.843805 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.843652 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htlnw\" (UniqueName: \"kubernetes.io/projected/99cbf845-7193-4e4d-8315-b5b0614cfd33-kube-api-access-htlnw\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:21:02.843805 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.843671 4982 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99cbf845-7193-4e4d-8315-b5b0614cfd33-istio-token\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:21:02.843805 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.843686 4982 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99cbf845-7193-4e4d-8315-b5b0614cfd33-istiod-ca-cert\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:21:02.845430 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.845407 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-pndg4"] Oct 30 14:21:02.893354 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.893326 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2b206b-t2tz7" Oct 30 14:21:02.901340 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.901308 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw"] Oct 30 14:21:02.905048 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.905022 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-sx4vw"] Oct 30 14:21:02.944249 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.944222 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a20886d-6388-4460-ae47-12a5a28e0eb6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a20886d-6388-4460-ae47-12a5a28e0eb6-istiod-ca-cert\") pod \"2a20886d-6388-4460-ae47-12a5a28e0eb6\" (UID: \"2a20886d-6388-4460-ae47-12a5a28e0eb6\") " Oct 30 14:21:02.944403 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.944283 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a20886d-6388-4460-ae47-12a5a28e0eb6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a20886d-6388-4460-ae47-12a5a28e0eb6-istio-envoy\") pod \"2a20886d-6388-4460-ae47-12a5a28e0eb6\" (UID: \"2a20886d-6388-4460-ae47-12a5a28e0eb6\") " Oct 30 14:21:02.944403 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.944327 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a20886d-6388-4460-ae47-12a5a28e0eb6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a20886d-6388-4460-ae47-12a5a28e0eb6-workload-certs\") pod \"2a20886d-6388-4460-ae47-12a5a28e0eb6\" (UID: \"2a20886d-6388-4460-ae47-12a5a28e0eb6\") " Oct 30 14:21:02.944403 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.944356 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a20886d-6388-4460-ae47-12a5a28e0eb6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a20886d-6388-4460-ae47-12a5a28e0eb6-istio-data\") pod \"2a20886d-6388-4460-ae47-12a5a28e0eb6\" (UID: \"2a20886d-6388-4460-ae47-12a5a28e0eb6\") " Oct 30 14:21:02.944534 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.944453 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a20886d-6388-4460-ae47-12a5a28e0eb6-kube-api-access-j4jth\" (UniqueName: \"kubernetes.io/projected/2a20886d-6388-4460-ae47-12a5a28e0eb6-kube-api-access-j4jth\") pod \"2a20886d-6388-4460-ae47-12a5a28e0eb6\" (UID: \"2a20886d-6388-4460-ae47-12a5a28e0eb6\") " Oct 30 14:21:02.944534 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.944482 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a20886d-6388-4460-ae47-12a5a28e0eb6-istio-token\" (UniqueName: \"kubernetes.io/projected/2a20886d-6388-4460-ae47-12a5a28e0eb6-istio-token\") pod \"2a20886d-6388-4460-ae47-12a5a28e0eb6\" (UID: \"2a20886d-6388-4460-ae47-12a5a28e0eb6\") " Oct 30 14:21:02.944628 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.944532 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2a20886d-6388-4460-ae47-12a5a28e0eb6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a20886d-6388-4460-ae47-12a5a28e0eb6-istio-podinfo\") pod \"2a20886d-6388-4460-ae47-12a5a28e0eb6\" (UID: \"2a20886d-6388-4460-ae47-12a5a28e0eb6\") " Oct 30 14:21:02.944628 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.944577 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a20886d-6388-4460-ae47-12a5a28e0eb6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a20886d-6388-4460-ae47-12a5a28e0eb6-workload-socket\") pod \"2a20886d-6388-4460-ae47-12a5a28e0eb6\" (UID: \"2a20886d-6388-4460-ae47-12a5a28e0eb6\") " Oct 30 14:21:02.944764 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.944680 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a20886d-6388-4460-ae47-12a5a28e0eb6-workload-certs" pod "2a20886d-6388-4460-ae47-12a5a28e0eb6" (UID: "2a20886d-6388-4460-ae47-12a5a28e0eb6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:21:02.944764 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.944697 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a20886d-6388-4460-ae47-12a5a28e0eb6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a20886d-6388-4460-ae47-12a5a28e0eb6-credential-socket\") pod \"2a20886d-6388-4460-ae47-12a5a28e0eb6\" (UID: \"2a20886d-6388-4460-ae47-12a5a28e0eb6\") " Oct 30 14:21:02.944997 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.944940 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a20886d-6388-4460-ae47-12a5a28e0eb6-workload-socket" pod "2a20886d-6388-4460-ae47-12a5a28e0eb6" (UID: "2a20886d-6388-4460-ae47-12a5a28e0eb6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:21:02.945078 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.945007 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a20886d-6388-4460-ae47-12a5a28e0eb6-istio-data" pod "2a20886d-6388-4460-ae47-12a5a28e0eb6" (UID: "2a20886d-6388-4460-ae47-12a5a28e0eb6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:21:02.945078 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.945018 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a20886d-6388-4460-ae47-12a5a28e0eb6-istiod-ca-cert" pod "2a20886d-6388-4460-ae47-12a5a28e0eb6" (UID: "2a20886d-6388-4460-ae47-12a5a28e0eb6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:21:02.945203 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.945181 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a20886d-6388-4460-ae47-12a5a28e0eb6-credential-socket" pod "2a20886d-6388-4460-ae47-12a5a28e0eb6" (UID: "2a20886d-6388-4460-ae47-12a5a28e0eb6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:21:02.945526 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.945504 4982 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a20886d-6388-4460-ae47-12a5a28e0eb6-workload-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:21:02.945619 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.945531 4982 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a20886d-6388-4460-ae47-12a5a28e0eb6-credential-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:21:02.945619 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.945547 4982 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a20886d-6388-4460-ae47-12a5a28e0eb6-istiod-ca-cert\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:21:02.945619 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.945562 4982 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a20886d-6388-4460-ae47-12a5a28e0eb6-workload-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:21:02.945619 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.945575 4982 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a20886d-6388-4460-ae47-12a5a28e0eb6-istio-data\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:21:02.946797 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.946774 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a20886d-6388-4460-ae47-12a5a28e0eb6-istio-envoy" pod "2a20886d-6388-4460-ae47-12a5a28e0eb6" (UID: "2a20886d-6388-4460-ae47-12a5a28e0eb6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:21:02.946889 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.946814 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a20886d-6388-4460-ae47-12a5a28e0eb6-kube-api-access-j4jth" pod "2a20886d-6388-4460-ae47-12a5a28e0eb6" (UID: "2a20886d-6388-4460-ae47-12a5a28e0eb6"). InnerVolumeSpecName "kube-api-access-j4jth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:21:02.947010 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.946987 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a20886d-6388-4460-ae47-12a5a28e0eb6-istio-podinfo" pod "2a20886d-6388-4460-ae47-12a5a28e0eb6" (UID: "2a20886d-6388-4460-ae47-12a5a28e0eb6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:21:02.947105 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:02.947071 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a20886d-6388-4460-ae47-12a5a28e0eb6-istio-token" pod "2a20886d-6388-4460-ae47-12a5a28e0eb6" (UID: "2a20886d-6388-4460-ae47-12a5a28e0eb6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:21:03.046304 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:03.046275 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c897e27-a3b6-4221-b7b2-7ae5cb159d36-kube-api-access-ktssl\" (UniqueName: \"kubernetes.io/projected/6c897e27-a3b6-4221-b7b2-7ae5cb159d36-kube-api-access-ktssl\") pod \"6c897e27-a3b6-4221-b7b2-7ae5cb159d36\" (UID: \"6c897e27-a3b6-4221-b7b2-7ae5cb159d36\") " Oct 30 14:21:03.046410 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:03.046331 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c897e27-a3b6-4221-b7b2-7ae5cb159d36-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c897e27-a3b6-4221-b7b2-7ae5cb159d36-maas-api-openshift-ca\") pod \"6c897e27-a3b6-4221-b7b2-7ae5cb159d36\" (UID: \"6c897e27-a3b6-4221-b7b2-7ae5cb159d36\") " Oct 30 14:21:03.046410 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:03.046371 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c897e27-a3b6-4221-b7b2-7ae5cb159d36-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c897e27-a3b6-4221-b7b2-7ae5cb159d36-maas-api-xks-ca\") pod \"6c897e27-a3b6-4221-b7b2-7ae5cb159d36\" (UID: \"6c897e27-a3b6-4221-b7b2-7ae5cb159d36\") " Oct 30 14:21:03.046735 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:03.046704 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c897e27-a3b6-4221-b7b2-7ae5cb159d36-maas-api-openshift-ca" pod "6c897e27-a3b6-4221-b7b2-7ae5cb159d36" (UID: "6c897e27-a3b6-4221-b7b2-7ae5cb159d36"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:21:03.047130 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:03.047109 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c897e27-a3b6-4221-b7b2-7ae5cb159d36-maas-api-openshift-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:21:03.047220 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:03.047136 4982 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a20886d-6388-4460-ae47-12a5a28e0eb6-istio-envoy\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:21:03.047220 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:03.047151 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4jth\" (UniqueName: \"kubernetes.io/projected/2a20886d-6388-4460-ae47-12a5a28e0eb6-kube-api-access-j4jth\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:21:03.047220 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:03.047166 4982 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a20886d-6388-4460-ae47-12a5a28e0eb6-istio-token\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:21:03.047220 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:03.047179 4982 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a20886d-6388-4460-ae47-12a5a28e0eb6-istio-podinfo\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:21:03.048564 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:03.048533 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c897e27-a3b6-4221-b7b2-7ae5cb159d36-kube-api-access-ktssl" pod "6c897e27-a3b6-4221-b7b2-7ae5cb159d36" (UID: "6c897e27-a3b6-4221-b7b2-7ae5cb159d36"). InnerVolumeSpecName "kube-api-access-ktssl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:21:03.048930 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:03.048908 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c897e27-a3b6-4221-b7b2-7ae5cb159d36-maas-api-xks-ca" pod "6c897e27-a3b6-4221-b7b2-7ae5cb159d36" (UID: "6c897e27-a3b6-4221-b7b2-7ae5cb159d36"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:21:03.090334 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:03.090297 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk"] Oct 30 14:21:03.101289 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:03.101257 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-qpdxk"] Oct 30 14:21:03.153823 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:03.153742 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktssl\" (UniqueName: \"kubernetes.io/projected/6c897e27-a3b6-4221-b7b2-7ae5cb159d36-kube-api-access-ktssl\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:21:03.153823 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:03.153770 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c897e27-a3b6-4221-b7b2-7ae5cb159d36-maas-api-xks-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:21:03.731131 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:03.731099 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2b206b-t2tz7" Oct 30 14:21:03.731540 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:03.731096 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2b206b-t2tz7" event={"ID":"6c897e27-a3b6-4221-b7b2-7ae5cb159d36","Type":"ContainerDied","Data":"23a2db58fb0a31e57fe403ba771144e528a26654dc919687dd739587767a6106"} Oct 30 14:21:03.731540 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:03.731214 4982 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23a2db58fb0a31e57fe403ba771144e528a26654dc919687dd739587767a6106" Oct 30 14:21:03.733045 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:03.733022 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls" Oct 30 14:21:03.733045 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:03.733028 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls" event={"ID":"2a20886d-6388-4460-ae47-12a5a28e0eb6","Type":"ContainerDied","Data":"a1bf02e796c442589821ada77100a9de2010f7449918f565b46fa42ec34ba0c1"} Oct 30 14:21:03.733239 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:03.733085 4982 scope.go:122] "RemoveContainer" containerID="ba609330d711f7655ff170ef31402eaac26bea8c9f5f1ecc33e14d6ba9d81a0a" Oct 30 14:21:03.815899 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:03.815864 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls"] Oct 30 14:21:03.827281 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:03.827248 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-lw5ls"] Oct 30 14:21:03.893053 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:03.893017 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:03.907723 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:03.907697 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d74808-c8z6d" Oct 30 14:21:03.961965 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:03.961928 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bff13918-6bc7-42b8-86ca-216d6baeb4f6-kube-api-access-6zbvp\" (UniqueName: \"kubernetes.io/projected/bff13918-6bc7-42b8-86ca-216d6baeb4f6-kube-api-access-6zbvp\") pod \"bff13918-6bc7-42b8-86ca-216d6baeb4f6\" (UID: \"bff13918-6bc7-42b8-86ca-216d6baeb4f6\") " Oct 30 14:21:03.964279 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:03.962041 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bff13918-6bc7-42b8-86ca-216d6baeb4f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bff13918-6bc7-42b8-86ca-216d6baeb4f6-maas-api-xks-ca\") pod \"bff13918-6bc7-42b8-86ca-216d6baeb4f6\" (UID: \"bff13918-6bc7-42b8-86ca-216d6baeb4f6\") " Oct 30 14:21:03.968497 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:03.968464 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bff13918-6bc7-42b8-86ca-216d6baeb4f6-kube-api-access-6zbvp" pod "bff13918-6bc7-42b8-86ca-216d6baeb4f6" (UID: "bff13918-6bc7-42b8-86ca-216d6baeb4f6"). InnerVolumeSpecName "kube-api-access-6zbvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:21:03.975093 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:03.975057 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bff13918-6bc7-42b8-86ca-216d6baeb4f6-maas-api-xks-ca" pod "bff13918-6bc7-42b8-86ca-216d6baeb4f6" (UID: "bff13918-6bc7-42b8-86ca-216d6baeb4f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:21:04.063480 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:04.063450 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bff13918-6bc7-42b8-86ca-216d6baeb4f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bff13918-6bc7-42b8-86ca-216d6baeb4f6-maas-api-openshift-ca\") pod \"bff13918-6bc7-42b8-86ca-216d6baeb4f6\" (UID: \"bff13918-6bc7-42b8-86ca-216d6baeb4f6\") " Oct 30 14:21:04.063833 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:04.063808 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bff13918-6bc7-42b8-86ca-216d6baeb4f6-maas-api-openshift-ca" pod "bff13918-6bc7-42b8-86ca-216d6baeb4f6" (UID: "bff13918-6bc7-42b8-86ca-216d6baeb4f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:21:04.064109 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:04.064090 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bff13918-6bc7-42b8-86ca-216d6baeb4f6-maas-api-openshift-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:21:04.064148 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:04.064112 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zbvp\" (UniqueName: \"kubernetes.io/projected/bff13918-6bc7-42b8-86ca-216d6baeb4f6-kube-api-access-6zbvp\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:21:04.064148 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:04.064121 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bff13918-6bc7-42b8-86ca-216d6baeb4f6-maas-api-xks-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:21:04.410045 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:04.410013 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a20886d-6388-4460-ae47-12a5a28e0eb6" path="/var/lib/kubelet/pods/2a20886d-6388-4460-ae47-12a5a28e0eb6/volumes" Oct 30 14:21:04.410492 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:04.410477 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28" path="/var/lib/kubelet/pods/2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28/volumes" Oct 30 14:21:04.410924 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:04.410912 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99cbf845-7193-4e4d-8315-b5b0614cfd33" path="/var/lib/kubelet/pods/99cbf845-7193-4e4d-8315-b5b0614cfd33/volumes" Oct 30 14:21:04.411661 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:04.411629 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b14bb682-071a-4c22-ad2d-cab7434a4192" path="/var/lib/kubelet/pods/b14bb682-071a-4c22-ad2d-cab7434a4192/volumes" Oct 30 14:21:04.742539 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:04.742448 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d74808-c8z6d" Oct 30 14:21:04.742539 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:04.742462 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d74808-c8z6d" event={"ID":"bff13918-6bc7-42b8-86ca-216d6baeb4f6","Type":"ContainerDied","Data":"2d8da68392f6d096d7df82af7d75e49e12cba30a2bb742a2d266cb8fe75f43c0"} Oct 30 14:21:04.742539 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:04.742510 4982 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d8da68392f6d096d7df82af7d75e49e12cba30a2bb742a2d266cb8fe75f43c0" Oct 30 14:21:06.022909 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:06.022864 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:07.834167 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:07.834129 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:09.668793 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:09.668757 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:10.214946 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.214909 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77"] Oct 30 14:21:10.215815 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.215789 4982 cpu_manager.go:414] "containerMap: removing container" podUID="2a20886d-6388-4460-ae47-12a5a28e0eb6" containerName="istio-proxy" Oct 30 14:21:10.215815 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.215815 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a20886d-6388-4460-ae47-12a5a28e0eb6" containerName="istio-proxy" Oct 30 14:21:10.215972 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.215874 4982 cpu_manager.go:414] "containerMap: removing container" podUID="2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28" containerName="istio-proxy" Oct 30 14:21:10.215972 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.215883 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28" containerName="istio-proxy" Oct 30 14:21:10.215972 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.215895 4982 cpu_manager.go:414] "containerMap: removing container" podUID="99cbf845-7193-4e4d-8315-b5b0614cfd33" containerName="istio-proxy" Oct 30 14:21:10.215972 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.215901 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99cbf845-7193-4e4d-8315-b5b0614cfd33" containerName="istio-proxy" Oct 30 14:21:10.215972 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.215918 4982 cpu_manager.go:414] "containerMap: removing container" podUID="bff13918-6bc7-42b8-86ca-216d6baeb4f6" containerName="revoke-keys" Oct 30 14:21:10.215972 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.215923 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bff13918-6bc7-42b8-86ca-216d6baeb4f6" containerName="revoke-keys" Oct 30 14:21:10.215972 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.215938 4982 cpu_manager.go:414] "containerMap: removing container" podUID="b14bb682-071a-4c22-ad2d-cab7434a4192" containerName="istio-proxy" Oct 30 14:21:10.215972 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.215942 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b14bb682-071a-4c22-ad2d-cab7434a4192" containerName="istio-proxy" Oct 30 14:21:10.215972 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.215962 4982 cpu_manager.go:414] "containerMap: removing container" podUID="6c897e27-a3b6-4221-b7b2-7ae5cb159d36" containerName="revoke-keys" Oct 30 14:21:10.215972 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.215967 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c897e27-a3b6-4221-b7b2-7ae5cb159d36" containerName="revoke-keys" Oct 30 14:21:10.216443 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.216063 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="99cbf845-7193-4e4d-8315-b5b0614cfd33" containerName="istio-proxy" Oct 30 14:21:10.216443 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.216075 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="bff13918-6bc7-42b8-86ca-216d6baeb4f6" containerName="revoke-keys" Oct 30 14:21:10.216443 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.216086 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="b14bb682-071a-4c22-ad2d-cab7434a4192" containerName="istio-proxy" Oct 30 14:21:10.216443 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.216093 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c897e27-a3b6-4221-b7b2-7ae5cb159d36" containerName="revoke-keys" Oct 30 14:21:10.216443 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.216099 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a20886d-6388-4460-ae47-12a5a28e0eb6" containerName="istio-proxy" Oct 30 14:21:10.216443 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.216114 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f8d2c5b-3fc9-4dbd-9c17-e39101f1cf28" containerName="istio-proxy" Oct 30 14:21:10.221066 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.221043 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77" Oct 30 14:21:10.232073 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.232042 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8"] Oct 30 14:21:10.236187 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.236168 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8" Oct 30 14:21:10.240854 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.240830 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-2a2835-openshift-default-dockercfg-s5hmr\"" Oct 30 14:21:10.246077 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.246051 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77"] Oct 30 14:21:10.267071 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.262171 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8"] Oct 30 14:21:10.354535 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.354498 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/115b91e0-f78d-4edf-9f99-8a497776c921-workload-certs\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77\" (UID: \"115b91e0-f78d-4edf-9f99-8a497776c921\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77" Oct 30 14:21:10.354751 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.354540 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmfh7\" (UniqueName: \"kubernetes.io/projected/f717a7d7-7930-4224-9872-0fc167f1f391-kube-api-access-cmfh7\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8\" (UID: \"f717a7d7-7930-4224-9872-0fc167f1f391\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8" Oct 30 14:21:10.354751 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.354612 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/115b91e0-f78d-4edf-9f99-8a497776c921-istiod-ca-cert\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77\" (UID: \"115b91e0-f78d-4edf-9f99-8a497776c921\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77" Oct 30 14:21:10.354751 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.354743 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f717a7d7-7930-4224-9872-0fc167f1f391-istiod-ca-cert\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8\" (UID: \"f717a7d7-7930-4224-9872-0fc167f1f391\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8" Oct 30 14:21:10.354928 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.354773 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/115b91e0-f78d-4edf-9f99-8a497776c921-istio-token\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77\" (UID: \"115b91e0-f78d-4edf-9f99-8a497776c921\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77" Oct 30 14:21:10.354928 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.354803 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxhmp\" (UniqueName: \"kubernetes.io/projected/115b91e0-f78d-4edf-9f99-8a497776c921-kube-api-access-kxhmp\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77\" (UID: \"115b91e0-f78d-4edf-9f99-8a497776c921\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77" Oct 30 14:21:10.354928 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.354847 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f717a7d7-7930-4224-9872-0fc167f1f391-istio-data\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8\" (UID: \"f717a7d7-7930-4224-9872-0fc167f1f391\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8" Oct 30 14:21:10.355056 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.354943 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f717a7d7-7930-4224-9872-0fc167f1f391-workload-socket\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8\" (UID: \"f717a7d7-7930-4224-9872-0fc167f1f391\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8" Oct 30 14:21:10.355056 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.355009 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/115b91e0-f78d-4edf-9f99-8a497776c921-istio-data\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77\" (UID: \"115b91e0-f78d-4edf-9f99-8a497776c921\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77" Oct 30 14:21:10.355056 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.355031 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f717a7d7-7930-4224-9872-0fc167f1f391-credential-socket\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8\" (UID: \"f717a7d7-7930-4224-9872-0fc167f1f391\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8" Oct 30 14:21:10.355162 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.355065 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f717a7d7-7930-4224-9872-0fc167f1f391-workload-certs\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8\" (UID: \"f717a7d7-7930-4224-9872-0fc167f1f391\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8" Oct 30 14:21:10.355198 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.355174 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/115b91e0-f78d-4edf-9f99-8a497776c921-istio-envoy\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77\" (UID: \"115b91e0-f78d-4edf-9f99-8a497776c921\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77" Oct 30 14:21:10.355198 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.355194 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/115b91e0-f78d-4edf-9f99-8a497776c921-workload-socket\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77\" (UID: \"115b91e0-f78d-4edf-9f99-8a497776c921\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77" Oct 30 14:21:10.355302 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.355290 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f717a7d7-7930-4224-9872-0fc167f1f391-istio-envoy\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8\" (UID: \"f717a7d7-7930-4224-9872-0fc167f1f391\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8" Oct 30 14:21:10.355335 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.355317 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/115b91e0-f78d-4edf-9f99-8a497776c921-credential-socket\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77\" (UID: \"115b91e0-f78d-4edf-9f99-8a497776c921\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77" Oct 30 14:21:10.355375 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.355363 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f717a7d7-7930-4224-9872-0fc167f1f391-istio-token\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8\" (UID: \"f717a7d7-7930-4224-9872-0fc167f1f391\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8" Oct 30 14:21:10.355425 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.355413 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f717a7d7-7930-4224-9872-0fc167f1f391-istio-podinfo\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8\" (UID: \"f717a7d7-7930-4224-9872-0fc167f1f391\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8" Oct 30 14:21:10.355458 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.355449 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/115b91e0-f78d-4edf-9f99-8a497776c921-istio-podinfo\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77\" (UID: \"115b91e0-f78d-4edf-9f99-8a497776c921\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77" Oct 30 14:21:10.456320 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.456273 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f717a7d7-7930-4224-9872-0fc167f1f391-istio-envoy\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8\" (UID: \"f717a7d7-7930-4224-9872-0fc167f1f391\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8" Oct 30 14:21:10.456320 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.456321 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/115b91e0-f78d-4edf-9f99-8a497776c921-credential-socket\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77\" (UID: \"115b91e0-f78d-4edf-9f99-8a497776c921\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77" Oct 30 14:21:10.456580 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.456355 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f717a7d7-7930-4224-9872-0fc167f1f391-istio-token\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8\" (UID: \"f717a7d7-7930-4224-9872-0fc167f1f391\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8" Oct 30 14:21:10.456580 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.456396 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f717a7d7-7930-4224-9872-0fc167f1f391-istio-podinfo\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8\" (UID: \"f717a7d7-7930-4224-9872-0fc167f1f391\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8" Oct 30 14:21:10.456580 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.456440 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/115b91e0-f78d-4edf-9f99-8a497776c921-istio-podinfo\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77\" (UID: \"115b91e0-f78d-4edf-9f99-8a497776c921\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77" Oct 30 14:21:10.456580 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.456480 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/115b91e0-f78d-4edf-9f99-8a497776c921-workload-certs\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77\" (UID: \"115b91e0-f78d-4edf-9f99-8a497776c921\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77" Oct 30 14:21:10.456830 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.456612 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmfh7\" (UniqueName: \"kubernetes.io/projected/f717a7d7-7930-4224-9872-0fc167f1f391-kube-api-access-cmfh7\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8\" (UID: \"f717a7d7-7930-4224-9872-0fc167f1f391\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8" Oct 30 14:21:10.456830 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.456716 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/115b91e0-f78d-4edf-9f99-8a497776c921-istiod-ca-cert\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77\" (UID: \"115b91e0-f78d-4edf-9f99-8a497776c921\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77" Oct 30 14:21:10.456830 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.456777 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f717a7d7-7930-4224-9872-0fc167f1f391-istiod-ca-cert\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8\" (UID: \"f717a7d7-7930-4224-9872-0fc167f1f391\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8" Oct 30 14:21:10.456830 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.456807 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/115b91e0-f78d-4edf-9f99-8a497776c921-istio-token\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77\" (UID: \"115b91e0-f78d-4edf-9f99-8a497776c921\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77" Oct 30 14:21:10.456830 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.456808 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/115b91e0-f78d-4edf-9f99-8a497776c921-credential-socket\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77\" (UID: \"115b91e0-f78d-4edf-9f99-8a497776c921\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77" Oct 30 14:21:10.457054 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.456861 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/115b91e0-f78d-4edf-9f99-8a497776c921-workload-certs\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77\" (UID: \"115b91e0-f78d-4edf-9f99-8a497776c921\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77" Oct 30 14:21:10.457054 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.456917 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxhmp\" (UniqueName: \"kubernetes.io/projected/115b91e0-f78d-4edf-9f99-8a497776c921-kube-api-access-kxhmp\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77\" (UID: \"115b91e0-f78d-4edf-9f99-8a497776c921\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77" Oct 30 14:21:10.457054 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.456980 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f717a7d7-7930-4224-9872-0fc167f1f391-istio-data\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8\" (UID: \"f717a7d7-7930-4224-9872-0fc167f1f391\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8" Oct 30 14:21:10.457054 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.457030 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f717a7d7-7930-4224-9872-0fc167f1f391-workload-socket\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8\" (UID: \"f717a7d7-7930-4224-9872-0fc167f1f391\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8" Oct 30 14:21:10.457252 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.457074 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/115b91e0-f78d-4edf-9f99-8a497776c921-istio-data\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77\" (UID: \"115b91e0-f78d-4edf-9f99-8a497776c921\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77" Oct 30 14:21:10.457252 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.457098 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f717a7d7-7930-4224-9872-0fc167f1f391-credential-socket\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8\" (UID: \"f717a7d7-7930-4224-9872-0fc167f1f391\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8" Oct 30 14:21:10.457252 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.457156 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f717a7d7-7930-4224-9872-0fc167f1f391-workload-certs\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8\" (UID: \"f717a7d7-7930-4224-9872-0fc167f1f391\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8" Oct 30 14:21:10.457418 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.457256 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/115b91e0-f78d-4edf-9f99-8a497776c921-istio-envoy\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77\" (UID: \"115b91e0-f78d-4edf-9f99-8a497776c921\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77" Oct 30 14:21:10.457418 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.457286 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/115b91e0-f78d-4edf-9f99-8a497776c921-workload-socket\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77\" (UID: \"115b91e0-f78d-4edf-9f99-8a497776c921\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77" Oct 30 14:21:10.457517 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.457427 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f717a7d7-7930-4224-9872-0fc167f1f391-workload-socket\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8\" (UID: \"f717a7d7-7930-4224-9872-0fc167f1f391\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8" Oct 30 14:21:10.457571 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.457514 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f717a7d7-7930-4224-9872-0fc167f1f391-istiod-ca-cert\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8\" (UID: \"f717a7d7-7930-4224-9872-0fc167f1f391\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8" Oct 30 14:21:10.457629 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.457578 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f717a7d7-7930-4224-9872-0fc167f1f391-istio-data\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8\" (UID: \"f717a7d7-7930-4224-9872-0fc167f1f391\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8" Oct 30 14:21:10.457700 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.457660 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/115b91e0-f78d-4edf-9f99-8a497776c921-istiod-ca-cert\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77\" (UID: \"115b91e0-f78d-4edf-9f99-8a497776c921\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77" Oct 30 14:21:10.457875 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.457849 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f717a7d7-7930-4224-9872-0fc167f1f391-credential-socket\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8\" (UID: \"f717a7d7-7930-4224-9872-0fc167f1f391\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8" Oct 30 14:21:10.458017 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.457955 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/115b91e0-f78d-4edf-9f99-8a497776c921-istio-data\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77\" (UID: \"115b91e0-f78d-4edf-9f99-8a497776c921\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77" Oct 30 14:21:10.458162 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.458134 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f717a7d7-7930-4224-9872-0fc167f1f391-workload-certs\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8\" (UID: \"f717a7d7-7930-4224-9872-0fc167f1f391\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8" Oct 30 14:21:10.458245 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.458219 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/115b91e0-f78d-4edf-9f99-8a497776c921-workload-socket\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77\" (UID: \"115b91e0-f78d-4edf-9f99-8a497776c921\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77" Oct 30 14:21:10.459578 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.459551 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f717a7d7-7930-4224-9872-0fc167f1f391-istio-podinfo\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8\" (UID: \"f717a7d7-7930-4224-9872-0fc167f1f391\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8" Oct 30 14:21:10.459707 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.459551 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/115b91e0-f78d-4edf-9f99-8a497776c921-istio-podinfo\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77\" (UID: \"115b91e0-f78d-4edf-9f99-8a497776c921\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77" Oct 30 14:21:10.459707 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.459609 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f717a7d7-7930-4224-9872-0fc167f1f391-istio-envoy\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8\" (UID: \"f717a7d7-7930-4224-9872-0fc167f1f391\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8" Oct 30 14:21:10.459829 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.459788 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/115b91e0-f78d-4edf-9f99-8a497776c921-istio-envoy\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77\" (UID: \"115b91e0-f78d-4edf-9f99-8a497776c921\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77" Oct 30 14:21:10.496725 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.487358 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f717a7d7-7930-4224-9872-0fc167f1f391-istio-token\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8\" (UID: \"f717a7d7-7930-4224-9872-0fc167f1f391\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8" Oct 30 14:21:10.496725 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.487358 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/115b91e0-f78d-4edf-9f99-8a497776c921-istio-token\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77\" (UID: \"115b91e0-f78d-4edf-9f99-8a497776c921\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77" Oct 30 14:21:10.505668 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.504326 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxhmp\" (UniqueName: \"kubernetes.io/projected/115b91e0-f78d-4edf-9f99-8a497776c921-kube-api-access-kxhmp\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77\" (UID: \"115b91e0-f78d-4edf-9f99-8a497776c921\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77" Oct 30 14:21:10.509659 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.508387 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmfh7\" (UniqueName: \"kubernetes.io/projected/f717a7d7-7930-4224-9872-0fc167f1f391-kube-api-access-cmfh7\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8\" (UID: \"f717a7d7-7930-4224-9872-0fc167f1f391\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8" Oct 30 14:21:10.536463 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.536255 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77" Oct 30 14:21:10.552317 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.552271 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8" Oct 30 14:21:10.572281 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.572249 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:10.723073 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.722993 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:10.878758 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.878724 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77"] Oct 30 14:21:10.883350 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.883274 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:21:10.883447 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.883383 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:21:10.883506 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.883443 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:21:10.906415 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.905934 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8"] Oct 30 14:21:10.906415 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:21:10.906162 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf717a7d7_7930_4224_9872_0fc167f1f391.slice/crio-4d371454a8c6efca3e445853261240f49160bb6c1148cee36581fd2743fab4ab WatchSource:0}: Error finding container 4d371454a8c6efca3e445853261240f49160bb6c1148cee36581fd2743fab4ab: Status 404 returned error can't find the container with id 4d371454a8c6efca3e445853261240f49160bb6c1148cee36581fd2743fab4ab Oct 30 14:21:10.911471 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.910271 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:21:10.911471 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.910355 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:21:10.911471 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:10.910396 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:21:11.331064 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:11.331030 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2b206b-5745bc7d79-5s7b7"] Oct 30 14:21:11.469774 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:11.469743 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:11.617321 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:11.617218 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:11.799912 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:11.799870 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8" event={"ID":"f717a7d7-7930-4224-9872-0fc167f1f391","Type":"ContainerStarted","Data":"36ede143e387390f1be9ed41a11ce57b62246f4b77c98b16b2581042a83edc5d"} Oct 30 14:21:11.800408 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:11.799967 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8" event={"ID":"f717a7d7-7930-4224-9872-0fc167f1f391","Type":"ContainerStarted","Data":"4d371454a8c6efca3e445853261240f49160bb6c1148cee36581fd2743fab4ab"} Oct 30 14:21:11.801703 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:11.801671 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77" event={"ID":"115b91e0-f78d-4edf-9f99-8a497776c921","Type":"ContainerStarted","Data":"d3732b1e3b5ccc4c94f6fb8efbfa4e7e7593c6f5b38999eb6d9195836398d286"} Oct 30 14:21:11.801867 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:11.801726 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77" event={"ID":"115b91e0-f78d-4edf-9f99-8a497776c921","Type":"ContainerStarted","Data":"c67cffb7fdc00e7cf803200b7aeaa47ce7dbaa1bbf8a9b83eea5779839ff38c8"} Oct 30 14:21:11.834744 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:11.834708 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:11.851068 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:11.851022 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8" podStartSLOduration=1.851007979 podStartE2EDuration="1.851007979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:21:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:21:11.849040558 +0000 UTC m=+1941.593174598" watchObservedRunningTime="2026-10-30 14:21:11.851007979 +0000 UTC m=+1941.595142020" Oct 30 14:21:11.900910 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:11.900361 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77" podStartSLOduration=1.9003388129999998 podStartE2EDuration="1.900338813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:21:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:21:11.896520699 +0000 UTC m=+1941.640654740" watchObservedRunningTime="2026-10-30 14:21:11.900338813 +0000 UTC m=+1941.644472857" Oct 30 14:21:12.544709 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:12.543067 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77" Oct 30 14:21:12.550191 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:12.549707 4982 patch_prober.go:29] interesting pod/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 14:21:12.550191 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:12.549772 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77" podUID="115b91e0-f78d-4edf-9f99-8a497776c921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 14:21:12.553760 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:12.553729 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8" Oct 30 14:21:12.554239 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:12.554218 4982 patch_prober.go:29] interesting pod/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 14:21:12.554292 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:12.554257 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8" podUID="f717a7d7-7930-4224-9872-0fc167f1f391" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 14:21:13.547034 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:13.546998 4982 patch_prober.go:29] interesting pod/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 14:21:13.547653 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:13.547605 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77" podUID="115b91e0-f78d-4edf-9f99-8a497776c921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 14:21:13.553481 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:13.553458 4982 patch_prober.go:29] interesting pod/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 14:21:13.553629 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:13.553613 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8" podUID="f717a7d7-7930-4224-9872-0fc167f1f391" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 14:21:13.805730 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:13.801934 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:14.545829 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:14.545788 4982 patch_prober.go:29] interesting pod/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 14:21:14.546036 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:14.545852 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77" podUID="115b91e0-f78d-4edf-9f99-8a497776c921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 14:21:14.553293 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:14.553258 4982 patch_prober.go:29] interesting pod/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 14:21:14.553635 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:14.553314 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8" podUID="f717a7d7-7930-4224-9872-0fc167f1f391" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 14:21:15.541664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:15.540785 4982 patch_prober.go:29] interesting pod/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 14:21:15.541664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:15.540847 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77" podUID="115b91e0-f78d-4edf-9f99-8a497776c921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 14:21:15.553653 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:15.553610 4982 patch_prober.go:29] interesting pod/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 14:21:15.554020 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:15.553710 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8" podUID="f717a7d7-7930-4224-9872-0fc167f1f391" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 14:21:15.741627 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:15.741593 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:16.336511 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:16.335722 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2b206b-5745bc7d79-5s7b7" podUID="f40cc27f-7493-4296-9ed2-c2183be0059f" containerName="payload-pre-processing" containerID="cri-o://80bde65d39d326d6889d5fe9b071ab54c98d17bec580bfd0350fc6a6cd4e2365" gracePeriod=25 Oct 30 14:21:16.543665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:16.540543 4982 patch_prober.go:29] interesting pod/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 14:21:16.543665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:16.540608 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77" podUID="115b91e0-f78d-4edf-9f99-8a497776c921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 14:21:16.553253 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:16.553222 4982 patch_prober.go:29] interesting pod/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 14:21:16.553385 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:16.553269 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8" podUID="f717a7d7-7930-4224-9872-0fc167f1f391" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 14:21:16.857564 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:16.857234 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2b206b-5745bc7d79-5s7b7" Oct 30 14:21:16.877570 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:16.877497 4982 generic.go:356] "Generic (PLEG): container finished" podID="f40cc27f-7493-4296-9ed2-c2183be0059f" containerID="80bde65d39d326d6889d5fe9b071ab54c98d17bec580bfd0350fc6a6cd4e2365" exitCode=0 Oct 30 14:21:16.877742 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:16.877731 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2b206b-5745bc7d79-5s7b7" event={"ID":"f40cc27f-7493-4296-9ed2-c2183be0059f","Type":"ContainerDied","Data":"80bde65d39d326d6889d5fe9b071ab54c98d17bec580bfd0350fc6a6cd4e2365"} Oct 30 14:21:16.877830 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:16.877771 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2b206b-5745bc7d79-5s7b7" event={"ID":"f40cc27f-7493-4296-9ed2-c2183be0059f","Type":"ContainerDied","Data":"7487df9434d2854031e8e30a8448ab917dd917d0e656500a07f89e555d0ebfe8"} Oct 30 14:21:16.877830 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:16.877793 4982 scope.go:122] "RemoveContainer" containerID="80bde65d39d326d6889d5fe9b071ab54c98d17bec580bfd0350fc6a6cd4e2365" Oct 30 14:21:16.877948 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:16.877907 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2b206b-5745bc7d79-5s7b7" Oct 30 14:21:16.957679 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:16.951688 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f40cc27f-7493-4296-9ed2-c2183be0059f-kube-api-access-nrdqp\" (UniqueName: \"kubernetes.io/projected/f40cc27f-7493-4296-9ed2-c2183be0059f-kube-api-access-nrdqp\") pod \"f40cc27f-7493-4296-9ed2-c2183be0059f\" (UID: \"f40cc27f-7493-4296-9ed2-c2183be0059f\") " Oct 30 14:21:16.957679 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:16.951993 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f40cc27f-7493-4296-9ed2-c2183be0059f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f40cc27f-7493-4296-9ed2-c2183be0059f-plugins-config-volume\") pod \"f40cc27f-7493-4296-9ed2-c2183be0059f\" (UID: \"f40cc27f-7493-4296-9ed2-c2183be0059f\") " Oct 30 14:21:16.957679 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:16.953412 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f40cc27f-7493-4296-9ed2-c2183be0059f-plugins-config-volume" pod "f40cc27f-7493-4296-9ed2-c2183be0059f" (UID: "f40cc27f-7493-4296-9ed2-c2183be0059f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:21:16.958216 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:16.958190 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f40cc27f-7493-4296-9ed2-c2183be0059f-kube-api-access-nrdqp" pod "f40cc27f-7493-4296-9ed2-c2183be0059f" (UID: "f40cc27f-7493-4296-9ed2-c2183be0059f"). InnerVolumeSpecName "kube-api-access-nrdqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:21:16.975145 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:16.975123 4982 scope.go:122] "RemoveContainer" containerID="80bde65d39d326d6889d5fe9b071ab54c98d17bec580bfd0350fc6a6cd4e2365" Oct 30 14:21:16.975505 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:21:16.975477 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80bde65d39d326d6889d5fe9b071ab54c98d17bec580bfd0350fc6a6cd4e2365\": container with ID starting with 80bde65d39d326d6889d5fe9b071ab54c98d17bec580bfd0350fc6a6cd4e2365 not found: ID does not exist" containerID="80bde65d39d326d6889d5fe9b071ab54c98d17bec580bfd0350fc6a6cd4e2365" Oct 30 14:21:16.975559 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:16.975517 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80bde65d39d326d6889d5fe9b071ab54c98d17bec580bfd0350fc6a6cd4e2365"} err="failed to get container status \"80bde65d39d326d6889d5fe9b071ab54c98d17bec580bfd0350fc6a6cd4e2365\": rpc error: code = NotFound desc = could not find container \"80bde65d39d326d6889d5fe9b071ab54c98d17bec580bfd0350fc6a6cd4e2365\": container with ID starting with 80bde65d39d326d6889d5fe9b071ab54c98d17bec580bfd0350fc6a6cd4e2365 not found: ID does not exist" Oct 30 14:21:17.054043 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:17.054010 4982 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f40cc27f-7493-4296-9ed2-c2183be0059f-plugins-config-volume\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:21:17.054043 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:17.054038 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrdqp\" (UniqueName: \"kubernetes.io/projected/f40cc27f-7493-4296-9ed2-c2183be0059f-kube-api-access-nrdqp\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:21:17.340807 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:17.340700 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2b206b-5745bc7d79-5s7b7"] Oct 30 14:21:17.345005 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:17.344577 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2b206b-5745bc7d79-5s7b7"] Oct 30 14:21:17.568661 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:17.558130 4982 patch_prober.go:29] interesting pod/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 14:21:17.568661 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:17.558190 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8" podUID="f717a7d7-7930-4224-9872-0fc167f1f391" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 14:21:17.568661 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:17.558709 4982 patch_prober.go:29] interesting pod/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 14:21:17.568661 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:17.558744 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77" podUID="115b91e0-f78d-4edf-9f99-8a497776c921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 14:21:17.895403 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:17.895110 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:18.406010 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:18.405960 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f40cc27f-7493-4296-9ed2-c2183be0059f" path="/var/lib/kubelet/pods/f40cc27f-7493-4296-9ed2-c2183be0059f/volumes" Oct 30 14:21:18.542274 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:18.542242 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77" Oct 30 14:21:18.543488 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:18.543466 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77" Oct 30 14:21:18.543770 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:18.543753 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77" Oct 30 14:21:18.558326 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:18.558306 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8" Oct 30 14:21:18.558587 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:18.558561 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8" Oct 30 14:21:18.559291 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:18.559274 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8" Oct 30 14:21:19.522578 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:19.522544 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:19.731662 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:19.724597 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:20.348101 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:20.348063 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:20.471262 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:20.471231 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d727dc-plmx7"] Oct 30 14:21:20.484242 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:20.484211 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d727dc-plmx7"] Oct 30 14:21:20.500211 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:20.500175 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:20.984827 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:20.984793 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:22.074554 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:22.074519 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:22.401483 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:22.401401 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3aebc69a-cfda-4a73-99f8-4a9ecbdcafb3" path="/var/lib/kubelet/pods/3aebc69a-cfda-4a73-99f8-4a9ecbdcafb3/volumes" Oct 30 14:21:22.484635 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:22.484602 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:23.148563 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:23.148533 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:23.626274 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:23.626234 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:24.097987 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:24.097929 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:24.310906 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:24.310870 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:24.508727 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:24.508599 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:24.821567 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:24.821535 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:25.481603 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:25.481552 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:26.505041 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:26.505003 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:27.556174 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:27.556142 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:27.775865 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:27.775834 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:28.011596 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:21:28.011562 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:21:28.042400 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:28.042367 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcb32943-75vlb"] Oct 30 14:21:28.055483 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:28.055456 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcb32943-75vlb"] Oct 30 14:21:28.237080 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:28.237049 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:28.435662 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:28.435446 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d77678e-4328-4e2e-8dca-19d17111b3cb" path="/var/lib/kubelet/pods/1d77678e-4328-4e2e-8dca-19d17111b3cb/volumes" Oct 30 14:21:28.849252 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:28.849216 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:29.099564 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:29.099451 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:29.265076 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:29.265035 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:29.434943 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:29.434848 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:29.579364 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:29.579330 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-972fee-4k5ct"] Oct 30 14:21:29.583123 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:29.583085 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:29.591251 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:29.591215 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-972fee-4k5ct"] Oct 30 14:21:29.713431 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:29.713358 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ca02c9-7fc799d5c8-pkcxp"] Oct 30 14:21:29.714325 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:29.714302 4982 cpu_manager.go:414] "containerMap: removing container" podUID="f40cc27f-7493-4296-9ed2-c2183be0059f" containerName="payload-pre-processing" Oct 30 14:21:29.714367 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:29.714329 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f40cc27f-7493-4296-9ed2-c2183be0059f" containerName="payload-pre-processing" Oct 30 14:21:29.714520 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:29.714507 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="f40cc27f-7493-4296-9ed2-c2183be0059f" containerName="payload-pre-processing" Oct 30 14:21:29.719621 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:29.719595 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ca02c9-7fc799d5c8-pkcxp" Oct 30 14:21:29.759223 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:29.759185 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ca02c9-7fc799d5c8-pkcxp"] Oct 30 14:21:29.851897 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:29.851866 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-ca02c9-dockercfg-p2hr9\"" Oct 30 14:21:29.892942 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:29.861125 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7thm4\" (UniqueName: \"kubernetes.io/projected/8c680f8b-331a-4286-a587-300ca1429de7-kube-api-access-7thm4\") pod \"payload-processing-e2e-inf-b-ca02c9-7fc799d5c8-pkcxp\" (UID: \"8c680f8b-331a-4286-a587-300ca1429de7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ca02c9-7fc799d5c8-pkcxp" Oct 30 14:21:29.915165 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:29.862487 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:29.922573 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:29.863331 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-ca02c9\"" Oct 30 14:21:29.922573 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:29.919291 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c680f8b-331a-4286-a587-300ca1429de7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ca02c9-7fc799d5c8-pkcxp\" (UID: \"8c680f8b-331a-4286-a587-300ca1429de7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ca02c9-7fc799d5c8-pkcxp" Oct 30 14:21:29.980622 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:29.980534 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b5ea88-vt6hg"] Oct 30 14:21:30.035836 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:30.024413 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c680f8b-331a-4286-a587-300ca1429de7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ca02c9-7fc799d5c8-pkcxp\" (UID: \"8c680f8b-331a-4286-a587-300ca1429de7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ca02c9-7fc799d5c8-pkcxp" Oct 30 14:21:30.035836 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:30.024473 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7thm4\" (UniqueName: \"kubernetes.io/projected/8c680f8b-331a-4286-a587-300ca1429de7-kube-api-access-7thm4\") pod \"payload-processing-e2e-inf-b-ca02c9-7fc799d5c8-pkcxp\" (UID: \"8c680f8b-331a-4286-a587-300ca1429de7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ca02c9-7fc799d5c8-pkcxp" Oct 30 14:21:30.035836 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:30.027936 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c680f8b-331a-4286-a587-300ca1429de7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ca02c9-7fc799d5c8-pkcxp\" (UID: \"8c680f8b-331a-4286-a587-300ca1429de7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ca02c9-7fc799d5c8-pkcxp" Oct 30 14:21:30.040707 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:30.039538 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b5ea88-vt6hg"] Oct 30 14:21:30.040707 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:30.039691 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b5ea88-vt6hg" Oct 30 14:21:30.058944 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:30.058917 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:21:30.059400 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:30.059380 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:21:30.061930 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:30.061850 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7thm4\" (UniqueName: \"kubernetes.io/projected/8c680f8b-331a-4286-a587-300ca1429de7-kube-api-access-7thm4\") pod \"payload-processing-e2e-inf-b-ca02c9-7fc799d5c8-pkcxp\" (UID: \"8c680f8b-331a-4286-a587-300ca1429de7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ca02c9-7fc799d5c8-pkcxp" Oct 30 14:21:30.066404 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:30.066369 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:30.137713 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:30.137666 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2369048-24d1-4882-bb7e-a3122a7a85bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b5ea88-vt6hg\" (UID: \"b2369048-24d1-4882-bb7e-a3122a7a85bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b5ea88-vt6hg" Oct 30 14:21:30.137912 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:30.137739 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5fjd\" (UniqueName: \"kubernetes.io/projected/b2369048-24d1-4882-bb7e-a3122a7a85bc-kube-api-access-c5fjd\") pod \"maas-api-revoke-keys-e2e-api-b-b5ea88-vt6hg\" (UID: \"b2369048-24d1-4882-bb7e-a3122a7a85bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b5ea88-vt6hg" Oct 30 14:21:30.137960 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:30.137905 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2369048-24d1-4882-bb7e-a3122a7a85bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b5ea88-vt6hg\" (UID: \"b2369048-24d1-4882-bb7e-a3122a7a85bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b5ea88-vt6hg" Oct 30 14:21:30.239686 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:30.239583 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2369048-24d1-4882-bb7e-a3122a7a85bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b5ea88-vt6hg\" (UID: \"b2369048-24d1-4882-bb7e-a3122a7a85bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b5ea88-vt6hg" Oct 30 14:21:30.239686 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:30.239680 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5fjd\" (UniqueName: \"kubernetes.io/projected/b2369048-24d1-4882-bb7e-a3122a7a85bc-kube-api-access-c5fjd\") pod \"maas-api-revoke-keys-e2e-api-b-b5ea88-vt6hg\" (UID: \"b2369048-24d1-4882-bb7e-a3122a7a85bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b5ea88-vt6hg" Oct 30 14:21:30.239947 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:30.239867 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2369048-24d1-4882-bb7e-a3122a7a85bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b5ea88-vt6hg\" (UID: \"b2369048-24d1-4882-bb7e-a3122a7a85bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b5ea88-vt6hg" Oct 30 14:21:30.240313 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:30.240285 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2369048-24d1-4882-bb7e-a3122a7a85bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b5ea88-vt6hg\" (UID: \"b2369048-24d1-4882-bb7e-a3122a7a85bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b5ea88-vt6hg" Oct 30 14:21:30.242129 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:30.242112 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2369048-24d1-4882-bb7e-a3122a7a85bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b5ea88-vt6hg\" (UID: \"b2369048-24d1-4882-bb7e-a3122a7a85bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b5ea88-vt6hg" Oct 30 14:21:30.265997 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:30.265967 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5fjd\" (UniqueName: \"kubernetes.io/projected/b2369048-24d1-4882-bb7e-a3122a7a85bc-kube-api-access-c5fjd\") pod \"maas-api-revoke-keys-e2e-api-b-b5ea88-vt6hg\" (UID: \"b2369048-24d1-4882-bb7e-a3122a7a85bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b5ea88-vt6hg" Oct 30 14:21:30.335763 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:30.335728 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ca02c9-7fc799d5c8-pkcxp" Oct 30 14:21:30.363782 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:30.363754 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b5ea88-vt6hg" Oct 30 14:21:30.402047 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:30.402015 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c518067a-74aa-4008-992b-83608da423f7" path="/var/lib/kubelet/pods/c518067a-74aa-4008-992b-83608da423f7/volumes" Oct 30 14:21:30.647065 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:30.646972 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ca02c9-7fc799d5c8-pkcxp"] Oct 30 14:21:30.702716 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:30.702691 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b5ea88-vt6hg"] Oct 30 14:21:30.703828 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:21:30.703801 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2369048_24d1_4882_bb7e_a3122a7a85bc.slice/crio-13af1a403fdf5a6ea489b1cce653a30cba5a11820061a68d29d96977d379ab60 WatchSource:0}: Error finding container 13af1a403fdf5a6ea489b1cce653a30cba5a11820061a68d29d96977d379ab60: Status 404 returned error can't find the container with id 13af1a403fdf5a6ea489b1cce653a30cba5a11820061a68d29d96977d379ab60 Oct 30 14:21:31.079275 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:31.079238 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ca02c9-7fc799d5c8-pkcxp" event={"ID":"8c680f8b-331a-4286-a587-300ca1429de7","Type":"ContainerStarted","Data":"65e61c20034e94e091691c72a2c81c5e22644ea4f70d9bc2f8bb390249ae8deb"} Oct 30 14:21:31.079738 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:31.079293 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ca02c9-7fc799d5c8-pkcxp" event={"ID":"8c680f8b-331a-4286-a587-300ca1429de7","Type":"ContainerStarted","Data":"31d31bb49f0ebcd85b2c2a1d9a344ea91e8b34d57f4cf0f65deee7abe1ff3469"} Oct 30 14:21:31.079738 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:31.079320 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-ca02c9-7fc799d5c8-pkcxp" Oct 30 14:21:31.080983 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:31.080957 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b5ea88-vt6hg" event={"ID":"b2369048-24d1-4882-bb7e-a3122a7a85bc","Type":"ContainerStarted","Data":"3a250ab6a9dd0472ffc9fc0969548993323975db47ccb7d10c1c2135f21c92cf"} Oct 30 14:21:31.081069 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:31.081004 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b5ea88-vt6hg" event={"ID":"b2369048-24d1-4882-bb7e-a3122a7a85bc","Type":"ContainerStarted","Data":"13af1a403fdf5a6ea489b1cce653a30cba5a11820061a68d29d96977d379ab60"} Oct 30 14:21:31.205960 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:31.205864 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b5ea88-vt6hg" podStartSLOduration=2.205850683 podStartE2EDuration="2.205850683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:21:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:21:31.203759751 +0000 UTC m=+1960.947893792" watchObservedRunningTime="2026-10-30 14:21:31.205850683 +0000 UTC m=+1960.949984723" Oct 30 14:21:31.206107 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:31.206061 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-ca02c9-7fc799d5c8-pkcxp" podStartSLOduration=2.206056548 podStartE2EDuration="2.206056548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:21:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:21:31.149719844 +0000 UTC m=+1960.893853884" watchObservedRunningTime="2026-10-30 14:21:31.206056548 +0000 UTC m=+1960.950190588" Oct 30 14:21:31.237440 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:31.237408 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:31.397544 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:31.397508 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:32.089538 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:32.089505 4982 generic.go:356] "Generic (PLEG): container finished" podID="b2369048-24d1-4882-bb7e-a3122a7a85bc" containerID="3a250ab6a9dd0472ffc9fc0969548993323975db47ccb7d10c1c2135f21c92cf" exitCode=0 Oct 30 14:21:32.089973 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:32.089590 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b5ea88-vt6hg" event={"ID":"b2369048-24d1-4882-bb7e-a3122a7a85bc","Type":"ContainerDied","Data":"3a250ab6a9dd0472ffc9fc0969548993323975db47ccb7d10c1c2135f21c92cf"} Oct 30 14:21:32.216585 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:32.216551 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:32.364838 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:32.364741 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:32.507955 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:32.507914 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg"] Oct 30 14:21:32.508217 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:32.508175 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg" podUID="7057acc6-78aa-43fa-bc57-6e5ecacf2e50" containerName="istio-proxy" containerID="cri-o://6a68340dffb035a63f91af47e2f3bbcba54e742ee429f812b080b6162d8b73ac" gracePeriod=30 Oct 30 14:21:32.574299 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:32.574265 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:32.821560 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:32.821524 4982 patch_prober.go:29] interesting pod/e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 14:21:32.821771 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:32.821587 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg" podUID="7057acc6-78aa-43fa-bc57-6e5ecacf2e50" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 14:21:33.274665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:33.274618 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b5ea88-vt6hg" Oct 30 14:21:33.309614 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:33.309575 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2369048-24d1-4882-bb7e-a3122a7a85bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2369048-24d1-4882-bb7e-a3122a7a85bc-maas-api-openshift-ca\") pod \"b2369048-24d1-4882-bb7e-a3122a7a85bc\" (UID: \"b2369048-24d1-4882-bb7e-a3122a7a85bc\") " Oct 30 14:21:33.309824 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:33.309733 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2369048-24d1-4882-bb7e-a3122a7a85bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2369048-24d1-4882-bb7e-a3122a7a85bc-maas-api-xks-ca\") pod \"b2369048-24d1-4882-bb7e-a3122a7a85bc\" (UID: \"b2369048-24d1-4882-bb7e-a3122a7a85bc\") " Oct 30 14:21:33.309824 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:33.309782 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2369048-24d1-4882-bb7e-a3122a7a85bc-kube-api-access-c5fjd\" (UniqueName: \"kubernetes.io/projected/b2369048-24d1-4882-bb7e-a3122a7a85bc-kube-api-access-c5fjd\") pod \"b2369048-24d1-4882-bb7e-a3122a7a85bc\" (UID: \"b2369048-24d1-4882-bb7e-a3122a7a85bc\") " Oct 30 14:21:33.310037 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:33.310004 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2369048-24d1-4882-bb7e-a3122a7a85bc-maas-api-openshift-ca" pod "b2369048-24d1-4882-bb7e-a3122a7a85bc" (UID: "b2369048-24d1-4882-bb7e-a3122a7a85bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:21:33.310410 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:33.310384 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2369048-24d1-4882-bb7e-a3122a7a85bc-maas-api-openshift-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:21:33.312416 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:33.312386 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2369048-24d1-4882-bb7e-a3122a7a85bc-maas-api-xks-ca" pod "b2369048-24d1-4882-bb7e-a3122a7a85bc" (UID: "b2369048-24d1-4882-bb7e-a3122a7a85bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:21:33.312546 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:33.312505 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2369048-24d1-4882-bb7e-a3122a7a85bc-kube-api-access-c5fjd" pod "b2369048-24d1-4882-bb7e-a3122a7a85bc" (UID: "b2369048-24d1-4882-bb7e-a3122a7a85bc"). InnerVolumeSpecName "kube-api-access-c5fjd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:21:33.412709 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:33.412545 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2369048-24d1-4882-bb7e-a3122a7a85bc-maas-api-xks-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:21:33.412709 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:33.412582 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5fjd\" (UniqueName: \"kubernetes.io/projected/b2369048-24d1-4882-bb7e-a3122a7a85bc-kube-api-access-c5fjd\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:21:33.616693 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:33.616156 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:34.111717 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:34.111685 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b5ea88-vt6hg" Oct 30 14:21:34.111919 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:34.111746 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b5ea88-vt6hg" event={"ID":"b2369048-24d1-4882-bb7e-a3122a7a85bc","Type":"ContainerDied","Data":"13af1a403fdf5a6ea489b1cce653a30cba5a11820061a68d29d96977d379ab60"} Oct 30 14:21:34.111919 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:34.111823 4982 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13af1a403fdf5a6ea489b1cce653a30cba5a11820061a68d29d96977d379ab60" Oct 30 14:21:35.548175 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:35.548117 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:35.977037 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:35.976990 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:36.141665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:36.134457 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:37.385227 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:37.385185 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:37.534074 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:37.534037 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:37.834932 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:37.834894 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:37.849775 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:37.849753 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg" Oct 30 14:21:37.917372 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:37.917329 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7057acc6-78aa-43fa-bc57-6e5ecacf2e50-kube-api-access-wcxpd\" (UniqueName: \"kubernetes.io/projected/7057acc6-78aa-43fa-bc57-6e5ecacf2e50-kube-api-access-wcxpd\") pod \"7057acc6-78aa-43fa-bc57-6e5ecacf2e50\" (UID: \"7057acc6-78aa-43fa-bc57-6e5ecacf2e50\") " Oct 30 14:21:37.917552 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:37.917426 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7057acc6-78aa-43fa-bc57-6e5ecacf2e50-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7057acc6-78aa-43fa-bc57-6e5ecacf2e50-workload-certs\") pod \"7057acc6-78aa-43fa-bc57-6e5ecacf2e50\" (UID: \"7057acc6-78aa-43fa-bc57-6e5ecacf2e50\") " Oct 30 14:21:37.917552 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:37.917494 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7057acc6-78aa-43fa-bc57-6e5ecacf2e50-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7057acc6-78aa-43fa-bc57-6e5ecacf2e50-istio-podinfo\") pod \"7057acc6-78aa-43fa-bc57-6e5ecacf2e50\" (UID: \"7057acc6-78aa-43fa-bc57-6e5ecacf2e50\") " Oct 30 14:21:37.917702 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:37.917590 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7057acc6-78aa-43fa-bc57-6e5ecacf2e50-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7057acc6-78aa-43fa-bc57-6e5ecacf2e50-istio-data\") pod \"7057acc6-78aa-43fa-bc57-6e5ecacf2e50\" (UID: \"7057acc6-78aa-43fa-bc57-6e5ecacf2e50\") " Oct 30 14:21:37.917702 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:37.917694 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7057acc6-78aa-43fa-bc57-6e5ecacf2e50-istio-token\" (UniqueName: \"kubernetes.io/projected/7057acc6-78aa-43fa-bc57-6e5ecacf2e50-istio-token\") pod \"7057acc6-78aa-43fa-bc57-6e5ecacf2e50\" (UID: \"7057acc6-78aa-43fa-bc57-6e5ecacf2e50\") " Oct 30 14:21:37.917818 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:37.917751 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7057acc6-78aa-43fa-bc57-6e5ecacf2e50-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7057acc6-78aa-43fa-bc57-6e5ecacf2e50-workload-socket\") pod \"7057acc6-78aa-43fa-bc57-6e5ecacf2e50\" (UID: \"7057acc6-78aa-43fa-bc57-6e5ecacf2e50\") " Oct 30 14:21:37.917875 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:37.917856 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7057acc6-78aa-43fa-bc57-6e5ecacf2e50-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7057acc6-78aa-43fa-bc57-6e5ecacf2e50-credential-socket\") pod \"7057acc6-78aa-43fa-bc57-6e5ecacf2e50\" (UID: \"7057acc6-78aa-43fa-bc57-6e5ecacf2e50\") " Oct 30 14:21:37.917924 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:37.917910 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7057acc6-78aa-43fa-bc57-6e5ecacf2e50-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7057acc6-78aa-43fa-bc57-6e5ecacf2e50-istiod-ca-cert\") pod \"7057acc6-78aa-43fa-bc57-6e5ecacf2e50\" (UID: \"7057acc6-78aa-43fa-bc57-6e5ecacf2e50\") " Oct 30 14:21:37.918033 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:37.918014 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7057acc6-78aa-43fa-bc57-6e5ecacf2e50-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7057acc6-78aa-43fa-bc57-6e5ecacf2e50-istio-envoy\") pod \"7057acc6-78aa-43fa-bc57-6e5ecacf2e50\" (UID: \"7057acc6-78aa-43fa-bc57-6e5ecacf2e50\") " Oct 30 14:21:37.920613 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:37.920577 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7057acc6-78aa-43fa-bc57-6e5ecacf2e50-istio-data" pod "7057acc6-78aa-43fa-bc57-6e5ecacf2e50" (UID: "7057acc6-78aa-43fa-bc57-6e5ecacf2e50"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:21:37.926221 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:37.926197 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7057acc6-78aa-43fa-bc57-6e5ecacf2e50-istio-envoy" pod "7057acc6-78aa-43fa-bc57-6e5ecacf2e50" (UID: "7057acc6-78aa-43fa-bc57-6e5ecacf2e50"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:21:37.926392 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:37.926241 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7057acc6-78aa-43fa-bc57-6e5ecacf2e50-kube-api-access-wcxpd" pod "7057acc6-78aa-43fa-bc57-6e5ecacf2e50" (UID: "7057acc6-78aa-43fa-bc57-6e5ecacf2e50"). InnerVolumeSpecName "kube-api-access-wcxpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:21:37.926494 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:37.926431 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7057acc6-78aa-43fa-bc57-6e5ecacf2e50-workload-certs" pod "7057acc6-78aa-43fa-bc57-6e5ecacf2e50" (UID: "7057acc6-78aa-43fa-bc57-6e5ecacf2e50"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:21:37.932692 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:37.929928 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7057acc6-78aa-43fa-bc57-6e5ecacf2e50-credential-socket" pod "7057acc6-78aa-43fa-bc57-6e5ecacf2e50" (UID: "7057acc6-78aa-43fa-bc57-6e5ecacf2e50"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:21:37.932692 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:37.930013 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7057acc6-78aa-43fa-bc57-6e5ecacf2e50-istiod-ca-cert" pod "7057acc6-78aa-43fa-bc57-6e5ecacf2e50" (UID: "7057acc6-78aa-43fa-bc57-6e5ecacf2e50"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:21:37.932692 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:37.930115 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7057acc6-78aa-43fa-bc57-6e5ecacf2e50-workload-socket" pod "7057acc6-78aa-43fa-bc57-6e5ecacf2e50" (UID: "7057acc6-78aa-43fa-bc57-6e5ecacf2e50"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:21:37.937115 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:37.937048 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7057acc6-78aa-43fa-bc57-6e5ecacf2e50-istio-podinfo" pod "7057acc6-78aa-43fa-bc57-6e5ecacf2e50" (UID: "7057acc6-78aa-43fa-bc57-6e5ecacf2e50"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:21:37.938364 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:37.938310 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7057acc6-78aa-43fa-bc57-6e5ecacf2e50-istio-token" pod "7057acc6-78aa-43fa-bc57-6e5ecacf2e50" (UID: "7057acc6-78aa-43fa-bc57-6e5ecacf2e50"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:21:38.045701 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:38.021854 4982 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7057acc6-78aa-43fa-bc57-6e5ecacf2e50-istio-data\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:21:38.045999 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:38.045970 4982 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7057acc6-78aa-43fa-bc57-6e5ecacf2e50-istio-token\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:21:38.046125 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:38.046111 4982 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7057acc6-78aa-43fa-bc57-6e5ecacf2e50-workload-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:21:38.046211 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:38.046198 4982 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7057acc6-78aa-43fa-bc57-6e5ecacf2e50-credential-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:21:38.046295 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:38.046283 4982 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7057acc6-78aa-43fa-bc57-6e5ecacf2e50-istiod-ca-cert\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:21:38.046377 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:38.046366 4982 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7057acc6-78aa-43fa-bc57-6e5ecacf2e50-istio-envoy\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:21:38.046463 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:38.046452 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcxpd\" (UniqueName: \"kubernetes.io/projected/7057acc6-78aa-43fa-bc57-6e5ecacf2e50-kube-api-access-wcxpd\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:21:38.046552 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:38.046539 4982 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7057acc6-78aa-43fa-bc57-6e5ecacf2e50-workload-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:21:38.046658 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:38.046630 4982 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7057acc6-78aa-43fa-bc57-6e5ecacf2e50-istio-podinfo\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:21:38.236310 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:38.236211 4982 generic.go:356] "Generic (PLEG): container finished" podID="7057acc6-78aa-43fa-bc57-6e5ecacf2e50" containerID="6a68340dffb035a63f91af47e2f3bbcba54e742ee429f812b080b6162d8b73ac" exitCode=0 Oct 30 14:21:38.236808 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:38.236505 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg" event={"ID":"7057acc6-78aa-43fa-bc57-6e5ecacf2e50","Type":"ContainerDied","Data":"6a68340dffb035a63f91af47e2f3bbcba54e742ee429f812b080b6162d8b73ac"} Oct 30 14:21:38.237053 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:38.237034 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg" event={"ID":"7057acc6-78aa-43fa-bc57-6e5ecacf2e50","Type":"ContainerDied","Data":"9c51b51be948bfd3b56f33003ee7a8e0018774021addfc1875c77b95ba271580"} Oct 30 14:21:38.237174 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:38.236585 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg" Oct 30 14:21:38.238232 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:38.238210 4982 scope.go:122] "RemoveContainer" containerID="6a68340dffb035a63f91af47e2f3bbcba54e742ee429f812b080b6162d8b73ac" Oct 30 14:21:38.255618 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:38.255597 4982 scope.go:122] "RemoveContainer" containerID="ad6543ac955d66904b53a21d988117a4636a57451e9d2407dd6c912eace2e435" Oct 30 14:21:38.266047 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:38.266027 4982 scope.go:122] "RemoveContainer" containerID="6a68340dffb035a63f91af47e2f3bbcba54e742ee429f812b080b6162d8b73ac" Oct 30 14:21:38.266370 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:21:38.266349 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a68340dffb035a63f91af47e2f3bbcba54e742ee429f812b080b6162d8b73ac\": container with ID starting with 6a68340dffb035a63f91af47e2f3bbcba54e742ee429f812b080b6162d8b73ac not found: ID does not exist" containerID="6a68340dffb035a63f91af47e2f3bbcba54e742ee429f812b080b6162d8b73ac" Oct 30 14:21:38.266419 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:38.266390 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a68340dffb035a63f91af47e2f3bbcba54e742ee429f812b080b6162d8b73ac"} err="failed to get container status \"6a68340dffb035a63f91af47e2f3bbcba54e742ee429f812b080b6162d8b73ac\": rpc error: code = NotFound desc = could not find container \"6a68340dffb035a63f91af47e2f3bbcba54e742ee429f812b080b6162d8b73ac\": container with ID starting with 6a68340dffb035a63f91af47e2f3bbcba54e742ee429f812b080b6162d8b73ac not found: ID does not exist" Oct 30 14:21:38.266419 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:38.266414 4982 scope.go:122] "RemoveContainer" containerID="ad6543ac955d66904b53a21d988117a4636a57451e9d2407dd6c912eace2e435" Oct 30 14:21:38.266754 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:21:38.266726 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad6543ac955d66904b53a21d988117a4636a57451e9d2407dd6c912eace2e435\": container with ID starting with ad6543ac955d66904b53a21d988117a4636a57451e9d2407dd6c912eace2e435 not found: ID does not exist" containerID="ad6543ac955d66904b53a21d988117a4636a57451e9d2407dd6c912eace2e435" Oct 30 14:21:38.266843 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:38.266766 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad6543ac955d66904b53a21d988117a4636a57451e9d2407dd6c912eace2e435"} err="failed to get container status \"ad6543ac955d66904b53a21d988117a4636a57451e9d2407dd6c912eace2e435\": rpc error: code = NotFound desc = could not find container \"ad6543ac955d66904b53a21d988117a4636a57451e9d2407dd6c912eace2e435\": container with ID starting with ad6543ac955d66904b53a21d988117a4636a57451e9d2407dd6c912eace2e435 not found: ID does not exist" Oct 30 14:21:38.278951 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:38.278929 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aab379-lbc58"] Oct 30 14:21:38.279607 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:38.279592 4982 cpu_manager.go:414] "containerMap: removing container" podUID="b2369048-24d1-4882-bb7e-a3122a7a85bc" containerName="revoke-keys" Oct 30 14:21:38.279674 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:38.279610 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2369048-24d1-4882-bb7e-a3122a7a85bc" containerName="revoke-keys" Oct 30 14:21:38.279674 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:38.279668 4982 cpu_manager.go:414] "containerMap: removing container" podUID="7057acc6-78aa-43fa-bc57-6e5ecacf2e50" containerName="istio-proxy" Oct 30 14:21:38.279768 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:38.279676 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7057acc6-78aa-43fa-bc57-6e5ecacf2e50" containerName="istio-proxy" Oct 30 14:21:38.279768 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:38.279689 4982 cpu_manager.go:414] "containerMap: removing container" podUID="7057acc6-78aa-43fa-bc57-6e5ecacf2e50" containerName="istio-proxy" Oct 30 14:21:38.279768 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:38.279694 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7057acc6-78aa-43fa-bc57-6e5ecacf2e50" containerName="istio-proxy" Oct 30 14:21:38.279904 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:38.279812 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2369048-24d1-4882-bb7e-a3122a7a85bc" containerName="revoke-keys" Oct 30 14:21:38.279904 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:38.279823 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="7057acc6-78aa-43fa-bc57-6e5ecacf2e50" containerName="istio-proxy" Oct 30 14:21:38.279904 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:38.279830 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="7057acc6-78aa-43fa-bc57-6e5ecacf2e50" containerName="istio-proxy" Oct 30 14:21:38.284918 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:38.284899 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aab379-lbc58" Oct 30 14:21:38.298918 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:38.298890 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg"] Oct 30 14:21:38.309661 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:38.308312 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aab379-lbc58"] Oct 30 14:21:38.309661 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:38.309307 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:21:38.309661 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:38.309433 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:21:38.314851 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:38.314822 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-9dba4af9-openshift-default-7d7d889b5-kgllg"] Oct 30 14:21:38.401761 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:38.401723 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7057acc6-78aa-43fa-bc57-6e5ecacf2e50" path="/var/lib/kubelet/pods/7057acc6-78aa-43fa-bc57-6e5ecacf2e50/volumes" Oct 30 14:21:38.453578 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:38.453543 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1010904-6f90-4efb-ab8b-30afbe7187a4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-aab379-lbc58\" (UID: \"f1010904-6f90-4efb-ab8b-30afbe7187a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aab379-lbc58" Oct 30 14:21:38.453799 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:38.453607 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2gn9\" (UniqueName: \"kubernetes.io/projected/f1010904-6f90-4efb-ab8b-30afbe7187a4-kube-api-access-h2gn9\") pod \"maas-api-revoke-keys-e2e-api-b-aab379-lbc58\" (UID: \"f1010904-6f90-4efb-ab8b-30afbe7187a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aab379-lbc58" Oct 30 14:21:38.453799 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:38.453706 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1010904-6f90-4efb-ab8b-30afbe7187a4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-aab379-lbc58\" (UID: \"f1010904-6f90-4efb-ab8b-30afbe7187a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aab379-lbc58" Oct 30 14:21:38.555306 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:38.555222 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1010904-6f90-4efb-ab8b-30afbe7187a4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-aab379-lbc58\" (UID: \"f1010904-6f90-4efb-ab8b-30afbe7187a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aab379-lbc58" Oct 30 14:21:38.555474 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:38.555322 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2gn9\" (UniqueName: \"kubernetes.io/projected/f1010904-6f90-4efb-ab8b-30afbe7187a4-kube-api-access-h2gn9\") pod \"maas-api-revoke-keys-e2e-api-b-aab379-lbc58\" (UID: \"f1010904-6f90-4efb-ab8b-30afbe7187a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aab379-lbc58" Oct 30 14:21:38.555474 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:38.555352 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1010904-6f90-4efb-ab8b-30afbe7187a4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-aab379-lbc58\" (UID: \"f1010904-6f90-4efb-ab8b-30afbe7187a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aab379-lbc58" Oct 30 14:21:38.555978 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:38.555949 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1010904-6f90-4efb-ab8b-30afbe7187a4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-aab379-lbc58\" (UID: \"f1010904-6f90-4efb-ab8b-30afbe7187a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aab379-lbc58" Oct 30 14:21:38.557718 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:38.557693 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1010904-6f90-4efb-ab8b-30afbe7187a4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-aab379-lbc58\" (UID: \"f1010904-6f90-4efb-ab8b-30afbe7187a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aab379-lbc58" Oct 30 14:21:38.581095 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:38.581067 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2gn9\" (UniqueName: \"kubernetes.io/projected/f1010904-6f90-4efb-ab8b-30afbe7187a4-kube-api-access-h2gn9\") pod \"maas-api-revoke-keys-e2e-api-b-aab379-lbc58\" (UID: \"f1010904-6f90-4efb-ab8b-30afbe7187a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aab379-lbc58" Oct 30 14:21:38.602800 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:38.602772 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aab379-lbc58" Oct 30 14:21:38.917494 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:38.917463 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aab379-lbc58"] Oct 30 14:21:38.919357 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:21:38.919326 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1010904_6f90_4efb_ab8b_30afbe7187a4.slice/crio-c6a36eccee0d23a79648549487b8418b3728aa2d15db076172f21d64bd11021e WatchSource:0}: Error finding container c6a36eccee0d23a79648549487b8418b3728aa2d15db076172f21d64bd11021e: Status 404 returned error can't find the container with id c6a36eccee0d23a79648549487b8418b3728aa2d15db076172f21d64bd11021e Oct 30 14:21:39.147889 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:39.147857 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dd54ca2-58vcb"] Oct 30 14:21:39.153534 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:39.153515 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dd54ca2-58vcb" Oct 30 14:21:39.174428 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:39.174347 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dd54ca2-58vcb"] Oct 30 14:21:39.244851 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:39.244817 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aab379-lbc58" event={"ID":"f1010904-6f90-4efb-ab8b-30afbe7187a4","Type":"ContainerStarted","Data":"c6a36eccee0d23a79648549487b8418b3728aa2d15db076172f21d64bd11021e"} Oct 30 14:21:39.267391 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:39.267364 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnzr6\" (UniqueName: \"kubernetes.io/projected/0ca3d212-3272-406a-9c00-d3968904a19c-kube-api-access-rnzr6\") pod \"maas-api-revoke-sub-5dd54ca2-58vcb\" (UID: \"0ca3d212-3272-406a-9c00-d3968904a19c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dd54ca2-58vcb" Oct 30 14:21:39.267590 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:39.267567 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ca3d212-3272-406a-9c00-d3968904a19c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5dd54ca2-58vcb\" (UID: \"0ca3d212-3272-406a-9c00-d3968904a19c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dd54ca2-58vcb" Oct 30 14:21:39.267635 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:39.267621 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ca3d212-3272-406a-9c00-d3968904a19c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5dd54ca2-58vcb\" (UID: \"0ca3d212-3272-406a-9c00-d3968904a19c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dd54ca2-58vcb" Oct 30 14:21:39.369165 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:39.369128 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ca3d212-3272-406a-9c00-d3968904a19c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5dd54ca2-58vcb\" (UID: \"0ca3d212-3272-406a-9c00-d3968904a19c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dd54ca2-58vcb" Oct 30 14:21:39.369316 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:39.369186 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ca3d212-3272-406a-9c00-d3968904a19c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5dd54ca2-58vcb\" (UID: \"0ca3d212-3272-406a-9c00-d3968904a19c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dd54ca2-58vcb" Oct 30 14:21:39.369316 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:39.369268 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnzr6\" (UniqueName: \"kubernetes.io/projected/0ca3d212-3272-406a-9c00-d3968904a19c-kube-api-access-rnzr6\") pod \"maas-api-revoke-sub-5dd54ca2-58vcb\" (UID: \"0ca3d212-3272-406a-9c00-d3968904a19c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dd54ca2-58vcb" Oct 30 14:21:39.369964 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:39.369941 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ca3d212-3272-406a-9c00-d3968904a19c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5dd54ca2-58vcb\" (UID: \"0ca3d212-3272-406a-9c00-d3968904a19c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dd54ca2-58vcb" Oct 30 14:21:39.371504 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:39.371473 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ca3d212-3272-406a-9c00-d3968904a19c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5dd54ca2-58vcb\" (UID: \"0ca3d212-3272-406a-9c00-d3968904a19c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dd54ca2-58vcb" Oct 30 14:21:39.376067 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:39.376044 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:39.435669 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:39.435575 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnzr6\" (UniqueName: \"kubernetes.io/projected/0ca3d212-3272-406a-9c00-d3968904a19c-kube-api-access-rnzr6\") pod \"maas-api-revoke-sub-5dd54ca2-58vcb\" (UID: \"0ca3d212-3272-406a-9c00-d3968904a19c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dd54ca2-58vcb" Oct 30 14:21:39.466838 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:39.466802 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dd54ca2-58vcb" Oct 30 14:21:39.781277 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:21:39.781247 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ca3d212_3272_406a_9c00_d3968904a19c.slice/crio-e6a58bad8a986a75765eb09d150c942e1e75489d26d8630b7dd581edca7669f6 WatchSource:0}: Error finding container e6a58bad8a986a75765eb09d150c942e1e75489d26d8630b7dd581edca7669f6: Status 404 returned error can't find the container with id e6a58bad8a986a75765eb09d150c942e1e75489d26d8630b7dd581edca7669f6 Oct 30 14:21:39.785745 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:39.785713 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dd54ca2-58vcb"] Oct 30 14:21:40.255617 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:40.255588 4982 generic.go:356] "Generic (PLEG): container finished" podID="f1010904-6f90-4efb-ab8b-30afbe7187a4" containerID="c690de79cf7a93b408455d6e843de01c3d86ad471ae9b28ebc331684b87fd281" exitCode=0 Oct 30 14:21:40.255803 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:40.255695 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aab379-lbc58" event={"ID":"f1010904-6f90-4efb-ab8b-30afbe7187a4","Type":"ContainerDied","Data":"c690de79cf7a93b408455d6e843de01c3d86ad471ae9b28ebc331684b87fd281"} Oct 30 14:21:40.257656 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:40.257579 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dd54ca2-58vcb" event={"ID":"0ca3d212-3272-406a-9c00-d3968904a19c","Type":"ContainerStarted","Data":"2d9714211b91d266cf176e5f58b77f7331efdf614ef9cd9e1e127eaffc973e1a"} Oct 30 14:21:40.257656 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:40.257627 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dd54ca2-58vcb" event={"ID":"0ca3d212-3272-406a-9c00-d3968904a19c","Type":"ContainerStarted","Data":"e6a58bad8a986a75765eb09d150c942e1e75489d26d8630b7dd581edca7669f6"} Oct 30 14:21:40.299111 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:40.299070 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:40.347811 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:40.347719 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dd54ca2-58vcb" podStartSLOduration=1.347707331 podStartE2EDuration="1.347707331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:21:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:21:40.347542477 +0000 UTC m=+1970.091676518" watchObservedRunningTime="2026-10-30 14:21:40.347707331 +0000 UTC m=+1970.091841371" Oct 30 14:21:40.611034 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:40.610942 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:40.792396 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:40.792010 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:41.291718 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:41.287542 4982 generic.go:356] "Generic (PLEG): container finished" podID="0ca3d212-3272-406a-9c00-d3968904a19c" containerID="2d9714211b91d266cf176e5f58b77f7331efdf614ef9cd9e1e127eaffc973e1a" exitCode=0 Oct 30 14:21:41.291718 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:41.288275 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dd54ca2-58vcb" event={"ID":"0ca3d212-3272-406a-9c00-d3968904a19c","Type":"ContainerDied","Data":"2d9714211b91d266cf176e5f58b77f7331efdf614ef9cd9e1e127eaffc973e1a"} Oct 30 14:21:41.488188 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:41.488161 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aab379-lbc58" Oct 30 14:21:41.620558 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:41.620522 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1010904-6f90-4efb-ab8b-30afbe7187a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1010904-6f90-4efb-ab8b-30afbe7187a4-maas-api-openshift-ca\") pod \"f1010904-6f90-4efb-ab8b-30afbe7187a4\" (UID: \"f1010904-6f90-4efb-ab8b-30afbe7187a4\") " Oct 30 14:21:41.621058 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:41.620690 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1010904-6f90-4efb-ab8b-30afbe7187a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1010904-6f90-4efb-ab8b-30afbe7187a4-maas-api-xks-ca\") pod \"f1010904-6f90-4efb-ab8b-30afbe7187a4\" (UID: \"f1010904-6f90-4efb-ab8b-30afbe7187a4\") " Oct 30 14:21:41.621058 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:41.620770 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1010904-6f90-4efb-ab8b-30afbe7187a4-kube-api-access-h2gn9\" (UniqueName: \"kubernetes.io/projected/f1010904-6f90-4efb-ab8b-30afbe7187a4-kube-api-access-h2gn9\") pod \"f1010904-6f90-4efb-ab8b-30afbe7187a4\" (UID: \"f1010904-6f90-4efb-ab8b-30afbe7187a4\") " Oct 30 14:21:41.621197 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:41.621064 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1010904-6f90-4efb-ab8b-30afbe7187a4-maas-api-openshift-ca" pod "f1010904-6f90-4efb-ab8b-30afbe7187a4" (UID: "f1010904-6f90-4efb-ab8b-30afbe7187a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:21:41.621505 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:41.621477 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1010904-6f90-4efb-ab8b-30afbe7187a4-maas-api-openshift-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:21:41.622950 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:41.622908 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1010904-6f90-4efb-ab8b-30afbe7187a4-maas-api-xks-ca" pod "f1010904-6f90-4efb-ab8b-30afbe7187a4" (UID: "f1010904-6f90-4efb-ab8b-30afbe7187a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:21:41.623060 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:41.623031 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1010904-6f90-4efb-ab8b-30afbe7187a4-kube-api-access-h2gn9" pod "f1010904-6f90-4efb-ab8b-30afbe7187a4" (UID: "f1010904-6f90-4efb-ab8b-30afbe7187a4"). InnerVolumeSpecName "kube-api-access-h2gn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:21:41.722964 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:41.722927 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1010904-6f90-4efb-ab8b-30afbe7187a4-maas-api-xks-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:21:41.722964 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:41.722961 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2gn9\" (UniqueName: \"kubernetes.io/projected/f1010904-6f90-4efb-ab8b-30afbe7187a4-kube-api-access-h2gn9\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:21:41.746673 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:41.746582 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:42.093394 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:42.093363 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-ca02c9-7fc799d5c8-pkcxp" Oct 30 14:21:42.194993 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:42.194953 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:42.297261 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:42.297225 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aab379-lbc58" Oct 30 14:21:42.297460 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:42.297253 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aab379-lbc58" event={"ID":"f1010904-6f90-4efb-ab8b-30afbe7187a4","Type":"ContainerDied","Data":"c6a36eccee0d23a79648549487b8418b3728aa2d15db076172f21d64bd11021e"} Oct 30 14:21:42.297460 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:42.297311 4982 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6a36eccee0d23a79648549487b8418b3728aa2d15db076172f21d64bd11021e" Oct 30 14:21:42.348021 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:42.347948 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ea9ed2-849b7b59dc-vp88x"] Oct 30 14:21:42.348557 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:42.348504 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:42.348805 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:42.348785 4982 cpu_manager.go:414] "containerMap: removing container" podUID="f1010904-6f90-4efb-ab8b-30afbe7187a4" containerName="revoke-keys" Oct 30 14:21:42.348858 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:42.348807 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1010904-6f90-4efb-ab8b-30afbe7187a4" containerName="revoke-keys" Oct 30 14:21:42.348969 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:42.348958 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1010904-6f90-4efb-ab8b-30afbe7187a4" containerName="revoke-keys" Oct 30 14:21:42.352607 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:42.352590 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ea9ed2-849b7b59dc-vp88x" Oct 30 14:21:42.377726 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:42.377691 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ea9ed2-849b7b59dc-vp88x"] Oct 30 14:21:42.384249 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:42.379069 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-ea9ed2-dockercfg-msdm5\"" Oct 30 14:21:42.384249 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:42.379414 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-ea9ed2\"" Oct 30 14:21:42.699172 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:42.613216 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56fc4cc3-42d8-4e38-bd33-a7ab94c7dea8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ea9ed2-849b7b59dc-vp88x\" (UID: \"56fc4cc3-42d8-4e38-bd33-a7ab94c7dea8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ea9ed2-849b7b59dc-vp88x" Oct 30 14:21:42.699172 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:42.613283 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zkwd\" (UniqueName: \"kubernetes.io/projected/56fc4cc3-42d8-4e38-bd33-a7ab94c7dea8-kube-api-access-4zkwd\") pod \"payload-pre-processing-e2e-api-b-ea9ed2-849b7b59dc-vp88x\" (UID: \"56fc4cc3-42d8-4e38-bd33-a7ab94c7dea8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ea9ed2-849b7b59dc-vp88x" Oct 30 14:21:42.700989 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:42.700848 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ea9ed2-78fc9fb9fb-ftnvk"] Oct 30 14:21:42.706475 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:42.706214 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ea9ed2-78fc9fb9fb-ftnvk"] Oct 30 14:21:42.706475 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:42.706332 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ea9ed2-78fc9fb9fb-ftnvk" Oct 30 14:21:42.716961 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:42.716934 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56fc4cc3-42d8-4e38-bd33-a7ab94c7dea8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ea9ed2-849b7b59dc-vp88x\" (UID: \"56fc4cc3-42d8-4e38-bd33-a7ab94c7dea8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ea9ed2-849b7b59dc-vp88x" Oct 30 14:21:42.717102 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:42.717006 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zkwd\" (UniqueName: \"kubernetes.io/projected/56fc4cc3-42d8-4e38-bd33-a7ab94c7dea8-kube-api-access-4zkwd\") pod \"payload-pre-processing-e2e-api-b-ea9ed2-849b7b59dc-vp88x\" (UID: \"56fc4cc3-42d8-4e38-bd33-a7ab94c7dea8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ea9ed2-849b7b59dc-vp88x" Oct 30 14:21:42.718050 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:42.718021 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56fc4cc3-42d8-4e38-bd33-a7ab94c7dea8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ea9ed2-849b7b59dc-vp88x\" (UID: \"56fc4cc3-42d8-4e38-bd33-a7ab94c7dea8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ea9ed2-849b7b59dc-vp88x" Oct 30 14:21:42.718435 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:42.718411 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a754700e-df12-456f-af9e-2b5f408ad14e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ea9ed2-78fc9fb9fb-ftnvk\" (UID: \"a754700e-df12-456f-af9e-2b5f408ad14e\") " pod="openshift-ingress/payload-processing-e2e-api-b-ea9ed2-78fc9fb9fb-ftnvk" Oct 30 14:21:42.718541 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:42.718504 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtkqr\" (UniqueName: \"kubernetes.io/projected/a754700e-df12-456f-af9e-2b5f408ad14e-kube-api-access-dtkqr\") pod \"payload-processing-e2e-api-b-ea9ed2-78fc9fb9fb-ftnvk\" (UID: \"a754700e-df12-456f-af9e-2b5f408ad14e\") " pod="openshift-ingress/payload-processing-e2e-api-b-ea9ed2-78fc9fb9fb-ftnvk" Oct 30 14:21:42.720509 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:42.720485 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dd54ca2-58vcb" Oct 30 14:21:42.743527 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:42.743500 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zkwd\" (UniqueName: \"kubernetes.io/projected/56fc4cc3-42d8-4e38-bd33-a7ab94c7dea8-kube-api-access-4zkwd\") pod \"payload-pre-processing-e2e-api-b-ea9ed2-849b7b59dc-vp88x\" (UID: \"56fc4cc3-42d8-4e38-bd33-a7ab94c7dea8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ea9ed2-849b7b59dc-vp88x" Oct 30 14:21:42.819371 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:42.819326 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ca3d212-3272-406a-9c00-d3968904a19c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ca3d212-3272-406a-9c00-d3968904a19c-maas-api-xks-ca\") pod \"0ca3d212-3272-406a-9c00-d3968904a19c\" (UID: \"0ca3d212-3272-406a-9c00-d3968904a19c\") " Oct 30 14:21:42.819541 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:42.819458 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ca3d212-3272-406a-9c00-d3968904a19c-kube-api-access-rnzr6\" (UniqueName: \"kubernetes.io/projected/0ca3d212-3272-406a-9c00-d3968904a19c-kube-api-access-rnzr6\") pod \"0ca3d212-3272-406a-9c00-d3968904a19c\" (UID: \"0ca3d212-3272-406a-9c00-d3968904a19c\") " Oct 30 14:21:42.819973 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:42.819935 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ca3d212-3272-406a-9c00-d3968904a19c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ca3d212-3272-406a-9c00-d3968904a19c-maas-api-openshift-ca\") pod \"0ca3d212-3272-406a-9c00-d3968904a19c\" (UID: \"0ca3d212-3272-406a-9c00-d3968904a19c\") " Oct 30 14:21:42.820455 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:42.820428 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a754700e-df12-456f-af9e-2b5f408ad14e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ea9ed2-78fc9fb9fb-ftnvk\" (UID: \"a754700e-df12-456f-af9e-2b5f408ad14e\") " pod="openshift-ingress/payload-processing-e2e-api-b-ea9ed2-78fc9fb9fb-ftnvk" Oct 30 14:21:42.820559 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:42.820521 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtkqr\" (UniqueName: \"kubernetes.io/projected/a754700e-df12-456f-af9e-2b5f408ad14e-kube-api-access-dtkqr\") pod \"payload-processing-e2e-api-b-ea9ed2-78fc9fb9fb-ftnvk\" (UID: \"a754700e-df12-456f-af9e-2b5f408ad14e\") " pod="openshift-ingress/payload-processing-e2e-api-b-ea9ed2-78fc9fb9fb-ftnvk" Oct 30 14:21:42.821737 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:42.821674 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a754700e-df12-456f-af9e-2b5f408ad14e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ea9ed2-78fc9fb9fb-ftnvk\" (UID: \"a754700e-df12-456f-af9e-2b5f408ad14e\") " pod="openshift-ingress/payload-processing-e2e-api-b-ea9ed2-78fc9fb9fb-ftnvk" Oct 30 14:21:42.822194 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:42.822169 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ca3d212-3272-406a-9c00-d3968904a19c-maas-api-openshift-ca" pod "0ca3d212-3272-406a-9c00-d3968904a19c" (UID: "0ca3d212-3272-406a-9c00-d3968904a19c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:21:42.823903 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:42.823868 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ca3d212-3272-406a-9c00-d3968904a19c-maas-api-xks-ca" pod "0ca3d212-3272-406a-9c00-d3968904a19c" (UID: "0ca3d212-3272-406a-9c00-d3968904a19c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:21:42.824281 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:42.824257 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ca3d212-3272-406a-9c00-d3968904a19c-kube-api-access-rnzr6" pod "0ca3d212-3272-406a-9c00-d3968904a19c" (UID: "0ca3d212-3272-406a-9c00-d3968904a19c"). InnerVolumeSpecName "kube-api-access-rnzr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:21:42.849914 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:42.849888 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtkqr\" (UniqueName: \"kubernetes.io/projected/a754700e-df12-456f-af9e-2b5f408ad14e-kube-api-access-dtkqr\") pod \"payload-processing-e2e-api-b-ea9ed2-78fc9fb9fb-ftnvk\" (UID: \"a754700e-df12-456f-af9e-2b5f408ad14e\") " pod="openshift-ingress/payload-processing-e2e-api-b-ea9ed2-78fc9fb9fb-ftnvk" Oct 30 14:21:42.925375 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:42.925331 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ca3d212-3272-406a-9c00-d3968904a19c-maas-api-xks-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:21:42.925375 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:42.925367 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnzr6\" (UniqueName: \"kubernetes.io/projected/0ca3d212-3272-406a-9c00-d3968904a19c-kube-api-access-rnzr6\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:21:42.925375 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:42.925382 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ca3d212-3272-406a-9c00-d3968904a19c-maas-api-openshift-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:21:42.972402 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:42.972322 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ea9ed2-849b7b59dc-vp88x" Oct 30 14:21:43.033534 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:43.033505 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ea9ed2-78fc9fb9fb-ftnvk" Oct 30 14:21:43.288138 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:43.288107 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ea9ed2-849b7b59dc-vp88x"] Oct 30 14:21:43.288242 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:21:43.288103 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56fc4cc3_42d8_4e38_bd33_a7ab94c7dea8.slice/crio-83d68674cc0cfb94085b3bd1cffdcd93ad97e378292b5e61ab44b5e38f177139 WatchSource:0}: Error finding container 83d68674cc0cfb94085b3bd1cffdcd93ad97e378292b5e61ab44b5e38f177139: Status 404 returned error can't find the container with id 83d68674cc0cfb94085b3bd1cffdcd93ad97e378292b5e61ab44b5e38f177139 Oct 30 14:21:43.310678 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:43.308067 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dd54ca2-58vcb" Oct 30 14:21:43.310678 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:43.308063 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dd54ca2-58vcb" event={"ID":"0ca3d212-3272-406a-9c00-d3968904a19c","Type":"ContainerDied","Data":"e6a58bad8a986a75765eb09d150c942e1e75489d26d8630b7dd581edca7669f6"} Oct 30 14:21:43.310678 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:43.308156 4982 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6a58bad8a986a75765eb09d150c942e1e75489d26d8630b7dd581edca7669f6" Oct 30 14:21:43.310678 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:43.309604 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ea9ed2-849b7b59dc-vp88x" event={"ID":"56fc4cc3-42d8-4e38-bd33-a7ab94c7dea8","Type":"ContainerStarted","Data":"83d68674cc0cfb94085b3bd1cffdcd93ad97e378292b5e61ab44b5e38f177139"} Oct 30 14:21:43.394761 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:43.394731 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ea9ed2-78fc9fb9fb-ftnvk"] Oct 30 14:21:43.396111 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:21:43.396081 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda754700e_df12_456f_af9e_2b5f408ad14e.slice/crio-53b7f965eb49c9730aa47a86442e0fe11ac12619124165e7da23702765fc6a13 WatchSource:0}: Error finding container 53b7f965eb49c9730aa47a86442e0fe11ac12619124165e7da23702765fc6a13: Status 404 returned error can't find the container with id 53b7f965eb49c9730aa47a86442e0fe11ac12619124165e7da23702765fc6a13 Oct 30 14:21:43.969427 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:43.969329 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:44.204080 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:44.204045 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:44.319274 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:44.319243 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ea9ed2-849b7b59dc-vp88x" event={"ID":"56fc4cc3-42d8-4e38-bd33-a7ab94c7dea8","Type":"ContainerStarted","Data":"d7d09adf67e7fae45155b06f6403ef983d73f87f97336255bc67ba5785ea0606"} Oct 30 14:21:44.319467 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:44.319336 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ea9ed2-849b7b59dc-vp88x" Oct 30 14:21:44.321206 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:44.321178 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ea9ed2-78fc9fb9fb-ftnvk" event={"ID":"a754700e-df12-456f-af9e-2b5f408ad14e","Type":"ContainerStarted","Data":"bd5444b7bf193899d9e657d596afe6d7f20d5e6f39deb148455132df5a6c2685"} Oct 30 14:21:44.321308 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:44.321216 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ea9ed2-78fc9fb9fb-ftnvk" event={"ID":"a754700e-df12-456f-af9e-2b5f408ad14e","Type":"ContainerStarted","Data":"53b7f965eb49c9730aa47a86442e0fe11ac12619124165e7da23702765fc6a13"} Oct 30 14:21:44.321308 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:44.321239 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-ea9ed2-78fc9fb9fb-ftnvk" Oct 30 14:21:44.365189 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:44.365141 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ea9ed2-849b7b59dc-vp88x" podStartSLOduration=2.365127769 podStartE2EDuration="2.365127769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:21:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:21:44.362045784 +0000 UTC m=+1974.106179825" watchObservedRunningTime="2026-10-30 14:21:44.365127769 +0000 UTC m=+1974.109261810" Oct 30 14:21:44.410372 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:44.410313 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-ea9ed2-78fc9fb9fb-ftnvk" podStartSLOduration=2.410294198 podStartE2EDuration="2.410294198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:21:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:21:44.405186317 +0000 UTC m=+1974.149320358" watchObservedRunningTime="2026-10-30 14:21:44.410294198 +0000 UTC m=+1974.154428242" Oct 30 14:21:46.097330 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:46.097288 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:47.964756 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:47.964717 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:48.901951 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:48.901908 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:49.094501 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:49.094452 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:49.862081 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:49.862037 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:50.017198 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:50.017160 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:50.450670 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:50.446597 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:50.805930 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:50.805888 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:51.846126 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:51.846094 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:53.666905 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:53.665004 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:54.511021 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:54.510985 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:54.670482 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:54.670449 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:55.332548 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:55.332518 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ea9ed2-849b7b59dc-vp88x" Oct 30 14:21:55.332737 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:55.332584 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-ea9ed2-78fc9fb9fb-ftnvk" Oct 30 14:21:55.987550 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:55.985233 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:56.363803 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:56.363767 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:56.424672 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:56.424628 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-648d5f-54b7c56458-79pb4"] Oct 30 14:21:56.425340 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:56.425319 4982 cpu_manager.go:414] "containerMap: removing container" podUID="0ca3d212-3272-406a-9c00-d3968904a19c" containerName="revoke-keys" Oct 30 14:21:56.425340 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:56.425339 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ca3d212-3272-406a-9c00-d3968904a19c" containerName="revoke-keys" Oct 30 14:21:56.425557 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:56.425542 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ca3d212-3272-406a-9c00-d3968904a19c" containerName="revoke-keys" Oct 30 14:21:56.430617 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:56.430597 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-648d5f-54b7c56458-79pb4" Oct 30 14:21:56.451871 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:56.451845 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-648d5f-54b7c56458-79pb4"] Oct 30 14:21:56.460448 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:56.460423 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-648d5f\"" Oct 30 14:21:56.521155 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:56.521125 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:56.562680 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:56.562058 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52mbw\" (UniqueName: \"kubernetes.io/projected/60f102c4-46ef-4dbb-8eba-cec0e78a3b3e-kube-api-access-52mbw\") pod \"payload-processing-e2e-inf-a-648d5f-54b7c56458-79pb4\" (UID: \"60f102c4-46ef-4dbb-8eba-cec0e78a3b3e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-648d5f-54b7c56458-79pb4" Oct 30 14:21:56.562680 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:56.562194 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60f102c4-46ef-4dbb-8eba-cec0e78a3b3e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-648d5f-54b7c56458-79pb4\" (UID: \"60f102c4-46ef-4dbb-8eba-cec0e78a3b3e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-648d5f-54b7c56458-79pb4" Oct 30 14:21:56.665707 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:56.663557 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60f102c4-46ef-4dbb-8eba-cec0e78a3b3e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-648d5f-54b7c56458-79pb4\" (UID: \"60f102c4-46ef-4dbb-8eba-cec0e78a3b3e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-648d5f-54b7c56458-79pb4" Oct 30 14:21:56.665707 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:56.663888 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52mbw\" (UniqueName: \"kubernetes.io/projected/60f102c4-46ef-4dbb-8eba-cec0e78a3b3e-kube-api-access-52mbw\") pod \"payload-processing-e2e-inf-a-648d5f-54b7c56458-79pb4\" (UID: \"60f102c4-46ef-4dbb-8eba-cec0e78a3b3e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-648d5f-54b7c56458-79pb4" Oct 30 14:21:56.665707 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:56.665163 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60f102c4-46ef-4dbb-8eba-cec0e78a3b3e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-648d5f-54b7c56458-79pb4\" (UID: \"60f102c4-46ef-4dbb-8eba-cec0e78a3b3e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-648d5f-54b7c56458-79pb4" Oct 30 14:21:56.695689 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:56.693977 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52mbw\" (UniqueName: \"kubernetes.io/projected/60f102c4-46ef-4dbb-8eba-cec0e78a3b3e-kube-api-access-52mbw\") pod \"payload-processing-e2e-inf-a-648d5f-54b7c56458-79pb4\" (UID: \"60f102c4-46ef-4dbb-8eba-cec0e78a3b3e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-648d5f-54b7c56458-79pb4" Oct 30 14:21:56.745093 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:56.745051 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-648d5f-54b7c56458-79pb4" Oct 30 14:21:57.091652 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:21:57.091610 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60f102c4_46ef_4dbb_8eba_cec0e78a3b3e.slice/crio-932e5d6e60455e30cfe4a8b71f19cec11b4ecbb581fe9cde339f829a5ca3d446 WatchSource:0}: Error finding container 932e5d6e60455e30cfe4a8b71f19cec11b4ecbb581fe9cde339f829a5ca3d446: Status 404 returned error can't find the container with id 932e5d6e60455e30cfe4a8b71f19cec11b4ecbb581fe9cde339f829a5ca3d446 Oct 30 14:21:57.092059 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:57.092025 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-648d5f-54b7c56458-79pb4"] Oct 30 14:21:57.430435 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:57.430352 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-648d5f-54b7c56458-79pb4" event={"ID":"60f102c4-46ef-4dbb-8eba-cec0e78a3b3e","Type":"ContainerStarted","Data":"5758d33bc72a8921fc1b8291c874b2650322cc4a087e611e4740b0e75ea71a9e"} Oct 30 14:21:57.430435 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:57.430407 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-648d5f-54b7c56458-79pb4" event={"ID":"60f102c4-46ef-4dbb-8eba-cec0e78a3b3e","Type":"ContainerStarted","Data":"932e5d6e60455e30cfe4a8b71f19cec11b4ecbb581fe9cde339f829a5ca3d446"} Oct 30 14:21:57.430608 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:57.430479 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-648d5f-54b7c56458-79pb4" Oct 30 14:21:57.478957 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:57.478903 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-648d5f-54b7c56458-79pb4" podStartSLOduration=1.4788905589999999 podStartE2EDuration="1.478890559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:21:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:21:57.474898636 +0000 UTC m=+1987.219032676" watchObservedRunningTime="2026-10-30 14:21:57.478890559 +0000 UTC m=+1987.223024600" Oct 30 14:21:58.012911 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:21:58.012880 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:21:58.174563 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:58.174524 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:21:59.806034 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:21:59.806002 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:22:00.476078 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:00.476042 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:22:00.606062 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:00.606021 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d74808-c8z6d"] Oct 30 14:22:00.618768 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:00.618735 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d74808-c8z6d"] Oct 30 14:22:00.632768 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:00.632712 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:22:00.852503 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:00.852471 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:22:01.145109 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:01.145004 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:22:02.401371 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:02.401334 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bff13918-6bc7-42b8-86ca-216d6baeb4f6" path="/var/lib/kubelet/pods/bff13918-6bc7-42b8-86ca-216d6baeb4f6/volumes" Oct 30 14:22:02.687509 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:02.687414 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:22:04.559524 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:04.558578 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:22:06.177944 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:06.177904 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:22:06.722654 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:06.722599 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:22:06.879815 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:06.879776 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2b206b-t2tz7"] Oct 30 14:22:06.882895 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:06.882863 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:22:06.887016 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:06.886994 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2b206b-t2tz7"] Oct 30 14:22:07.052629 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:07.052530 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:22:07.416991 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:07.416959 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:22:08.404128 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:08.404075 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c897e27-a3b6-4221-b7b2-7ae5cb159d36" path="/var/lib/kubelet/pods/6c897e27-a3b6-4221-b7b2-7ae5cb159d36/volumes" Oct 30 14:22:08.441763 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:08.441732 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-648d5f-54b7c56458-79pb4" Oct 30 14:22:09.030863 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:09.030827 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:22:09.231607 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:09.231575 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:22:09.389955 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:09.389922 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:22:09.551732 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:09.551630 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:22:10.040202 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:10.040167 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a2835-l59d2"] Oct 30 14:22:10.046129 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:10.046110 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a2835-l59d2" Oct 30 14:22:10.063339 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:10.063310 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:22:10.063458 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:10.063364 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:22:10.066067 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:10.066045 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a2835-l59d2"] Oct 30 14:22:10.151558 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:10.151529 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fp6k\" (UniqueName: \"kubernetes.io/projected/9fa389f0-fd2d-4cd9-a57b-45f00a79dbb8-kube-api-access-5fp6k\") pod \"maas-api-revoke-keys-e2e-api-b-2a2835-l59d2\" (UID: \"9fa389f0-fd2d-4cd9-a57b-45f00a79dbb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a2835-l59d2" Oct 30 14:22:10.151735 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:10.151572 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fa389f0-fd2d-4cd9-a57b-45f00a79dbb8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2a2835-l59d2\" (UID: \"9fa389f0-fd2d-4cd9-a57b-45f00a79dbb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a2835-l59d2" Oct 30 14:22:10.151861 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:10.151838 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fa389f0-fd2d-4cd9-a57b-45f00a79dbb8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2a2835-l59d2\" (UID: \"9fa389f0-fd2d-4cd9-a57b-45f00a79dbb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a2835-l59d2" Oct 30 14:22:10.252904 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:10.252870 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fp6k\" (UniqueName: \"kubernetes.io/projected/9fa389f0-fd2d-4cd9-a57b-45f00a79dbb8-kube-api-access-5fp6k\") pod \"maas-api-revoke-keys-e2e-api-b-2a2835-l59d2\" (UID: \"9fa389f0-fd2d-4cd9-a57b-45f00a79dbb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a2835-l59d2" Oct 30 14:22:10.253063 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:10.252917 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fa389f0-fd2d-4cd9-a57b-45f00a79dbb8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2a2835-l59d2\" (UID: \"9fa389f0-fd2d-4cd9-a57b-45f00a79dbb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a2835-l59d2" Oct 30 14:22:10.253149 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:10.253129 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fa389f0-fd2d-4cd9-a57b-45f00a79dbb8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2a2835-l59d2\" (UID: \"9fa389f0-fd2d-4cd9-a57b-45f00a79dbb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a2835-l59d2" Oct 30 14:22:10.253539 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:10.253521 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fa389f0-fd2d-4cd9-a57b-45f00a79dbb8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2a2835-l59d2\" (UID: \"9fa389f0-fd2d-4cd9-a57b-45f00a79dbb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a2835-l59d2" Oct 30 14:22:10.255578 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:10.255557 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fa389f0-fd2d-4cd9-a57b-45f00a79dbb8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2a2835-l59d2\" (UID: \"9fa389f0-fd2d-4cd9-a57b-45f00a79dbb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a2835-l59d2" Oct 30 14:22:10.278481 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:10.278454 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fp6k\" (UniqueName: \"kubernetes.io/projected/9fa389f0-fd2d-4cd9-a57b-45f00a79dbb8-kube-api-access-5fp6k\") pod \"maas-api-revoke-keys-e2e-api-b-2a2835-l59d2\" (UID: \"9fa389f0-fd2d-4cd9-a57b-45f00a79dbb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a2835-l59d2" Oct 30 14:22:10.359994 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:10.359956 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a2835-l59d2" Oct 30 14:22:10.675973 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:22:10.675938 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fa389f0_fd2d_4cd9_a57b_45f00a79dbb8.slice/crio-75875d7c9c2b0cc869e497562186de6c2c43b7057511fad20142c1d9c7887299 WatchSource:0}: Error finding container 75875d7c9c2b0cc869e497562186de6c2c43b7057511fad20142c1d9c7887299: Status 404 returned error can't find the container with id 75875d7c9c2b0cc869e497562186de6c2c43b7057511fad20142c1d9c7887299 Oct 30 14:22:10.676734 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:10.676706 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a2835-l59d2"] Oct 30 14:22:10.990182 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:10.990150 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:22:11.540077 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:11.540037 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a2835-l59d2" event={"ID":"9fa389f0-fd2d-4cd9-a57b-45f00a79dbb8","Type":"ContainerStarted","Data":"0517059342f6695688d9c422acc67f45aa1a913db0540c180b94530f916d2c98"} Oct 30 14:22:11.540266 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:11.540089 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a2835-l59d2" event={"ID":"9fa389f0-fd2d-4cd9-a57b-45f00a79dbb8","Type":"ContainerStarted","Data":"75875d7c9c2b0cc869e497562186de6c2c43b7057511fad20142c1d9c7887299"} Oct 30 14:22:11.586279 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:11.586225 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a2835-l59d2" podStartSLOduration=1.586212073 podStartE2EDuration="1.586212073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:22:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:22:11.583899153 +0000 UTC m=+2001.328033195" watchObservedRunningTime="2026-10-30 14:22:11.586212073 +0000 UTC m=+2001.330346113" Oct 30 14:22:12.548765 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:12.548732 4982 generic.go:356] "Generic (PLEG): container finished" podID="9fa389f0-fd2d-4cd9-a57b-45f00a79dbb8" containerID="0517059342f6695688d9c422acc67f45aa1a913db0540c180b94530f916d2c98" exitCode=0 Oct 30 14:22:12.549168 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:12.548800 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a2835-l59d2" event={"ID":"9fa389f0-fd2d-4cd9-a57b-45f00a79dbb8","Type":"ContainerDied","Data":"0517059342f6695688d9c422acc67f45aa1a913db0540c180b94530f916d2c98"} Oct 30 14:22:13.041572 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:13.041528 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:22:13.743120 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:13.743095 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a2835-l59d2" Oct 30 14:22:13.804270 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:13.804234 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fa389f0-fd2d-4cd9-a57b-45f00a79dbb8-kube-api-access-5fp6k\" (UniqueName: \"kubernetes.io/projected/9fa389f0-fd2d-4cd9-a57b-45f00a79dbb8-kube-api-access-5fp6k\") pod \"9fa389f0-fd2d-4cd9-a57b-45f00a79dbb8\" (UID: \"9fa389f0-fd2d-4cd9-a57b-45f00a79dbb8\") " Oct 30 14:22:13.804445 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:13.804284 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fa389f0-fd2d-4cd9-a57b-45f00a79dbb8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fa389f0-fd2d-4cd9-a57b-45f00a79dbb8-maas-api-xks-ca\") pod \"9fa389f0-fd2d-4cd9-a57b-45f00a79dbb8\" (UID: \"9fa389f0-fd2d-4cd9-a57b-45f00a79dbb8\") " Oct 30 14:22:13.804522 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:13.804506 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fa389f0-fd2d-4cd9-a57b-45f00a79dbb8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fa389f0-fd2d-4cd9-a57b-45f00a79dbb8-maas-api-openshift-ca\") pod \"9fa389f0-fd2d-4cd9-a57b-45f00a79dbb8\" (UID: \"9fa389f0-fd2d-4cd9-a57b-45f00a79dbb8\") " Oct 30 14:22:13.804999 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:13.804971 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fa389f0-fd2d-4cd9-a57b-45f00a79dbb8-maas-api-openshift-ca" pod "9fa389f0-fd2d-4cd9-a57b-45f00a79dbb8" (UID: "9fa389f0-fd2d-4cd9-a57b-45f00a79dbb8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:22:13.805271 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:13.805250 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fa389f0-fd2d-4cd9-a57b-45f00a79dbb8-maas-api-openshift-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:22:13.807042 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:13.807009 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fa389f0-fd2d-4cd9-a57b-45f00a79dbb8-maas-api-xks-ca" pod "9fa389f0-fd2d-4cd9-a57b-45f00a79dbb8" (UID: "9fa389f0-fd2d-4cd9-a57b-45f00a79dbb8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:22:13.807669 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:13.807611 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fa389f0-fd2d-4cd9-a57b-45f00a79dbb8-kube-api-access-5fp6k" pod "9fa389f0-fd2d-4cd9-a57b-45f00a79dbb8" (UID: "9fa389f0-fd2d-4cd9-a57b-45f00a79dbb8"). InnerVolumeSpecName "kube-api-access-5fp6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:22:13.906855 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:13.906822 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fp6k\" (UniqueName: \"kubernetes.io/projected/9fa389f0-fd2d-4cd9-a57b-45f00a79dbb8-kube-api-access-5fp6k\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:22:13.906855 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:13.906856 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fa389f0-fd2d-4cd9-a57b-45f00a79dbb8-maas-api-xks-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:22:14.071540 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:14.071453 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:22:14.215587 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:14.215550 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:22:14.468402 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:14.468370 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:22:14.665582 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:14.663686 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a2835-l59d2" event={"ID":"9fa389f0-fd2d-4cd9-a57b-45f00a79dbb8","Type":"ContainerDied","Data":"75875d7c9c2b0cc869e497562186de6c2c43b7057511fad20142c1d9c7887299"} Oct 30 14:22:14.665582 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:14.663793 4982 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75875d7c9c2b0cc869e497562186de6c2c43b7057511fad20142c1d9c7887299" Oct 30 14:22:14.665582 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:14.664009 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a2835-l59d2" Oct 30 14:22:14.967597 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:14.967558 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c4b82-8xk6g"] Oct 30 14:22:14.968580 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:14.968554 4982 cpu_manager.go:414] "containerMap: removing container" podUID="9fa389f0-fd2d-4cd9-a57b-45f00a79dbb8" containerName="revoke-keys" Oct 30 14:22:14.968580 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:14.968577 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fa389f0-fd2d-4cd9-a57b-45f00a79dbb8" containerName="revoke-keys" Oct 30 14:22:14.968790 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:14.968749 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fa389f0-fd2d-4cd9-a57b-45f00a79dbb8" containerName="revoke-keys" Oct 30 14:22:14.972732 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:14.972710 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c4b82-8xk6g" Oct 30 14:22:14.979134 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:14.979107 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:22:15.012692 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:15.003571 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:22:15.012692 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:15.004357 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:22:15.012692 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:15.011737 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c4b82-8xk6g"] Oct 30 14:22:15.045101 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:15.045064 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4be44544-6e2f-4959-b0db-56d093d5f6b0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1c4b82-8xk6g\" (UID: \"4be44544-6e2f-4959-b0db-56d093d5f6b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c4b82-8xk6g" Oct 30 14:22:15.045295 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:15.045232 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbp6m\" (UniqueName: \"kubernetes.io/projected/4be44544-6e2f-4959-b0db-56d093d5f6b0-kube-api-access-vbp6m\") pod \"maas-api-revoke-keys-e2e-api-b-1c4b82-8xk6g\" (UID: \"4be44544-6e2f-4959-b0db-56d093d5f6b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c4b82-8xk6g" Oct 30 14:22:15.045445 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:15.045427 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4be44544-6e2f-4959-b0db-56d093d5f6b0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1c4b82-8xk6g\" (UID: \"4be44544-6e2f-4959-b0db-56d093d5f6b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c4b82-8xk6g" Oct 30 14:22:15.147379 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:15.147342 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4be44544-6e2f-4959-b0db-56d093d5f6b0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1c4b82-8xk6g\" (UID: \"4be44544-6e2f-4959-b0db-56d093d5f6b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c4b82-8xk6g" Oct 30 14:22:15.147562 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:15.147450 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4be44544-6e2f-4959-b0db-56d093d5f6b0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1c4b82-8xk6g\" (UID: \"4be44544-6e2f-4959-b0db-56d093d5f6b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c4b82-8xk6g" Oct 30 14:22:15.147630 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:15.147565 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbp6m\" (UniqueName: \"kubernetes.io/projected/4be44544-6e2f-4959-b0db-56d093d5f6b0-kube-api-access-vbp6m\") pod \"maas-api-revoke-keys-e2e-api-b-1c4b82-8xk6g\" (UID: \"4be44544-6e2f-4959-b0db-56d093d5f6b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c4b82-8xk6g" Oct 30 14:22:15.148456 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:15.148431 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4be44544-6e2f-4959-b0db-56d093d5f6b0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1c4b82-8xk6g\" (UID: \"4be44544-6e2f-4959-b0db-56d093d5f6b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c4b82-8xk6g" Oct 30 14:22:15.150077 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:15.150057 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4be44544-6e2f-4959-b0db-56d093d5f6b0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1c4b82-8xk6g\" (UID: \"4be44544-6e2f-4959-b0db-56d093d5f6b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c4b82-8xk6g" Oct 30 14:22:15.173424 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:15.173396 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbp6m\" (UniqueName: \"kubernetes.io/projected/4be44544-6e2f-4959-b0db-56d093d5f6b0-kube-api-access-vbp6m\") pod \"maas-api-revoke-keys-e2e-api-b-1c4b82-8xk6g\" (UID: \"4be44544-6e2f-4959-b0db-56d093d5f6b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c4b82-8xk6g" Oct 30 14:22:15.301359 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:15.301257 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c4b82-8xk6g" Oct 30 14:22:15.611837 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:22:15.611810 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4be44544_6e2f_4959_b0db_56d093d5f6b0.slice/crio-603123d36d6e03299199a3da47bb948c56c13803db3507a074fe604ba64a51a1 WatchSource:0}: Error finding container 603123d36d6e03299199a3da47bb948c56c13803db3507a074fe604ba64a51a1: Status 404 returned error can't find the container with id 603123d36d6e03299199a3da47bb948c56c13803db3507a074fe604ba64a51a1 Oct 30 14:22:15.611946 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:15.611858 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c4b82-8xk6g"] Oct 30 14:22:15.671779 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:15.671750 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c4b82-8xk6g" event={"ID":"4be44544-6e2f-4959-b0db-56d093d5f6b0","Type":"ContainerStarted","Data":"603123d36d6e03299199a3da47bb948c56c13803db3507a074fe604ba64a51a1"} Oct 30 14:22:16.680838 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:16.680806 4982 generic.go:356] "Generic (PLEG): container finished" podID="4be44544-6e2f-4959-b0db-56d093d5f6b0" containerID="dfe3e682b67b312c010fed9debc7182ee3ef806744ae3d6313b8aa9f3fb888b0" exitCode=0 Oct 30 14:22:16.681289 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:16.680843 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c4b82-8xk6g" event={"ID":"4be44544-6e2f-4959-b0db-56d093d5f6b0","Type":"ContainerDied","Data":"dfe3e682b67b312c010fed9debc7182ee3ef806744ae3d6313b8aa9f3fb888b0"} Oct 30 14:22:17.022803 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:17.022765 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:22:17.876730 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:17.876708 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c4b82-8xk6g" Oct 30 14:22:17.989081 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:17.988994 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4be44544-6e2f-4959-b0db-56d093d5f6b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4be44544-6e2f-4959-b0db-56d093d5f6b0-maas-api-xks-ca\") pod \"4be44544-6e2f-4959-b0db-56d093d5f6b0\" (UID: \"4be44544-6e2f-4959-b0db-56d093d5f6b0\") " Oct 30 14:22:17.989081 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:17.989065 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4be44544-6e2f-4959-b0db-56d093d5f6b0-kube-api-access-vbp6m\" (UniqueName: \"kubernetes.io/projected/4be44544-6e2f-4959-b0db-56d093d5f6b0-kube-api-access-vbp6m\") pod \"4be44544-6e2f-4959-b0db-56d093d5f6b0\" (UID: \"4be44544-6e2f-4959-b0db-56d093d5f6b0\") " Oct 30 14:22:17.989306 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:17.989188 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4be44544-6e2f-4959-b0db-56d093d5f6b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4be44544-6e2f-4959-b0db-56d093d5f6b0-maas-api-openshift-ca\") pod \"4be44544-6e2f-4959-b0db-56d093d5f6b0\" (UID: \"4be44544-6e2f-4959-b0db-56d093d5f6b0\") " Oct 30 14:22:17.989732 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:17.989707 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4be44544-6e2f-4959-b0db-56d093d5f6b0-maas-api-openshift-ca" pod "4be44544-6e2f-4959-b0db-56d093d5f6b0" (UID: "4be44544-6e2f-4959-b0db-56d093d5f6b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:22:17.991543 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:17.991511 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4be44544-6e2f-4959-b0db-56d093d5f6b0-maas-api-xks-ca" pod "4be44544-6e2f-4959-b0db-56d093d5f6b0" (UID: "4be44544-6e2f-4959-b0db-56d093d5f6b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:22:17.991683 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:17.991606 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4be44544-6e2f-4959-b0db-56d093d5f6b0-kube-api-access-vbp6m" pod "4be44544-6e2f-4959-b0db-56d093d5f6b0" (UID: "4be44544-6e2f-4959-b0db-56d093d5f6b0"). InnerVolumeSpecName "kube-api-access-vbp6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:22:18.091121 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:18.091087 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4be44544-6e2f-4959-b0db-56d093d5f6b0-maas-api-xks-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:22:18.091121 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:18.091120 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbp6m\" (UniqueName: \"kubernetes.io/projected/4be44544-6e2f-4959-b0db-56d093d5f6b0-kube-api-access-vbp6m\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:22:18.091288 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:18.091134 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4be44544-6e2f-4959-b0db-56d093d5f6b0-maas-api-openshift-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:22:18.698082 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:18.697986 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c4b82-8xk6g" event={"ID":"4be44544-6e2f-4959-b0db-56d093d5f6b0","Type":"ContainerDied","Data":"603123d36d6e03299199a3da47bb948c56c13803db3507a074fe604ba64a51a1"} Oct 30 14:22:18.698082 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:18.698008 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c4b82-8xk6g" Oct 30 14:22:18.698082 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:18.698074 4982 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="603123d36d6e03299199a3da47bb948c56c13803db3507a074fe604ba64a51a1" Oct 30 14:22:18.974698 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:18.974582 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:22:20.644135 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:20.644096 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:22:20.794952 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:20.794909 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:22:20.954509 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:20.954421 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:22:22.946174 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:22.946137 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:22:24.823996 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:24.823960 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:22:25.575887 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:25.575855 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:22:25.720948 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:25.720922 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:22:26.850390 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:26.850352 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:22:28.014694 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:22:28.014658 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:22:28.844012 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:28.843949 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:22:30.629835 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:30.629802 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:22:32.625920 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:32.625884 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:22:34.434376 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:34.434297 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:22:34.667228 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:34.667195 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:22:36.539434 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:36.539382 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:22:38.037818 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:38.037784 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2e8b995-lslrj"] Oct 30 14:22:38.050479 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:38.050453 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2e8b995-lslrj"] Oct 30 14:22:38.400281 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:38.400253 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce6f63ba-b1b7-4c90-a89c-a8d5e1c1e50a" path="/var/lib/kubelet/pods/ce6f63ba-b1b7-4c90-a89c-a8d5e1c1e50a/volumes" Oct 30 14:22:38.765177 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:38.765096 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:22:39.441324 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:39.441288 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:22:40.590870 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:40.590839 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:22:42.425529 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:42.425494 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:22:44.316695 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:44.316661 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:22:44.459812 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:44.459731 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:22:46.206364 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:46.206297 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:22:47.749593 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:47.749538 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:22:48.297423 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:48.297388 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:22:48.419483 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:48.419451 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b5ea88-vt6hg"] Oct 30 14:22:48.432447 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:48.432415 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b5ea88-vt6hg"] Oct 30 14:22:48.449265 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:48.449224 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:22:48.599673 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:48.599601 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:22:50.410322 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:50.410283 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2369048-24d1-4882-bb7e-a3122a7a85bc" path="/var/lib/kubelet/pods/b2369048-24d1-4882-bb7e-a3122a7a85bc/volumes" Oct 30 14:22:50.791589 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:50.791483 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:22:50.859806 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:50.859773 4982 scope.go:122] "RemoveContainer" containerID="71f3493b5caff4cbf22b653bec40cdfea317252cd104b3dd2cd1bc6de8086b5c" Oct 30 14:22:52.966155 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:52.966096 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:22:54.545029 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:54.544989 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:22:55.140458 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:55.140417 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:22:55.293189 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:55.293150 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:22:55.312663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:55.302436 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aab379-lbc58"] Oct 30 14:22:55.326540 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:55.326506 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aab379-lbc58"] Oct 30 14:22:55.445054 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:55.444973 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:22:55.822461 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:55.822431 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:22:56.403160 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:56.403125 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1010904-6f90-4efb-ab8b-30afbe7187a4" path="/var/lib/kubelet/pods/f1010904-6f90-4efb-ab8b-30afbe7187a4/volumes" Oct 30 14:22:57.456674 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:57.456605 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:22:57.892582 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:57.892546 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc"] Oct 30 14:22:57.892773 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:57.892751 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc" podUID="6a3afaea-c040-4642-8632-6b5a0dca807c" containerName="istio-proxy" containerID="cri-o://8b46911b829d0492fd3e4bdf74b8302dfd5918d5a4ba8a5dacadd15aa77c02a1" gracePeriod=30 Oct 30 14:22:58.050926 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:22:58.050880 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:22:58.950313 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:58.950281 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:22:59.203845 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:22:59.203762 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:00.232511 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:00.232475 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:00.373577 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:00.373543 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:00.557382 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:00.557352 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:00.713067 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:00.713036 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:01.250413 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:01.250377 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:03.157740 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:03.157662 4982 generic.go:356] "Generic (PLEG): container finished" podID="6a3afaea-c040-4642-8632-6b5a0dca807c" containerID="8b46911b829d0492fd3e4bdf74b8302dfd5918d5a4ba8a5dacadd15aa77c02a1" exitCode=0 Oct 30 14:23:03.158146 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:03.157772 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc" event={"ID":"6a3afaea-c040-4642-8632-6b5a0dca807c","Type":"ContainerDied","Data":"8b46911b829d0492fd3e4bdf74b8302dfd5918d5a4ba8a5dacadd15aa77c02a1"} Oct 30 14:23:03.186076 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:03.186051 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc" Oct 30 14:23:03.212758 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:03.212727 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a3afaea-c040-4642-8632-6b5a0dca807c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a3afaea-c040-4642-8632-6b5a0dca807c-workload-certs\") pod \"6a3afaea-c040-4642-8632-6b5a0dca807c\" (UID: \"6a3afaea-c040-4642-8632-6b5a0dca807c\") " Oct 30 14:23:03.212910 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:03.212789 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a3afaea-c040-4642-8632-6b5a0dca807c-istio-token\" (UniqueName: \"kubernetes.io/projected/6a3afaea-c040-4642-8632-6b5a0dca807c-istio-token\") pod \"6a3afaea-c040-4642-8632-6b5a0dca807c\" (UID: \"6a3afaea-c040-4642-8632-6b5a0dca807c\") " Oct 30 14:23:03.212910 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:03.212843 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6a3afaea-c040-4642-8632-6b5a0dca807c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a3afaea-c040-4642-8632-6b5a0dca807c-istio-podinfo\") pod \"6a3afaea-c040-4642-8632-6b5a0dca807c\" (UID: \"6a3afaea-c040-4642-8632-6b5a0dca807c\") " Oct 30 14:23:03.213060 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:03.213031 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a3afaea-c040-4642-8632-6b5a0dca807c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a3afaea-c040-4642-8632-6b5a0dca807c-istio-data\") pod \"6a3afaea-c040-4642-8632-6b5a0dca807c\" (UID: \"6a3afaea-c040-4642-8632-6b5a0dca807c\") " Oct 30 14:23:03.213133 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:03.213102 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a3afaea-c040-4642-8632-6b5a0dca807c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a3afaea-c040-4642-8632-6b5a0dca807c-workload-socket\") pod \"6a3afaea-c040-4642-8632-6b5a0dca807c\" (UID: \"6a3afaea-c040-4642-8632-6b5a0dca807c\") " Oct 30 14:23:03.213192 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:03.213164 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a3afaea-c040-4642-8632-6b5a0dca807c-kube-api-access-f8qth\" (UniqueName: \"kubernetes.io/projected/6a3afaea-c040-4642-8632-6b5a0dca807c-kube-api-access-f8qth\") pod \"6a3afaea-c040-4642-8632-6b5a0dca807c\" (UID: \"6a3afaea-c040-4642-8632-6b5a0dca807c\") " Oct 30 14:23:03.213246 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:03.213223 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a3afaea-c040-4642-8632-6b5a0dca807c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a3afaea-c040-4642-8632-6b5a0dca807c-credential-socket\") pod \"6a3afaea-c040-4642-8632-6b5a0dca807c\" (UID: \"6a3afaea-c040-4642-8632-6b5a0dca807c\") " Oct 30 14:23:03.213299 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:03.213269 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a3afaea-c040-4642-8632-6b5a0dca807c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a3afaea-c040-4642-8632-6b5a0dca807c-istio-envoy\") pod \"6a3afaea-c040-4642-8632-6b5a0dca807c\" (UID: \"6a3afaea-c040-4642-8632-6b5a0dca807c\") " Oct 30 14:23:03.213349 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:03.213097 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a3afaea-c040-4642-8632-6b5a0dca807c-workload-certs" pod "6a3afaea-c040-4642-8632-6b5a0dca807c" (UID: "6a3afaea-c040-4642-8632-6b5a0dca807c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:23:03.213349 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:03.213297 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a3afaea-c040-4642-8632-6b5a0dca807c-istio-data" pod "6a3afaea-c040-4642-8632-6b5a0dca807c" (UID: "6a3afaea-c040-4642-8632-6b5a0dca807c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:23:03.213467 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:03.213435 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a3afaea-c040-4642-8632-6b5a0dca807c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a3afaea-c040-4642-8632-6b5a0dca807c-istiod-ca-cert\") pod \"6a3afaea-c040-4642-8632-6b5a0dca807c\" (UID: \"6a3afaea-c040-4642-8632-6b5a0dca807c\") " Oct 30 14:23:03.213467 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:03.213452 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a3afaea-c040-4642-8632-6b5a0dca807c-workload-socket" pod "6a3afaea-c040-4642-8632-6b5a0dca807c" (UID: "6a3afaea-c040-4642-8632-6b5a0dca807c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:23:03.213612 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:03.213588 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a3afaea-c040-4642-8632-6b5a0dca807c-credential-socket" pod "6a3afaea-c040-4642-8632-6b5a0dca807c" (UID: "6a3afaea-c040-4642-8632-6b5a0dca807c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:23:03.214269 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:03.214240 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a3afaea-c040-4642-8632-6b5a0dca807c-istiod-ca-cert" pod "6a3afaea-c040-4642-8632-6b5a0dca807c" (UID: "6a3afaea-c040-4642-8632-6b5a0dca807c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:23:03.214604 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:03.214586 4982 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a3afaea-c040-4642-8632-6b5a0dca807c-istio-data\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:23:03.214728 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:03.214606 4982 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a3afaea-c040-4642-8632-6b5a0dca807c-workload-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:23:03.214728 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:03.214633 4982 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a3afaea-c040-4642-8632-6b5a0dca807c-credential-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:23:03.214728 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:03.214663 4982 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a3afaea-c040-4642-8632-6b5a0dca807c-istiod-ca-cert\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:23:03.214728 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:03.214677 4982 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a3afaea-c040-4642-8632-6b5a0dca807c-workload-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:23:03.215428 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:03.215393 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a3afaea-c040-4642-8632-6b5a0dca807c-istio-token" pod "6a3afaea-c040-4642-8632-6b5a0dca807c" (UID: "6a3afaea-c040-4642-8632-6b5a0dca807c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:23:03.215931 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:03.215910 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a3afaea-c040-4642-8632-6b5a0dca807c-istio-envoy" pod "6a3afaea-c040-4642-8632-6b5a0dca807c" (UID: "6a3afaea-c040-4642-8632-6b5a0dca807c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:23:03.216213 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:03.216184 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a3afaea-c040-4642-8632-6b5a0dca807c-kube-api-access-f8qth" pod "6a3afaea-c040-4642-8632-6b5a0dca807c" (UID: "6a3afaea-c040-4642-8632-6b5a0dca807c"). InnerVolumeSpecName "kube-api-access-f8qth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:23:03.218758 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:03.218738 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6a3afaea-c040-4642-8632-6b5a0dca807c-istio-podinfo" pod "6a3afaea-c040-4642-8632-6b5a0dca807c" (UID: "6a3afaea-c040-4642-8632-6b5a0dca807c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:23:03.230716 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:03.230683 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:03.316435 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:03.316384 4982 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a3afaea-c040-4642-8632-6b5a0dca807c-istio-token\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:23:03.316435 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:03.316435 4982 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a3afaea-c040-4642-8632-6b5a0dca807c-istio-podinfo\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:23:03.316705 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:03.316449 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8qth\" (UniqueName: \"kubernetes.io/projected/6a3afaea-c040-4642-8632-6b5a0dca807c-kube-api-access-f8qth\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:23:03.316705 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:03.316465 4982 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a3afaea-c040-4642-8632-6b5a0dca807c-istio-envoy\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:23:03.392554 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:03.392497 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:03.724006 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:03.723957 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:04.167410 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:04.167374 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc" Oct 30 14:23:04.167410 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:04.167386 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc" event={"ID":"6a3afaea-c040-4642-8632-6b5a0dca807c","Type":"ContainerDied","Data":"63981d4c4da5f27462823e1737f5e998e243f3c1c2217f902ac61e01082f75c0"} Oct 30 14:23:04.167901 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:04.167446 4982 scope.go:122] "RemoveContainer" containerID="8b46911b829d0492fd3e4bdf74b8302dfd5918d5a4ba8a5dacadd15aa77c02a1" Oct 30 14:23:04.232498 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:04.232466 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:04.238845 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:04.238816 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc"] Oct 30 14:23:04.255214 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:04.255174 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-5hnmc"] Oct 30 14:23:04.402000 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:04.401963 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a3afaea-c040-4642-8632-6b5a0dca807c" path="/var/lib/kubelet/pods/6a3afaea-c040-4642-8632-6b5a0dca807c/volumes" Oct 30 14:23:04.533713 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:04.533615 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:05.016514 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:05.016479 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:05.512541 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:05.512511 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:05.721765 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:05.708232 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:05.879248 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:05.879213 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:06.348989 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:06.348954 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5"] Oct 30 14:23:06.349814 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:06.349787 4982 cpu_manager.go:414] "containerMap: removing container" podUID="6a3afaea-c040-4642-8632-6b5a0dca807c" containerName="istio-proxy" Oct 30 14:23:06.350024 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:06.349827 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a3afaea-c040-4642-8632-6b5a0dca807c" containerName="istio-proxy" Oct 30 14:23:06.350024 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:06.349858 4982 cpu_manager.go:414] "containerMap: removing container" podUID="4be44544-6e2f-4959-b0db-56d093d5f6b0" containerName="revoke-keys" Oct 30 14:23:06.350024 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:06.349866 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4be44544-6e2f-4959-b0db-56d093d5f6b0" containerName="revoke-keys" Oct 30 14:23:06.350024 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:06.350146 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a3afaea-c040-4642-8632-6b5a0dca807c" containerName="istio-proxy" Oct 30 14:23:06.350024 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:06.350169 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="4be44544-6e2f-4959-b0db-56d093d5f6b0" containerName="revoke-keys" Oct 30 14:23:06.355754 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:06.355730 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5" Oct 30 14:23:06.389016 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:06.386850 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5"] Oct 30 14:23:06.440406 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:06.440363 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d85ef9d1-c82a-4957-a041-6995e428f4a1-istio-podinfo\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5\" (UID: \"d85ef9d1-c82a-4957-a041-6995e428f4a1\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5" Oct 30 14:23:06.440595 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:06.440424 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d85ef9d1-c82a-4957-a041-6995e428f4a1-istio-token\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5\" (UID: \"d85ef9d1-c82a-4957-a041-6995e428f4a1\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5" Oct 30 14:23:06.440595 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:06.440477 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d85ef9d1-c82a-4957-a041-6995e428f4a1-istio-data\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5\" (UID: \"d85ef9d1-c82a-4957-a041-6995e428f4a1\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5" Oct 30 14:23:06.440595 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:06.440524 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d85ef9d1-c82a-4957-a041-6995e428f4a1-workload-certs\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5\" (UID: \"d85ef9d1-c82a-4957-a041-6995e428f4a1\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5" Oct 30 14:23:06.440880 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:06.440595 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-278qx\" (UniqueName: \"kubernetes.io/projected/d85ef9d1-c82a-4957-a041-6995e428f4a1-kube-api-access-278qx\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5\" (UID: \"d85ef9d1-c82a-4957-a041-6995e428f4a1\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5" Oct 30 14:23:06.440880 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:06.440704 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d85ef9d1-c82a-4957-a041-6995e428f4a1-istio-envoy\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5\" (UID: \"d85ef9d1-c82a-4957-a041-6995e428f4a1\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5" Oct 30 14:23:06.440880 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:06.440731 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85ef9d1-c82a-4957-a041-6995e428f4a1-credential-socket\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5\" (UID: \"d85ef9d1-c82a-4957-a041-6995e428f4a1\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5" Oct 30 14:23:06.440880 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:06.440790 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d85ef9d1-c82a-4957-a041-6995e428f4a1-istiod-ca-cert\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5\" (UID: \"d85ef9d1-c82a-4957-a041-6995e428f4a1\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5" Oct 30 14:23:06.440880 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:06.440846 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85ef9d1-c82a-4957-a041-6995e428f4a1-workload-socket\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5\" (UID: \"d85ef9d1-c82a-4957-a041-6995e428f4a1\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5" Oct 30 14:23:06.449208 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:06.449175 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-ea9ed2-openshift-default-dockercfg-fk8hk\"" Oct 30 14:23:06.542141 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:06.542107 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d85ef9d1-c82a-4957-a041-6995e428f4a1-istiod-ca-cert\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5\" (UID: \"d85ef9d1-c82a-4957-a041-6995e428f4a1\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5" Oct 30 14:23:06.542613 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:06.542183 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85ef9d1-c82a-4957-a041-6995e428f4a1-workload-socket\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5\" (UID: \"d85ef9d1-c82a-4957-a041-6995e428f4a1\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5" Oct 30 14:23:06.542613 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:06.542302 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d85ef9d1-c82a-4957-a041-6995e428f4a1-istio-podinfo\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5\" (UID: \"d85ef9d1-c82a-4957-a041-6995e428f4a1\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5" Oct 30 14:23:06.542613 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:06.542328 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d85ef9d1-c82a-4957-a041-6995e428f4a1-istio-token\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5\" (UID: \"d85ef9d1-c82a-4957-a041-6995e428f4a1\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5" Oct 30 14:23:06.542613 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:06.542377 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d85ef9d1-c82a-4957-a041-6995e428f4a1-istio-data\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5\" (UID: \"d85ef9d1-c82a-4957-a041-6995e428f4a1\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5" Oct 30 14:23:06.542904 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:06.542617 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85ef9d1-c82a-4957-a041-6995e428f4a1-workload-socket\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5\" (UID: \"d85ef9d1-c82a-4957-a041-6995e428f4a1\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5" Oct 30 14:23:06.542904 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:06.542698 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d85ef9d1-c82a-4957-a041-6995e428f4a1-istio-data\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5\" (UID: \"d85ef9d1-c82a-4957-a041-6995e428f4a1\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5" Oct 30 14:23:06.542904 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:06.542806 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d85ef9d1-c82a-4957-a041-6995e428f4a1-workload-certs\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5\" (UID: \"d85ef9d1-c82a-4957-a041-6995e428f4a1\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5" Oct 30 14:23:06.542904 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:06.542860 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d85ef9d1-c82a-4957-a041-6995e428f4a1-istiod-ca-cert\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5\" (UID: \"d85ef9d1-c82a-4957-a041-6995e428f4a1\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5" Oct 30 14:23:06.543110 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:06.542922 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-278qx\" (UniqueName: \"kubernetes.io/projected/d85ef9d1-c82a-4957-a041-6995e428f4a1-kube-api-access-278qx\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5\" (UID: \"d85ef9d1-c82a-4957-a041-6995e428f4a1\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5" Oct 30 14:23:06.543110 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:06.543007 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d85ef9d1-c82a-4957-a041-6995e428f4a1-istio-envoy\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5\" (UID: \"d85ef9d1-c82a-4957-a041-6995e428f4a1\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5" Oct 30 14:23:06.543110 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:06.543038 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85ef9d1-c82a-4957-a041-6995e428f4a1-credential-socket\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5\" (UID: \"d85ef9d1-c82a-4957-a041-6995e428f4a1\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5" Oct 30 14:23:06.543110 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:06.543077 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d85ef9d1-c82a-4957-a041-6995e428f4a1-workload-certs\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5\" (UID: \"d85ef9d1-c82a-4957-a041-6995e428f4a1\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5" Oct 30 14:23:06.543430 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:06.543408 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85ef9d1-c82a-4957-a041-6995e428f4a1-credential-socket\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5\" (UID: \"d85ef9d1-c82a-4957-a041-6995e428f4a1\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5" Oct 30 14:23:06.545213 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:06.545191 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d85ef9d1-c82a-4957-a041-6995e428f4a1-istio-envoy\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5\" (UID: \"d85ef9d1-c82a-4957-a041-6995e428f4a1\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5" Oct 30 14:23:06.545700 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:06.545684 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d85ef9d1-c82a-4957-a041-6995e428f4a1-istio-podinfo\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5\" (UID: \"d85ef9d1-c82a-4957-a041-6995e428f4a1\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5" Oct 30 14:23:06.568505 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:06.568238 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d85ef9d1-c82a-4957-a041-6995e428f4a1-istio-token\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5\" (UID: \"d85ef9d1-c82a-4957-a041-6995e428f4a1\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5" Oct 30 14:23:06.570130 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:06.570102 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-278qx\" (UniqueName: \"kubernetes.io/projected/d85ef9d1-c82a-4957-a041-6995e428f4a1-kube-api-access-278qx\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5\" (UID: \"d85ef9d1-c82a-4957-a041-6995e428f4a1\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5" Oct 30 14:23:06.683188 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:06.683103 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5" Oct 30 14:23:06.691404 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:06.691375 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:07.019795 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:07.019466 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5"] Oct 30 14:23:07.022407 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:23:07.022372 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd85ef9d1_c82a_4957_a041_6995e428f4a1.slice/crio-3d39ea8214fb74b6dba10cd08918453ae72fb58c342a43a7e57e7d4a6481620d WatchSource:0}: Error finding container 3d39ea8214fb74b6dba10cd08918453ae72fb58c342a43a7e57e7d4a6481620d: Status 404 returned error can't find the container with id 3d39ea8214fb74b6dba10cd08918453ae72fb58c342a43a7e57e7d4a6481620d Oct 30 14:23:07.026248 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:07.026212 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:23:07.026336 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:07.026292 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:23:07.026377 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:07.026332 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:23:07.139679 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:07.139149 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:07.287881 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:07.287791 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:07.324372 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:07.324334 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5" event={"ID":"d85ef9d1-c82a-4957-a041-6995e428f4a1","Type":"ContainerStarted","Data":"c9e6f9a5bc9e3be4dba76192a17491b9b76115568e29499bf60b91ee0ce47016"} Oct 30 14:23:07.324561 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:07.324394 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5" event={"ID":"d85ef9d1-c82a-4957-a041-6995e428f4a1","Type":"ContainerStarted","Data":"3d39ea8214fb74b6dba10cd08918453ae72fb58c342a43a7e57e7d4a6481620d"} Oct 30 14:23:07.380661 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:07.378342 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5" podStartSLOduration=1.378321733 podStartE2EDuration="1.378321733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:23:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:23:07.377057712 +0000 UTC m=+2057.121191752" watchObservedRunningTime="2026-10-30 14:23:07.378321733 +0000 UTC m=+2057.122455775" Oct 30 14:23:07.472679 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:07.472390 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:07.687212 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:07.687162 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5" Oct 30 14:23:07.695361 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:07.695236 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:08.046706 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:08.046610 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4bf33a-sgtxq"] Oct 30 14:23:08.051091 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:08.051066 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4bf33a-sgtxq" Oct 30 14:23:08.072677 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:08.072628 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4bf33a-sgtxq"] Oct 30 14:23:08.130665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:08.119873 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:23:08.130665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:08.120045 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:23:08.130665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:08.122677 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt6rl\" (UniqueName: \"kubernetes.io/projected/69ad8491-34f8-494b-89ba-a9ddd63cac3d-kube-api-access-tt6rl\") pod \"maas-api-revoke-keys-e2e-api-a-4bf33a-sgtxq\" (UID: \"69ad8491-34f8-494b-89ba-a9ddd63cac3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4bf33a-sgtxq" Oct 30 14:23:08.130665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:08.122871 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69ad8491-34f8-494b-89ba-a9ddd63cac3d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4bf33a-sgtxq\" (UID: \"69ad8491-34f8-494b-89ba-a9ddd63cac3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4bf33a-sgtxq" Oct 30 14:23:08.130665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:08.122934 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69ad8491-34f8-494b-89ba-a9ddd63cac3d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4bf33a-sgtxq\" (UID: \"69ad8491-34f8-494b-89ba-a9ddd63cac3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4bf33a-sgtxq" Oct 30 14:23:08.223864 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:08.223830 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69ad8491-34f8-494b-89ba-a9ddd63cac3d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4bf33a-sgtxq\" (UID: \"69ad8491-34f8-494b-89ba-a9ddd63cac3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4bf33a-sgtxq" Oct 30 14:23:08.224068 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:08.223892 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69ad8491-34f8-494b-89ba-a9ddd63cac3d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4bf33a-sgtxq\" (UID: \"69ad8491-34f8-494b-89ba-a9ddd63cac3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4bf33a-sgtxq" Oct 30 14:23:08.224068 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:08.223980 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt6rl\" (UniqueName: \"kubernetes.io/projected/69ad8491-34f8-494b-89ba-a9ddd63cac3d-kube-api-access-tt6rl\") pod \"maas-api-revoke-keys-e2e-api-a-4bf33a-sgtxq\" (UID: \"69ad8491-34f8-494b-89ba-a9ddd63cac3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4bf33a-sgtxq" Oct 30 14:23:08.224635 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:08.224613 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69ad8491-34f8-494b-89ba-a9ddd63cac3d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4bf33a-sgtxq\" (UID: \"69ad8491-34f8-494b-89ba-a9ddd63cac3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4bf33a-sgtxq" Oct 30 14:23:08.226361 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:08.226340 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69ad8491-34f8-494b-89ba-a9ddd63cac3d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4bf33a-sgtxq\" (UID: \"69ad8491-34f8-494b-89ba-a9ddd63cac3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4bf33a-sgtxq" Oct 30 14:23:08.249280 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:08.249254 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt6rl\" (UniqueName: \"kubernetes.io/projected/69ad8491-34f8-494b-89ba-a9ddd63cac3d-kube-api-access-tt6rl\") pod \"maas-api-revoke-keys-e2e-api-a-4bf33a-sgtxq\" (UID: \"69ad8491-34f8-494b-89ba-a9ddd63cac3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4bf33a-sgtxq" Oct 30 14:23:08.367089 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:08.367053 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4bf33a-sgtxq" Oct 30 14:23:08.367992 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:08.367962 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:08.699392 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:08.699305 4982 patch_prober.go:29] interesting pod/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 14:23:08.699982 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:08.699950 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5" podUID="d85ef9d1-c82a-4957-a041-6995e428f4a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 14:23:08.731855 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:08.731430 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4bf33a-sgtxq"] Oct 30 14:23:08.733336 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:23:08.733309 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69ad8491_34f8_494b_89ba_a9ddd63cac3d.slice/crio-95d165e4557c01c6f9f86fdadd27fcb7ce1bccf237dddf2751ff73a3055018c4 WatchSource:0}: Error finding container 95d165e4557c01c6f9f86fdadd27fcb7ce1bccf237dddf2751ff73a3055018c4: Status 404 returned error can't find the container with id 95d165e4557c01c6f9f86fdadd27fcb7ce1bccf237dddf2751ff73a3055018c4 Oct 30 14:23:08.840686 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:08.839804 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:09.165926 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:09.165872 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:09.350869 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:09.350755 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:09.362293 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:09.362158 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4bf33a-sgtxq" event={"ID":"69ad8491-34f8-494b-89ba-a9ddd63cac3d","Type":"ContainerStarted","Data":"fadcccdc9cfc6ef91e7e7ca4cfa445a651cb77718e29f3454a37414540e3de3c"} Oct 30 14:23:09.362293 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:09.362214 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4bf33a-sgtxq" event={"ID":"69ad8491-34f8-494b-89ba-a9ddd63cac3d","Type":"ContainerStarted","Data":"95d165e4557c01c6f9f86fdadd27fcb7ce1bccf237dddf2751ff73a3055018c4"} Oct 30 14:23:09.409716 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:09.408997 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4bf33a-sgtxq" podStartSLOduration=1.40897613 podStartE2EDuration="1.40897613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:23:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:23:09.407562612 +0000 UTC m=+2059.151696654" watchObservedRunningTime="2026-10-30 14:23:09.40897613 +0000 UTC m=+2059.153110171" Oct 30 14:23:09.515527 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:09.515483 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:09.687000 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:09.686912 4982 patch_prober.go:29] interesting pod/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 14:23:09.687275 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:09.687250 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5" podUID="d85ef9d1-c82a-4957-a041-6995e428f4a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 14:23:09.743523 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:09.743472 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:10.369886 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:10.369851 4982 generic.go:356] "Generic (PLEG): container finished" podID="69ad8491-34f8-494b-89ba-a9ddd63cac3d" containerID="fadcccdc9cfc6ef91e7e7ca4cfa445a651cb77718e29f3454a37414540e3de3c" exitCode=0 Oct 30 14:23:10.370380 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:10.369898 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4bf33a-sgtxq" event={"ID":"69ad8491-34f8-494b-89ba-a9ddd63cac3d","Type":"ContainerDied","Data":"fadcccdc9cfc6ef91e7e7ca4cfa445a651cb77718e29f3454a37414540e3de3c"} Oct 30 14:23:10.684717 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:10.684009 4982 patch_prober.go:29] interesting pod/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 14:23:10.684717 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:10.684069 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5" podUID="d85ef9d1-c82a-4957-a041-6995e428f4a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 14:23:11.121869 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:11.121809 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:11.305562 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:11.305526 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:11.687658 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:11.683984 4982 patch_prober.go:29] interesting pod/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 14:23:11.687932 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:11.687900 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5" podUID="d85ef9d1-c82a-4957-a041-6995e428f4a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 14:23:11.969336 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:11.969289 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4bf33a-sgtxq" Oct 30 14:23:12.027658 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:12.018554 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:12.143545 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:12.142168 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69ad8491-34f8-494b-89ba-a9ddd63cac3d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69ad8491-34f8-494b-89ba-a9ddd63cac3d-maas-api-xks-ca\") pod \"69ad8491-34f8-494b-89ba-a9ddd63cac3d\" (UID: \"69ad8491-34f8-494b-89ba-a9ddd63cac3d\") " Oct 30 14:23:12.143545 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:12.142310 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69ad8491-34f8-494b-89ba-a9ddd63cac3d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69ad8491-34f8-494b-89ba-a9ddd63cac3d-maas-api-openshift-ca\") pod \"69ad8491-34f8-494b-89ba-a9ddd63cac3d\" (UID: \"69ad8491-34f8-494b-89ba-a9ddd63cac3d\") " Oct 30 14:23:12.143545 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:12.142364 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69ad8491-34f8-494b-89ba-a9ddd63cac3d-kube-api-access-tt6rl\" (UniqueName: \"kubernetes.io/projected/69ad8491-34f8-494b-89ba-a9ddd63cac3d-kube-api-access-tt6rl\") pod \"69ad8491-34f8-494b-89ba-a9ddd63cac3d\" (UID: \"69ad8491-34f8-494b-89ba-a9ddd63cac3d\") " Oct 30 14:23:12.144190 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:12.144117 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69ad8491-34f8-494b-89ba-a9ddd63cac3d-maas-api-openshift-ca" pod "69ad8491-34f8-494b-89ba-a9ddd63cac3d" (UID: "69ad8491-34f8-494b-89ba-a9ddd63cac3d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:23:12.151800 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:12.150204 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69ad8491-34f8-494b-89ba-a9ddd63cac3d-kube-api-access-tt6rl" pod "69ad8491-34f8-494b-89ba-a9ddd63cac3d" (UID: "69ad8491-34f8-494b-89ba-a9ddd63cac3d"). InnerVolumeSpecName "kube-api-access-tt6rl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:23:12.152172 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:12.152131 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69ad8491-34f8-494b-89ba-a9ddd63cac3d-maas-api-xks-ca" pod "69ad8491-34f8-494b-89ba-a9ddd63cac3d" (UID: "69ad8491-34f8-494b-89ba-a9ddd63cac3d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:23:12.244335 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:12.244256 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69ad8491-34f8-494b-89ba-a9ddd63cac3d-maas-api-xks-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:23:12.244335 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:12.244287 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69ad8491-34f8-494b-89ba-a9ddd63cac3d-maas-api-openshift-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:23:12.244335 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:12.244298 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tt6rl\" (UniqueName: \"kubernetes.io/projected/69ad8491-34f8-494b-89ba-a9ddd63cac3d-kube-api-access-tt6rl\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:23:12.687767 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:12.687727 4982 patch_prober.go:29] interesting pod/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 14:23:12.688027 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:12.688000 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5" podUID="d85ef9d1-c82a-4957-a041-6995e428f4a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 14:23:12.727971 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:12.727929 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4bf33a-sgtxq" event={"ID":"69ad8491-34f8-494b-89ba-a9ddd63cac3d","Type":"ContainerDied","Data":"95d165e4557c01c6f9f86fdadd27fcb7ce1bccf237dddf2751ff73a3055018c4"} Oct 30 14:23:12.728147 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:12.727992 4982 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95d165e4557c01c6f9f86fdadd27fcb7ce1bccf237dddf2751ff73a3055018c4" Oct 30 14:23:12.728147 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:12.728061 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4bf33a-sgtxq" Oct 30 14:23:13.134771 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:13.134737 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx"] Oct 30 14:23:13.134964 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:13.134927 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx" podUID="d5162bf6-1ca4-4925-aa7d-0275e3881ce8" containerName="istio-proxy" containerID="cri-o://95a2c47dd418c2eb1f7033531d53ca7a70bb428f60d42f89cca9e09040d90380" gracePeriod=30 Oct 30 14:23:13.659828 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:13.659783 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:13.803559 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:13.803527 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5" Oct 30 14:23:13.805067 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:13.805044 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5" Oct 30 14:23:13.806906 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:13.806885 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5" Oct 30 14:23:13.813830 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:13.813777 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:14.378507 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:14.378470 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:15.543573 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:15.543532 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:15.836674 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:15.823988 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:16.382120 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:16.382082 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:18.439846 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:18.439822 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx" Oct 30 14:23:18.587862 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:18.587827 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5162bf6-1ca4-4925-aa7d-0275e3881ce8-kube-api-access-48cmt\" (UniqueName: \"kubernetes.io/projected/d5162bf6-1ca4-4925-aa7d-0275e3881ce8-kube-api-access-48cmt\") pod \"d5162bf6-1ca4-4925-aa7d-0275e3881ce8\" (UID: \"d5162bf6-1ca4-4925-aa7d-0275e3881ce8\") " Oct 30 14:23:18.588027 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:18.587891 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5162bf6-1ca4-4925-aa7d-0275e3881ce8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5162bf6-1ca4-4925-aa7d-0275e3881ce8-istiod-ca-cert\") pod \"d5162bf6-1ca4-4925-aa7d-0275e3881ce8\" (UID: \"d5162bf6-1ca4-4925-aa7d-0275e3881ce8\") " Oct 30 14:23:18.588027 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:18.587937 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5162bf6-1ca4-4925-aa7d-0275e3881ce8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5162bf6-1ca4-4925-aa7d-0275e3881ce8-credential-socket\") pod \"d5162bf6-1ca4-4925-aa7d-0275e3881ce8\" (UID: \"d5162bf6-1ca4-4925-aa7d-0275e3881ce8\") " Oct 30 14:23:18.588027 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:18.587982 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5162bf6-1ca4-4925-aa7d-0275e3881ce8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5162bf6-1ca4-4925-aa7d-0275e3881ce8-workload-certs\") pod \"d5162bf6-1ca4-4925-aa7d-0275e3881ce8\" (UID: \"d5162bf6-1ca4-4925-aa7d-0275e3881ce8\") " Oct 30 14:23:18.588143 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:18.588033 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d5162bf6-1ca4-4925-aa7d-0275e3881ce8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5162bf6-1ca4-4925-aa7d-0275e3881ce8-istio-podinfo\") pod \"d5162bf6-1ca4-4925-aa7d-0275e3881ce8\" (UID: \"d5162bf6-1ca4-4925-aa7d-0275e3881ce8\") " Oct 30 14:23:18.588143 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:18.588132 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5162bf6-1ca4-4925-aa7d-0275e3881ce8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5162bf6-1ca4-4925-aa7d-0275e3881ce8-istio-data\") pod \"d5162bf6-1ca4-4925-aa7d-0275e3881ce8\" (UID: \"d5162bf6-1ca4-4925-aa7d-0275e3881ce8\") " Oct 30 14:23:18.588250 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:18.588216 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5162bf6-1ca4-4925-aa7d-0275e3881ce8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5162bf6-1ca4-4925-aa7d-0275e3881ce8-workload-socket\") pod \"d5162bf6-1ca4-4925-aa7d-0275e3881ce8\" (UID: \"d5162bf6-1ca4-4925-aa7d-0275e3881ce8\") " Oct 30 14:23:18.588476 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:18.588460 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5162bf6-1ca4-4925-aa7d-0275e3881ce8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5162bf6-1ca4-4925-aa7d-0275e3881ce8-istio-envoy\") pod \"d5162bf6-1ca4-4925-aa7d-0275e3881ce8\" (UID: \"d5162bf6-1ca4-4925-aa7d-0275e3881ce8\") " Oct 30 14:23:18.588629 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:18.588424 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5162bf6-1ca4-4925-aa7d-0275e3881ce8-istiod-ca-cert" pod "d5162bf6-1ca4-4925-aa7d-0275e3881ce8" (UID: "d5162bf6-1ca4-4925-aa7d-0275e3881ce8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:23:18.588712 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:18.588596 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5162bf6-1ca4-4925-aa7d-0275e3881ce8-credential-socket" pod "d5162bf6-1ca4-4925-aa7d-0275e3881ce8" (UID: "d5162bf6-1ca4-4925-aa7d-0275e3881ce8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:23:18.588712 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:18.588608 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5162bf6-1ca4-4925-aa7d-0275e3881ce8-istio-token\" (UniqueName: \"kubernetes.io/projected/d5162bf6-1ca4-4925-aa7d-0275e3881ce8-istio-token\") pod \"d5162bf6-1ca4-4925-aa7d-0275e3881ce8\" (UID: \"d5162bf6-1ca4-4925-aa7d-0275e3881ce8\") " Oct 30 14:23:18.588712 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:18.588679 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5162bf6-1ca4-4925-aa7d-0275e3881ce8-workload-socket" pod "d5162bf6-1ca4-4925-aa7d-0275e3881ce8" (UID: "d5162bf6-1ca4-4925-aa7d-0275e3881ce8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:23:18.588861 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:18.588753 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5162bf6-1ca4-4925-aa7d-0275e3881ce8-istio-data" pod "d5162bf6-1ca4-4925-aa7d-0275e3881ce8" (UID: "d5162bf6-1ca4-4925-aa7d-0275e3881ce8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:23:18.589395 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:18.589370 4982 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5162bf6-1ca4-4925-aa7d-0275e3881ce8-istio-data\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:23:18.589488 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:18.589400 4982 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5162bf6-1ca4-4925-aa7d-0275e3881ce8-workload-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:23:18.589488 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:18.589416 4982 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5162bf6-1ca4-4925-aa7d-0275e3881ce8-istiod-ca-cert\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:23:18.589488 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:18.589431 4982 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5162bf6-1ca4-4925-aa7d-0275e3881ce8-credential-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:23:18.589740 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:18.589717 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5162bf6-1ca4-4925-aa7d-0275e3881ce8-workload-certs" pod "d5162bf6-1ca4-4925-aa7d-0275e3881ce8" (UID: "d5162bf6-1ca4-4925-aa7d-0275e3881ce8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:23:18.590615 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:18.590578 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5162bf6-1ca4-4925-aa7d-0275e3881ce8-kube-api-access-48cmt" pod "d5162bf6-1ca4-4925-aa7d-0275e3881ce8" (UID: "d5162bf6-1ca4-4925-aa7d-0275e3881ce8"). InnerVolumeSpecName "kube-api-access-48cmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:23:18.591702 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:18.591669 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5162bf6-1ca4-4925-aa7d-0275e3881ce8-istio-envoy" pod "d5162bf6-1ca4-4925-aa7d-0275e3881ce8" (UID: "d5162bf6-1ca4-4925-aa7d-0275e3881ce8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:23:18.591888 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:18.591862 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5162bf6-1ca4-4925-aa7d-0275e3881ce8-istio-podinfo" pod "d5162bf6-1ca4-4925-aa7d-0275e3881ce8" (UID: "d5162bf6-1ca4-4925-aa7d-0275e3881ce8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:23:18.592800 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:18.592771 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5162bf6-1ca4-4925-aa7d-0275e3881ce8-istio-token" pod "d5162bf6-1ca4-4925-aa7d-0275e3881ce8" (UID: "d5162bf6-1ca4-4925-aa7d-0275e3881ce8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:23:18.672521 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:18.672421 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:18.691296 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:18.691254 4982 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5162bf6-1ca4-4925-aa7d-0275e3881ce8-workload-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:23:18.691296 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:18.691284 4982 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5162bf6-1ca4-4925-aa7d-0275e3881ce8-istio-podinfo\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:23:18.691296 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:18.691292 4982 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5162bf6-1ca4-4925-aa7d-0275e3881ce8-istio-envoy\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:23:18.691296 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:18.691300 4982 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5162bf6-1ca4-4925-aa7d-0275e3881ce8-istio-token\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:23:18.691622 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:18.691309 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48cmt\" (UniqueName: \"kubernetes.io/projected/d5162bf6-1ca4-4925-aa7d-0275e3881ce8-kube-api-access-48cmt\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:23:18.772143 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:18.772114 4982 generic.go:356] "Generic (PLEG): container finished" podID="d5162bf6-1ca4-4925-aa7d-0275e3881ce8" containerID="95a2c47dd418c2eb1f7033531d53ca7a70bb428f60d42f89cca9e09040d90380" exitCode=0 Oct 30 14:23:18.772295 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:18.772155 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx" Oct 30 14:23:18.772295 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:18.772194 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx" event={"ID":"d5162bf6-1ca4-4925-aa7d-0275e3881ce8","Type":"ContainerDied","Data":"95a2c47dd418c2eb1f7033531d53ca7a70bb428f60d42f89cca9e09040d90380"} Oct 30 14:23:18.772295 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:18.772252 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx" event={"ID":"d5162bf6-1ca4-4925-aa7d-0275e3881ce8","Type":"ContainerDied","Data":"11cdf8ce2ec488e806f0680f9b134a1e9996400801468b2dc6914b5c6c0c589f"} Oct 30 14:23:18.772295 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:18.772278 4982 scope.go:122] "RemoveContainer" containerID="95a2c47dd418c2eb1f7033531d53ca7a70bb428f60d42f89cca9e09040d90380" Oct 30 14:23:18.783736 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:18.783718 4982 scope.go:122] "RemoveContainer" containerID="95a2c47dd418c2eb1f7033531d53ca7a70bb428f60d42f89cca9e09040d90380" Oct 30 14:23:18.783986 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:23:18.783969 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95a2c47dd418c2eb1f7033531d53ca7a70bb428f60d42f89cca9e09040d90380\": container with ID starting with 95a2c47dd418c2eb1f7033531d53ca7a70bb428f60d42f89cca9e09040d90380 not found: ID does not exist" containerID="95a2c47dd418c2eb1f7033531d53ca7a70bb428f60d42f89cca9e09040d90380" Oct 30 14:23:18.784051 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:18.783995 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95a2c47dd418c2eb1f7033531d53ca7a70bb428f60d42f89cca9e09040d90380"} err="failed to get container status \"95a2c47dd418c2eb1f7033531d53ca7a70bb428f60d42f89cca9e09040d90380\": rpc error: code = NotFound desc = could not find container \"95a2c47dd418c2eb1f7033531d53ca7a70bb428f60d42f89cca9e09040d90380\": container with ID starting with 95a2c47dd418c2eb1f7033531d53ca7a70bb428f60d42f89cca9e09040d90380 not found: ID does not exist" Oct 30 14:23:18.832806 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:18.832783 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx"] Oct 30 14:23:18.846011 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:18.845992 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx"] Oct 30 14:23:19.283336 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:19.283293 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4bf33a-6bb59459b-4qdp6"] Oct 30 14:23:19.315524 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:19.315487 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4bf33a-7f5fb57b55-9mjfq"] Oct 30 14:23:19.355215 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:19.355180 4982 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:23:19.355380 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:19.355241 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-qcqdx" podUID="d5162bf6-1ca4-4925-aa7d-0275e3881ce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:23:19.557541 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:19.557501 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:19.742607 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:19.742575 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:20.401621 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:20.401586 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5162bf6-1ca4-4925-aa7d-0275e3881ce8" path="/var/lib/kubelet/pods/d5162bf6-1ca4-4925-aa7d-0275e3881ce8/volumes" Oct 30 14:23:20.777388 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:20.775581 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:21.024052 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:21.024017 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:21.216370 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:21.216336 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:21.368542 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:21.368510 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:21.522863 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:21.522791 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:22.024161 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:22.024117 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:22.235161 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:22.235120 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:22.827072 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:22.827038 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:24.284701 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:24.284618 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-4bf33a-6bb59459b-4qdp6" podUID="d34de819-45d5-4dd0-a4a2-b9f742983fad" containerName="payload-processing" containerID="cri-o://a0625d95c03484145dd21b7791009b469d9ad144007860859c7ddbb5a6a545e6" gracePeriod=25 Oct 30 14:23:24.316468 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:24.316435 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4bf33a-7f5fb57b55-9mjfq" podUID="65c30204-f1f4-4017-9ecf-71bbe81dbff6" containerName="payload-pre-processing" containerID="cri-o://72038def3f5a26a62400c2ba2bc6c708975393f7f4097cb75b46902e0cd7daa8" gracePeriod=25 Oct 30 14:23:24.621155 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:24.621124 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4bf33a-6bb59459b-4qdp6" Oct 30 14:23:24.643523 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:24.643496 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4bf33a-7f5fb57b55-9mjfq" Oct 30 14:23:24.738968 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:24.738939 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65c30204-f1f4-4017-9ecf-71bbe81dbff6-kube-api-access-5wwfh\" (UniqueName: \"kubernetes.io/projected/65c30204-f1f4-4017-9ecf-71bbe81dbff6-kube-api-access-5wwfh\") pod \"65c30204-f1f4-4017-9ecf-71bbe81dbff6\" (UID: \"65c30204-f1f4-4017-9ecf-71bbe81dbff6\") " Oct 30 14:23:24.739147 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:24.738992 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65c30204-f1f4-4017-9ecf-71bbe81dbff6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65c30204-f1f4-4017-9ecf-71bbe81dbff6-plugins-config-volume\") pod \"65c30204-f1f4-4017-9ecf-71bbe81dbff6\" (UID: \"65c30204-f1f4-4017-9ecf-71bbe81dbff6\") " Oct 30 14:23:24.739147 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:24.739038 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d34de819-45d5-4dd0-a4a2-b9f742983fad-kube-api-access-fsn7t\" (UniqueName: \"kubernetes.io/projected/d34de819-45d5-4dd0-a4a2-b9f742983fad-kube-api-access-fsn7t\") pod \"d34de819-45d5-4dd0-a4a2-b9f742983fad\" (UID: \"d34de819-45d5-4dd0-a4a2-b9f742983fad\") " Oct 30 14:23:24.739147 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:24.739099 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d34de819-45d5-4dd0-a4a2-b9f742983fad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d34de819-45d5-4dd0-a4a2-b9f742983fad-plugins-config-volume\") pod \"d34de819-45d5-4dd0-a4a2-b9f742983fad\" (UID: \"d34de819-45d5-4dd0-a4a2-b9f742983fad\") " Oct 30 14:23:24.739510 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:24.739487 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65c30204-f1f4-4017-9ecf-71bbe81dbff6-plugins-config-volume" pod "65c30204-f1f4-4017-9ecf-71bbe81dbff6" (UID: "65c30204-f1f4-4017-9ecf-71bbe81dbff6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:23:24.739623 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:24.739570 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d34de819-45d5-4dd0-a4a2-b9f742983fad-plugins-config-volume" pod "d34de819-45d5-4dd0-a4a2-b9f742983fad" (UID: "d34de819-45d5-4dd0-a4a2-b9f742983fad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:23:24.739812 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:24.739790 4982 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d34de819-45d5-4dd0-a4a2-b9f742983fad-plugins-config-volume\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:23:24.739894 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:24.739816 4982 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65c30204-f1f4-4017-9ecf-71bbe81dbff6-plugins-config-volume\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:23:24.741853 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:24.741824 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65c30204-f1f4-4017-9ecf-71bbe81dbff6-kube-api-access-5wwfh" pod "65c30204-f1f4-4017-9ecf-71bbe81dbff6" (UID: "65c30204-f1f4-4017-9ecf-71bbe81dbff6"). InnerVolumeSpecName "kube-api-access-5wwfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:23:24.742143 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:24.742122 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d34de819-45d5-4dd0-a4a2-b9f742983fad-kube-api-access-fsn7t" pod "d34de819-45d5-4dd0-a4a2-b9f742983fad" (UID: "d34de819-45d5-4dd0-a4a2-b9f742983fad"). InnerVolumeSpecName "kube-api-access-fsn7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:23:24.838071 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:24.838041 4982 generic.go:356] "Generic (PLEG): container finished" podID="65c30204-f1f4-4017-9ecf-71bbe81dbff6" containerID="72038def3f5a26a62400c2ba2bc6c708975393f7f4097cb75b46902e0cd7daa8" exitCode=0 Oct 30 14:23:24.838252 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:24.838092 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4bf33a-7f5fb57b55-9mjfq" Oct 30 14:23:24.838252 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:24.838122 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4bf33a-7f5fb57b55-9mjfq" event={"ID":"65c30204-f1f4-4017-9ecf-71bbe81dbff6","Type":"ContainerDied","Data":"72038def3f5a26a62400c2ba2bc6c708975393f7f4097cb75b46902e0cd7daa8"} Oct 30 14:23:24.838252 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:24.838179 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4bf33a-7f5fb57b55-9mjfq" event={"ID":"65c30204-f1f4-4017-9ecf-71bbe81dbff6","Type":"ContainerDied","Data":"c32bc904e8b0d320e1b0c51ae219106b9d5ba09d686c7c44747afbcea955fca2"} Oct 30 14:23:24.838252 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:24.838211 4982 scope.go:122] "RemoveContainer" containerID="72038def3f5a26a62400c2ba2bc6c708975393f7f4097cb75b46902e0cd7daa8" Oct 30 14:23:24.839673 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:24.839622 4982 generic.go:356] "Generic (PLEG): container finished" podID="d34de819-45d5-4dd0-a4a2-b9f742983fad" containerID="a0625d95c03484145dd21b7791009b469d9ad144007860859c7ddbb5a6a545e6" exitCode=0 Oct 30 14:23:24.839801 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:24.839682 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4bf33a-6bb59459b-4qdp6" event={"ID":"d34de819-45d5-4dd0-a4a2-b9f742983fad","Type":"ContainerDied","Data":"a0625d95c03484145dd21b7791009b469d9ad144007860859c7ddbb5a6a545e6"} Oct 30 14:23:24.839801 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:24.839702 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4bf33a-6bb59459b-4qdp6" Oct 30 14:23:24.839801 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:24.839723 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4bf33a-6bb59459b-4qdp6" event={"ID":"d34de819-45d5-4dd0-a4a2-b9f742983fad","Type":"ContainerDied","Data":"d81943e6365ebba4223791c23dca147e01173b9c8eb3fcf00b9e990fdf0099c1"} Oct 30 14:23:24.841339 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:24.841314 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wwfh\" (UniqueName: \"kubernetes.io/projected/65c30204-f1f4-4017-9ecf-71bbe81dbff6-kube-api-access-5wwfh\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:23:24.841419 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:24.841346 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsn7t\" (UniqueName: \"kubernetes.io/projected/d34de819-45d5-4dd0-a4a2-b9f742983fad-kube-api-access-fsn7t\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:23:24.850079 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:24.850060 4982 scope.go:122] "RemoveContainer" containerID="72038def3f5a26a62400c2ba2bc6c708975393f7f4097cb75b46902e0cd7daa8" Oct 30 14:23:24.850470 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:23:24.850396 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72038def3f5a26a62400c2ba2bc6c708975393f7f4097cb75b46902e0cd7daa8\": container with ID starting with 72038def3f5a26a62400c2ba2bc6c708975393f7f4097cb75b46902e0cd7daa8 not found: ID does not exist" containerID="72038def3f5a26a62400c2ba2bc6c708975393f7f4097cb75b46902e0cd7daa8" Oct 30 14:23:24.850470 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:24.850435 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72038def3f5a26a62400c2ba2bc6c708975393f7f4097cb75b46902e0cd7daa8"} err="failed to get container status \"72038def3f5a26a62400c2ba2bc6c708975393f7f4097cb75b46902e0cd7daa8\": rpc error: code = NotFound desc = could not find container \"72038def3f5a26a62400c2ba2bc6c708975393f7f4097cb75b46902e0cd7daa8\": container with ID starting with 72038def3f5a26a62400c2ba2bc6c708975393f7f4097cb75b46902e0cd7daa8 not found: ID does not exist" Oct 30 14:23:24.850470 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:24.850457 4982 scope.go:122] "RemoveContainer" containerID="a0625d95c03484145dd21b7791009b469d9ad144007860859c7ddbb5a6a545e6" Oct 30 14:23:24.861711 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:24.861693 4982 scope.go:122] "RemoveContainer" containerID="a0625d95c03484145dd21b7791009b469d9ad144007860859c7ddbb5a6a545e6" Oct 30 14:23:24.862001 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:23:24.861977 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0625d95c03484145dd21b7791009b469d9ad144007860859c7ddbb5a6a545e6\": container with ID starting with a0625d95c03484145dd21b7791009b469d9ad144007860859c7ddbb5a6a545e6 not found: ID does not exist" containerID="a0625d95c03484145dd21b7791009b469d9ad144007860859c7ddbb5a6a545e6" Oct 30 14:23:24.862102 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:24.862006 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0625d95c03484145dd21b7791009b469d9ad144007860859c7ddbb5a6a545e6"} err="failed to get container status \"a0625d95c03484145dd21b7791009b469d9ad144007860859c7ddbb5a6a545e6\": rpc error: code = NotFound desc = could not find container \"a0625d95c03484145dd21b7791009b469d9ad144007860859c7ddbb5a6a545e6\": container with ID starting with a0625d95c03484145dd21b7791009b469d9ad144007860859c7ddbb5a6a545e6 not found: ID does not exist" Oct 30 14:23:24.907770 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:24.907746 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4bf33a-7f5fb57b55-9mjfq"] Oct 30 14:23:24.916133 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:24.916109 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4bf33a-7f5fb57b55-9mjfq"] Oct 30 14:23:24.966609 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:24.966575 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4bf33a-6bb59459b-4qdp6"] Oct 30 14:23:24.969707 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:24.969685 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4bf33a-6bb59459b-4qdp6"] Oct 30 14:23:25.334600 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:25.334565 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:26.401459 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:26.401425 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65c30204-f1f4-4017-9ecf-71bbe81dbff6" path="/var/lib/kubelet/pods/65c30204-f1f4-4017-9ecf-71bbe81dbff6/volumes" Oct 30 14:23:26.401866 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:26.401852 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d34de819-45d5-4dd0-a4a2-b9f742983fad" path="/var/lib/kubelet/pods/d34de819-45d5-4dd0-a4a2-b9f742983fad/volumes" Oct 30 14:23:26.533356 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:26.533322 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:26.731152 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:26.731047 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:27.725256 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:27.725222 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:28.014345 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:23:28.014259 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:23:28.353423 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:28.353389 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:28.491214 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:28.491183 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a9542c-794b65877f-sp225"] Oct 30 14:23:28.491794 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:28.491778 4982 cpu_manager.go:414] "containerMap: removing container" podUID="65c30204-f1f4-4017-9ecf-71bbe81dbff6" containerName="payload-pre-processing" Oct 30 14:23:28.491887 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:28.491794 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65c30204-f1f4-4017-9ecf-71bbe81dbff6" containerName="payload-pre-processing" Oct 30 14:23:28.491887 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:28.491806 4982 cpu_manager.go:414] "containerMap: removing container" podUID="d5162bf6-1ca4-4925-aa7d-0275e3881ce8" containerName="istio-proxy" Oct 30 14:23:28.491887 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:28.491811 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5162bf6-1ca4-4925-aa7d-0275e3881ce8" containerName="istio-proxy" Oct 30 14:23:28.491887 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:28.491823 4982 cpu_manager.go:414] "containerMap: removing container" podUID="69ad8491-34f8-494b-89ba-a9ddd63cac3d" containerName="revoke-keys" Oct 30 14:23:28.491887 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:28.491829 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69ad8491-34f8-494b-89ba-a9ddd63cac3d" containerName="revoke-keys" Oct 30 14:23:28.491887 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:28.491852 4982 cpu_manager.go:414] "containerMap: removing container" podUID="d34de819-45d5-4dd0-a4a2-b9f742983fad" containerName="payload-processing" Oct 30 14:23:28.491887 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:28.491857 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d34de819-45d5-4dd0-a4a2-b9f742983fad" containerName="payload-processing" Oct 30 14:23:28.492290 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:28.491946 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="d34de819-45d5-4dd0-a4a2-b9f742983fad" containerName="payload-processing" Oct 30 14:23:28.492290 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:28.491959 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="65c30204-f1f4-4017-9ecf-71bbe81dbff6" containerName="payload-pre-processing" Oct 30 14:23:28.492290 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:28.491967 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5162bf6-1ca4-4925-aa7d-0275e3881ce8" containerName="istio-proxy" Oct 30 14:23:28.492290 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:28.491973 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="69ad8491-34f8-494b-89ba-a9ddd63cac3d" containerName="revoke-keys" Oct 30 14:23:28.497146 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:28.497125 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a9542c-794b65877f-sp225" Oct 30 14:23:28.519849 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:28.519822 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a9542c-794b65877f-sp225"] Oct 30 14:23:28.527692 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:28.527661 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-a9542c\"" Oct 30 14:23:28.527816 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:28.527705 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-a9542c-dockercfg-qh7q6\"" Oct 30 14:23:28.550869 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:28.550846 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:28.658050 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:28.657958 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt2mc\" (UniqueName: \"kubernetes.io/projected/cdf6bd0e-f6ff-49b9-8f1a-b90052fe7787-kube-api-access-wt2mc\") pod \"payload-pre-processing-e2e-inf-b-a9542c-794b65877f-sp225\" (UID: \"cdf6bd0e-f6ff-49b9-8f1a-b90052fe7787\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a9542c-794b65877f-sp225" Oct 30 14:23:28.658380 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:28.658356 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdf6bd0e-f6ff-49b9-8f1a-b90052fe7787-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a9542c-794b65877f-sp225\" (UID: \"cdf6bd0e-f6ff-49b9-8f1a-b90052fe7787\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a9542c-794b65877f-sp225" Oct 30 14:23:28.658894 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:28.658869 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a9542c-69bcb5578b-rzp2q"] Oct 30 14:23:28.779296 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:28.761098 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a9542c-69bcb5578b-rzp2q"] Oct 30 14:23:28.779296 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:28.761267 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a9542c-69bcb5578b-rzp2q" Oct 30 14:23:28.801881 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:28.796380 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdf6bd0e-f6ff-49b9-8f1a-b90052fe7787-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a9542c-794b65877f-sp225\" (UID: \"cdf6bd0e-f6ff-49b9-8f1a-b90052fe7787\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a9542c-794b65877f-sp225" Oct 30 14:23:28.801881 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:28.797464 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdf6bd0e-f6ff-49b9-8f1a-b90052fe7787-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a9542c-794b65877f-sp225\" (UID: \"cdf6bd0e-f6ff-49b9-8f1a-b90052fe7787\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a9542c-794b65877f-sp225" Oct 30 14:23:28.807658 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:28.803379 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt2mc\" (UniqueName: \"kubernetes.io/projected/cdf6bd0e-f6ff-49b9-8f1a-b90052fe7787-kube-api-access-wt2mc\") pod \"payload-pre-processing-e2e-inf-b-a9542c-794b65877f-sp225\" (UID: \"cdf6bd0e-f6ff-49b9-8f1a-b90052fe7787\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a9542c-794b65877f-sp225" Oct 30 14:23:28.832511 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:28.832368 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt2mc\" (UniqueName: \"kubernetes.io/projected/cdf6bd0e-f6ff-49b9-8f1a-b90052fe7787-kube-api-access-wt2mc\") pod \"payload-pre-processing-e2e-inf-b-a9542c-794b65877f-sp225\" (UID: \"cdf6bd0e-f6ff-49b9-8f1a-b90052fe7787\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a9542c-794b65877f-sp225" Oct 30 14:23:28.905570 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:28.905534 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhkjc\" (UniqueName: \"kubernetes.io/projected/73f0efb5-d969-4b91-8558-75d5ebe12a43-kube-api-access-mhkjc\") pod \"payload-processing-e2e-inf-b-a9542c-69bcb5578b-rzp2q\" (UID: \"73f0efb5-d969-4b91-8558-75d5ebe12a43\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a9542c-69bcb5578b-rzp2q" Oct 30 14:23:28.905773 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:28.905738 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73f0efb5-d969-4b91-8558-75d5ebe12a43-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a9542c-69bcb5578b-rzp2q\" (UID: \"73f0efb5-d969-4b91-8558-75d5ebe12a43\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a9542c-69bcb5578b-rzp2q" Oct 30 14:23:29.007749 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:29.007657 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73f0efb5-d969-4b91-8558-75d5ebe12a43-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a9542c-69bcb5578b-rzp2q\" (UID: \"73f0efb5-d969-4b91-8558-75d5ebe12a43\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a9542c-69bcb5578b-rzp2q" Oct 30 14:23:29.007923 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:29.007906 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhkjc\" (UniqueName: \"kubernetes.io/projected/73f0efb5-d969-4b91-8558-75d5ebe12a43-kube-api-access-mhkjc\") pod \"payload-processing-e2e-inf-b-a9542c-69bcb5578b-rzp2q\" (UID: \"73f0efb5-d969-4b91-8558-75d5ebe12a43\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a9542c-69bcb5578b-rzp2q" Oct 30 14:23:29.008427 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:29.008405 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73f0efb5-d969-4b91-8558-75d5ebe12a43-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a9542c-69bcb5578b-rzp2q\" (UID: \"73f0efb5-d969-4b91-8558-75d5ebe12a43\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a9542c-69bcb5578b-rzp2q" Oct 30 14:23:29.036544 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:29.036514 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhkjc\" (UniqueName: \"kubernetes.io/projected/73f0efb5-d969-4b91-8558-75d5ebe12a43-kube-api-access-mhkjc\") pod \"payload-processing-e2e-inf-b-a9542c-69bcb5578b-rzp2q\" (UID: \"73f0efb5-d969-4b91-8558-75d5ebe12a43\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a9542c-69bcb5578b-rzp2q" Oct 30 14:23:29.108339 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:29.108311 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a9542c-69bcb5578b-rzp2q" Oct 30 14:23:29.111358 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:29.111326 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a9542c-794b65877f-sp225" Oct 30 14:23:29.433603 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:29.433575 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a9542c-69bcb5578b-rzp2q"] Oct 30 14:23:29.434772 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:23:29.434741 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73f0efb5_d969_4b91_8558_75d5ebe12a43.slice/crio-729d426411d174bde80748c9fa96f95b6fa79fc5e6c5c04e4eeb39d253fb352b WatchSource:0}: Error finding container 729d426411d174bde80748c9fa96f95b6fa79fc5e6c5c04e4eeb39d253fb352b: Status 404 returned error can't find the container with id 729d426411d174bde80748c9fa96f95b6fa79fc5e6c5c04e4eeb39d253fb352b Oct 30 14:23:29.441512 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:23:29.441484 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdf6bd0e_f6ff_49b9_8f1a_b90052fe7787.slice/crio-c88c89395d1070360857e8bb252eabb7cfb333cf5074ed6142548dbbb0eed53a WatchSource:0}: Error finding container c88c89395d1070360857e8bb252eabb7cfb333cf5074ed6142548dbbb0eed53a: Status 404 returned error can't find the container with id c88c89395d1070360857e8bb252eabb7cfb333cf5074ed6142548dbbb0eed53a Oct 30 14:23:29.443107 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:29.443079 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a9542c-794b65877f-sp225"] Oct 30 14:23:29.881432 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:29.881390 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a9542c-794b65877f-sp225" event={"ID":"cdf6bd0e-f6ff-49b9-8f1a-b90052fe7787","Type":"ContainerStarted","Data":"480cf0ec02d620b0a9ad9188b060c296e9bd12ee799147bb2fc1f907ba55c234"} Oct 30 14:23:29.881940 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:29.881459 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a9542c-794b65877f-sp225" event={"ID":"cdf6bd0e-f6ff-49b9-8f1a-b90052fe7787","Type":"ContainerStarted","Data":"c88c89395d1070360857e8bb252eabb7cfb333cf5074ed6142548dbbb0eed53a"} Oct 30 14:23:29.881940 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:29.881716 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a9542c-794b65877f-sp225" Oct 30 14:23:29.884187 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:29.884155 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a9542c-69bcb5578b-rzp2q" event={"ID":"73f0efb5-d969-4b91-8558-75d5ebe12a43","Type":"ContainerStarted","Data":"2e9387560467b28eae38822c7ae72f2cfa3e72cde643a9697a62fd0751d45910"} Oct 30 14:23:29.884503 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:29.884206 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a9542c-69bcb5578b-rzp2q" event={"ID":"73f0efb5-d969-4b91-8558-75d5ebe12a43","Type":"ContainerStarted","Data":"729d426411d174bde80748c9fa96f95b6fa79fc5e6c5c04e4eeb39d253fb352b"} Oct 30 14:23:29.884503 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:29.884324 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-a9542c-69bcb5578b-rzp2q" Oct 30 14:23:29.928517 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:29.928260 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a9542c-794b65877f-sp225" podStartSLOduration=1.928242478 podStartE2EDuration="1.928242478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:23:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:23:29.926786836 +0000 UTC m=+2079.670920878" watchObservedRunningTime="2026-10-30 14:23:29.928242478 +0000 UTC m=+2079.672376518" Oct 30 14:23:29.973468 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:29.973414 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-a9542c-69bcb5578b-rzp2q" podStartSLOduration=1.9734009810000002 podStartE2EDuration="1.973400981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:23:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:23:29.970968981 +0000 UTC m=+2079.715103020" watchObservedRunningTime="2026-10-30 14:23:29.973400981 +0000 UTC m=+2079.717535047" Oct 30 14:23:30.317976 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:30.317928 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:32.631292 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:32.631248 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:34.343148 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:34.343093 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:34.704627 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:34.704529 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:34.910662 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:34.897321 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:34.991933 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:34.991842 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c4b82-8xk6g"] Oct 30 14:23:35.012827 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:35.012785 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c4b82-8xk6g"] Oct 30 14:23:35.073105 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:35.073027 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:35.427059 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:35.427022 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:35.514999 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:35.514965 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a2835-l59d2"] Oct 30 14:23:35.530512 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:35.530479 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2a2835-l59d2"] Oct 30 14:23:35.599120 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:35.599088 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:35.815985 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:35.815936 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:36.093173 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:36.093084 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:36.399985 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:36.399915 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4be44544-6e2f-4959-b0db-56d093d5f6b0" path="/var/lib/kubelet/pods/4be44544-6e2f-4959-b0db-56d093d5f6b0/volumes" Oct 30 14:23:36.400398 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:36.400382 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fa389f0-fd2d-4cd9-a57b-45f00a79dbb8" path="/var/lib/kubelet/pods/9fa389f0-fd2d-4cd9-a57b-45f00a79dbb8/volumes" Oct 30 14:23:38.209318 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:38.209281 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:40.047041 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:40.046998 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:40.896699 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:40.896654 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-a9542c-69bcb5578b-rzp2q" Oct 30 14:23:40.896890 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:40.896823 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a9542c-794b65877f-sp225" Oct 30 14:23:41.875170 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:41.875132 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:42.938860 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:42.938822 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:43.080706 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:43.080635 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:43.278029 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:43.277944 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:44.070377 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:44.070326 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:44.253234 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:44.253198 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:44.450203 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:44.450116 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:44.472899 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:44.472864 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77"] Oct 30 14:23:44.473351 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:44.473318 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77" podUID="115b91e0-f78d-4edf-9f99-8a497776c921" containerName="istio-proxy" containerID="cri-o://d3732b1e3b5ccc4c94f6fb8efbfa4e7e7593c6f5b38999eb6d9195836398d286" gracePeriod=30 Oct 30 14:23:44.484951 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:44.484925 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8"] Oct 30 14:23:44.485104 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:44.485085 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8" podUID="f717a7d7-7930-4224-9872-0fc167f1f391" containerName="istio-proxy" containerID="cri-o://36ede143e387390f1be9ed41a11ce57b62246f4b77c98b16b2581042a83edc5d" gracePeriod=30 Oct 30 14:23:44.676343 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:44.676297 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:44.893664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:44.887567 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:45.340953 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:45.340908 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:46.022605 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:46.022552 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:46.207121 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:46.207083 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:48.103257 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:48.102322 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:48.546267 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:48.546188 4982 patch_prober.go:29] interesting pod/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 14:23:48.546267 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:48.546246 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77" podUID="115b91e0-f78d-4edf-9f99-8a497776c921" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 14:23:48.560880 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:48.560844 4982 patch_prober.go:29] interesting pod/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 14:23:48.561022 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:48.560935 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8" podUID="f717a7d7-7930-4224-9872-0fc167f1f391" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 14:23:49.080249 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:49.080209 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:49.257110 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:49.257033 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:49.795655 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:49.795614 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77" Oct 30 14:23:49.819574 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:49.819543 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8" Oct 30 14:23:49.923200 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:49.923161 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f717a7d7-7930-4224-9872-0fc167f1f391-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f717a7d7-7930-4224-9872-0fc167f1f391-workload-socket\") pod \"f717a7d7-7930-4224-9872-0fc167f1f391\" (UID: \"f717a7d7-7930-4224-9872-0fc167f1f391\") " Oct 30 14:23:49.923384 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:49.923220 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f717a7d7-7930-4224-9872-0fc167f1f391-kube-api-access-cmfh7\" (UniqueName: \"kubernetes.io/projected/f717a7d7-7930-4224-9872-0fc167f1f391-kube-api-access-cmfh7\") pod \"f717a7d7-7930-4224-9872-0fc167f1f391\" (UID: \"f717a7d7-7930-4224-9872-0fc167f1f391\") " Oct 30 14:23:49.923384 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:49.923256 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f717a7d7-7930-4224-9872-0fc167f1f391-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f717a7d7-7930-4224-9872-0fc167f1f391-istio-envoy\") pod \"f717a7d7-7930-4224-9872-0fc167f1f391\" (UID: \"f717a7d7-7930-4224-9872-0fc167f1f391\") " Oct 30 14:23:49.923384 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:49.923312 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/115b91e0-f78d-4edf-9f99-8a497776c921-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/115b91e0-f78d-4edf-9f99-8a497776c921-workload-certs\") pod \"115b91e0-f78d-4edf-9f99-8a497776c921\" (UID: \"115b91e0-f78d-4edf-9f99-8a497776c921\") " Oct 30 14:23:49.923384 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:49.923349 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/115b91e0-f78d-4edf-9f99-8a497776c921-istio-token\" (UniqueName: \"kubernetes.io/projected/115b91e0-f78d-4edf-9f99-8a497776c921-istio-token\") pod \"115b91e0-f78d-4edf-9f99-8a497776c921\" (UID: \"115b91e0-f78d-4edf-9f99-8a497776c921\") " Oct 30 14:23:49.923384 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:49.923375 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f717a7d7-7930-4224-9872-0fc167f1f391-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f717a7d7-7930-4224-9872-0fc167f1f391-credential-socket\") pod \"f717a7d7-7930-4224-9872-0fc167f1f391\" (UID: \"f717a7d7-7930-4224-9872-0fc167f1f391\") " Oct 30 14:23:49.923696 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:49.923424 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/115b91e0-f78d-4edf-9f99-8a497776c921-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/115b91e0-f78d-4edf-9f99-8a497776c921-workload-socket\") pod \"115b91e0-f78d-4edf-9f99-8a497776c921\" (UID: \"115b91e0-f78d-4edf-9f99-8a497776c921\") " Oct 30 14:23:49.923696 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:49.923454 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f717a7d7-7930-4224-9872-0fc167f1f391-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f717a7d7-7930-4224-9872-0fc167f1f391-istiod-ca-cert\") pod \"f717a7d7-7930-4224-9872-0fc167f1f391\" (UID: \"f717a7d7-7930-4224-9872-0fc167f1f391\") " Oct 30 14:23:49.923696 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:49.923509 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f717a7d7-7930-4224-9872-0fc167f1f391-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f717a7d7-7930-4224-9872-0fc167f1f391-istio-podinfo\") pod \"f717a7d7-7930-4224-9872-0fc167f1f391\" (UID: \"f717a7d7-7930-4224-9872-0fc167f1f391\") " Oct 30 14:23:49.923696 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:49.923577 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/115b91e0-f78d-4edf-9f99-8a497776c921-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/115b91e0-f78d-4edf-9f99-8a497776c921-istiod-ca-cert\") pod \"115b91e0-f78d-4edf-9f99-8a497776c921\" (UID: \"115b91e0-f78d-4edf-9f99-8a497776c921\") " Oct 30 14:23:49.923696 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:49.923619 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f717a7d7-7930-4224-9872-0fc167f1f391-istio-token\" (UniqueName: \"kubernetes.io/projected/f717a7d7-7930-4224-9872-0fc167f1f391-istio-token\") pod \"f717a7d7-7930-4224-9872-0fc167f1f391\" (UID: \"f717a7d7-7930-4224-9872-0fc167f1f391\") " Oct 30 14:23:49.923696 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:49.923662 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f717a7d7-7930-4224-9872-0fc167f1f391-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f717a7d7-7930-4224-9872-0fc167f1f391-istio-data\") pod \"f717a7d7-7930-4224-9872-0fc167f1f391\" (UID: \"f717a7d7-7930-4224-9872-0fc167f1f391\") " Oct 30 14:23:49.924002 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:49.923701 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/115b91e0-f78d-4edf-9f99-8a497776c921-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/115b91e0-f78d-4edf-9f99-8a497776c921-istio-data\") pod \"115b91e0-f78d-4edf-9f99-8a497776c921\" (UID: \"115b91e0-f78d-4edf-9f99-8a497776c921\") " Oct 30 14:23:49.924002 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:49.923746 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/115b91e0-f78d-4edf-9f99-8a497776c921-kube-api-access-kxhmp\" (UniqueName: \"kubernetes.io/projected/115b91e0-f78d-4edf-9f99-8a497776c921-kube-api-access-kxhmp\") pod \"115b91e0-f78d-4edf-9f99-8a497776c921\" (UID: \"115b91e0-f78d-4edf-9f99-8a497776c921\") " Oct 30 14:23:49.924002 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:49.923777 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/115b91e0-f78d-4edf-9f99-8a497776c921-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/115b91e0-f78d-4edf-9f99-8a497776c921-istio-envoy\") pod \"115b91e0-f78d-4edf-9f99-8a497776c921\" (UID: \"115b91e0-f78d-4edf-9f99-8a497776c921\") " Oct 30 14:23:49.924002 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:49.923806 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f717a7d7-7930-4224-9872-0fc167f1f391-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f717a7d7-7930-4224-9872-0fc167f1f391-workload-certs\") pod \"f717a7d7-7930-4224-9872-0fc167f1f391\" (UID: \"f717a7d7-7930-4224-9872-0fc167f1f391\") " Oct 30 14:23:49.924002 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:49.923828 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/115b91e0-f78d-4edf-9f99-8a497776c921-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/115b91e0-f78d-4edf-9f99-8a497776c921-credential-socket\") pod \"115b91e0-f78d-4edf-9f99-8a497776c921\" (UID: \"115b91e0-f78d-4edf-9f99-8a497776c921\") " Oct 30 14:23:49.924002 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:49.923974 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/115b91e0-f78d-4edf-9f99-8a497776c921-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/115b91e0-f78d-4edf-9f99-8a497776c921-istio-podinfo\") pod \"115b91e0-f78d-4edf-9f99-8a497776c921\" (UID: \"115b91e0-f78d-4edf-9f99-8a497776c921\") " Oct 30 14:23:49.925941 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:49.925910 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/115b91e0-f78d-4edf-9f99-8a497776c921-istio-data" pod "115b91e0-f78d-4edf-9f99-8a497776c921" (UID: "115b91e0-f78d-4edf-9f99-8a497776c921"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:23:49.926596 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:49.926262 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/115b91e0-f78d-4edf-9f99-8a497776c921-istiod-ca-cert" pod "115b91e0-f78d-4edf-9f99-8a497776c921" (UID: "115b91e0-f78d-4edf-9f99-8a497776c921"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:23:49.926596 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:49.926309 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f717a7d7-7930-4224-9872-0fc167f1f391-workload-socket" pod "f717a7d7-7930-4224-9872-0fc167f1f391" (UID: "f717a7d7-7930-4224-9872-0fc167f1f391"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:23:49.926596 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:49.926240 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f717a7d7-7930-4224-9872-0fc167f1f391-credential-socket" pod "f717a7d7-7930-4224-9872-0fc167f1f391" (UID: "f717a7d7-7930-4224-9872-0fc167f1f391"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:23:49.926596 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:49.926525 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f717a7d7-7930-4224-9872-0fc167f1f391-istio-data" pod "f717a7d7-7930-4224-9872-0fc167f1f391" (UID: "f717a7d7-7930-4224-9872-0fc167f1f391"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:23:49.926941 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:49.926744 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/115b91e0-f78d-4edf-9f99-8a497776c921-workload-socket" pod "115b91e0-f78d-4edf-9f99-8a497776c921" (UID: "115b91e0-f78d-4edf-9f99-8a497776c921"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:23:49.926941 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:49.926915 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f717a7d7-7930-4224-9872-0fc167f1f391-workload-certs" pod "f717a7d7-7930-4224-9872-0fc167f1f391" (UID: "f717a7d7-7930-4224-9872-0fc167f1f391"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:23:49.927054 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:49.926956 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/115b91e0-f78d-4edf-9f99-8a497776c921-workload-certs" pod "115b91e0-f78d-4edf-9f99-8a497776c921" (UID: "115b91e0-f78d-4edf-9f99-8a497776c921"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:23:49.927163 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:49.927141 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/115b91e0-f78d-4edf-9f99-8a497776c921-credential-socket" pod "115b91e0-f78d-4edf-9f99-8a497776c921" (UID: "115b91e0-f78d-4edf-9f99-8a497776c921"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:23:49.927457 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:49.927431 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f717a7d7-7930-4224-9872-0fc167f1f391-istiod-ca-cert" pod "f717a7d7-7930-4224-9872-0fc167f1f391" (UID: "f717a7d7-7930-4224-9872-0fc167f1f391"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:23:49.930851 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:49.930825 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f717a7d7-7930-4224-9872-0fc167f1f391-istio-envoy" pod "f717a7d7-7930-4224-9872-0fc167f1f391" (UID: "f717a7d7-7930-4224-9872-0fc167f1f391"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:23:49.930962 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:49.930930 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/115b91e0-f78d-4edf-9f99-8a497776c921-istio-token" pod "115b91e0-f78d-4edf-9f99-8a497776c921" (UID: "115b91e0-f78d-4edf-9f99-8a497776c921"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:23:49.931747 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:49.931719 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/115b91e0-f78d-4edf-9f99-8a497776c921-istio-podinfo" pod "115b91e0-f78d-4edf-9f99-8a497776c921" (UID: "115b91e0-f78d-4edf-9f99-8a497776c921"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:23:49.931747 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:49.931728 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f717a7d7-7930-4224-9872-0fc167f1f391-istio-token" pod "f717a7d7-7930-4224-9872-0fc167f1f391" (UID: "f717a7d7-7930-4224-9872-0fc167f1f391"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:23:49.931918 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:49.931898 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/115b91e0-f78d-4edf-9f99-8a497776c921-kube-api-access-kxhmp" pod "115b91e0-f78d-4edf-9f99-8a497776c921" (UID: "115b91e0-f78d-4edf-9f99-8a497776c921"). InnerVolumeSpecName "kube-api-access-kxhmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:23:49.932047 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:49.932027 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f717a7d7-7930-4224-9872-0fc167f1f391-istio-podinfo" pod "f717a7d7-7930-4224-9872-0fc167f1f391" (UID: "f717a7d7-7930-4224-9872-0fc167f1f391"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:23:49.932720 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:49.932690 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f717a7d7-7930-4224-9872-0fc167f1f391-kube-api-access-cmfh7" pod "f717a7d7-7930-4224-9872-0fc167f1f391" (UID: "f717a7d7-7930-4224-9872-0fc167f1f391"). InnerVolumeSpecName "kube-api-access-cmfh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:23:49.933241 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:49.933214 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/115b91e0-f78d-4edf-9f99-8a497776c921-istio-envoy" pod "115b91e0-f78d-4edf-9f99-8a497776c921" (UID: "115b91e0-f78d-4edf-9f99-8a497776c921"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:23:50.026334 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:50.026236 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxhmp\" (UniqueName: \"kubernetes.io/projected/115b91e0-f78d-4edf-9f99-8a497776c921-kube-api-access-kxhmp\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:23:50.026334 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:50.026271 4982 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/115b91e0-f78d-4edf-9f99-8a497776c921-istio-envoy\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:23:50.026334 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:50.026284 4982 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f717a7d7-7930-4224-9872-0fc167f1f391-workload-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:23:50.026334 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:50.026295 4982 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/115b91e0-f78d-4edf-9f99-8a497776c921-credential-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:23:50.026334 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:50.026310 4982 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/115b91e0-f78d-4edf-9f99-8a497776c921-istio-podinfo\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:23:50.026334 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:50.026322 4982 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f717a7d7-7930-4224-9872-0fc167f1f391-workload-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:23:50.026334 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:50.026334 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmfh7\" (UniqueName: \"kubernetes.io/projected/f717a7d7-7930-4224-9872-0fc167f1f391-kube-api-access-cmfh7\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:23:50.026709 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:50.026346 4982 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f717a7d7-7930-4224-9872-0fc167f1f391-istio-envoy\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:23:50.026709 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:50.026357 4982 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/115b91e0-f78d-4edf-9f99-8a497776c921-workload-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:23:50.026709 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:50.026367 4982 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/115b91e0-f78d-4edf-9f99-8a497776c921-istio-token\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:23:50.026709 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:50.026378 4982 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f717a7d7-7930-4224-9872-0fc167f1f391-credential-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:23:50.026709 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:50.026389 4982 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/115b91e0-f78d-4edf-9f99-8a497776c921-workload-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:23:50.026709 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:50.026400 4982 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f717a7d7-7930-4224-9872-0fc167f1f391-istiod-ca-cert\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:23:50.026709 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:50.026411 4982 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f717a7d7-7930-4224-9872-0fc167f1f391-istio-podinfo\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:23:50.026709 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:50.026422 4982 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/115b91e0-f78d-4edf-9f99-8a497776c921-istiod-ca-cert\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:23:50.026709 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:50.026433 4982 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f717a7d7-7930-4224-9872-0fc167f1f391-istio-token\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:23:50.026709 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:50.026444 4982 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f717a7d7-7930-4224-9872-0fc167f1f391-istio-data\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:23:50.026709 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:50.026454 4982 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/115b91e0-f78d-4edf-9f99-8a497776c921-istio-data\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:23:50.076299 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:50.076267 4982 generic.go:356] "Generic (PLEG): container finished" podID="f717a7d7-7930-4224-9872-0fc167f1f391" containerID="36ede143e387390f1be9ed41a11ce57b62246f4b77c98b16b2581042a83edc5d" exitCode=0 Oct 30 14:23:50.076497 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:50.076323 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8" Oct 30 14:23:50.076497 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:50.076358 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8" event={"ID":"f717a7d7-7930-4224-9872-0fc167f1f391","Type":"ContainerDied","Data":"36ede143e387390f1be9ed41a11ce57b62246f4b77c98b16b2581042a83edc5d"} Oct 30 14:23:50.076497 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:50.076428 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8" event={"ID":"f717a7d7-7930-4224-9872-0fc167f1f391","Type":"ContainerDied","Data":"4d371454a8c6efca3e445853261240f49160bb6c1148cee36581fd2743fab4ab"} Oct 30 14:23:50.076497 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:50.076462 4982 scope.go:122] "RemoveContainer" containerID="36ede143e387390f1be9ed41a11ce57b62246f4b77c98b16b2581042a83edc5d" Oct 30 14:23:50.077872 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:50.077847 4982 generic.go:356] "Generic (PLEG): container finished" podID="115b91e0-f78d-4edf-9f99-8a497776c921" containerID="d3732b1e3b5ccc4c94f6fb8efbfa4e7e7593c6f5b38999eb6d9195836398d286" exitCode=0 Oct 30 14:23:50.077996 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:50.077898 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77" Oct 30 14:23:50.077996 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:50.077933 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77" event={"ID":"115b91e0-f78d-4edf-9f99-8a497776c921","Type":"ContainerDied","Data":"d3732b1e3b5ccc4c94f6fb8efbfa4e7e7593c6f5b38999eb6d9195836398d286"} Oct 30 14:23:50.077996 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:50.077988 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77" event={"ID":"115b91e0-f78d-4edf-9f99-8a497776c921","Type":"ContainerDied","Data":"c67cffb7fdc00e7cf803200b7aeaa47ce7dbaa1bbf8a9b83eea5779839ff38c8"} Oct 30 14:23:50.088320 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:50.088290 4982 scope.go:122] "RemoveContainer" containerID="36ede143e387390f1be9ed41a11ce57b62246f4b77c98b16b2581042a83edc5d" Oct 30 14:23:50.088848 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:23:50.088818 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36ede143e387390f1be9ed41a11ce57b62246f4b77c98b16b2581042a83edc5d\": container with ID starting with 36ede143e387390f1be9ed41a11ce57b62246f4b77c98b16b2581042a83edc5d not found: ID does not exist" containerID="36ede143e387390f1be9ed41a11ce57b62246f4b77c98b16b2581042a83edc5d" Oct 30 14:23:50.088958 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:50.088862 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36ede143e387390f1be9ed41a11ce57b62246f4b77c98b16b2581042a83edc5d"} err="failed to get container status \"36ede143e387390f1be9ed41a11ce57b62246f4b77c98b16b2581042a83edc5d\": rpc error: code = NotFound desc = could not find container \"36ede143e387390f1be9ed41a11ce57b62246f4b77c98b16b2581042a83edc5d\": container with ID starting with 36ede143e387390f1be9ed41a11ce57b62246f4b77c98b16b2581042a83edc5d not found: ID does not exist" Oct 30 14:23:50.088958 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:50.088889 4982 scope.go:122] "RemoveContainer" containerID="d3732b1e3b5ccc4c94f6fb8efbfa4e7e7593c6f5b38999eb6d9195836398d286" Oct 30 14:23:50.101839 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:50.101814 4982 scope.go:122] "RemoveContainer" containerID="d3732b1e3b5ccc4c94f6fb8efbfa4e7e7593c6f5b38999eb6d9195836398d286" Oct 30 14:23:50.102164 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:23:50.102143 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3732b1e3b5ccc4c94f6fb8efbfa4e7e7593c6f5b38999eb6d9195836398d286\": container with ID starting with d3732b1e3b5ccc4c94f6fb8efbfa4e7e7593c6f5b38999eb6d9195836398d286 not found: ID does not exist" containerID="d3732b1e3b5ccc4c94f6fb8efbfa4e7e7593c6f5b38999eb6d9195836398d286" Oct 30 14:23:50.102249 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:50.102175 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3732b1e3b5ccc4c94f6fb8efbfa4e7e7593c6f5b38999eb6d9195836398d286"} err="failed to get container status \"d3732b1e3b5ccc4c94f6fb8efbfa4e7e7593c6f5b38999eb6d9195836398d286\": rpc error: code = NotFound desc = could not find container \"d3732b1e3b5ccc4c94f6fb8efbfa4e7e7593c6f5b38999eb6d9195836398d286\": container with ID starting with d3732b1e3b5ccc4c94f6fb8efbfa4e7e7593c6f5b38999eb6d9195836398d286 not found: ID does not exist" Oct 30 14:23:50.135407 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:50.135371 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77"] Oct 30 14:23:50.146381 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:50.146345 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-nxs77"] Oct 30 14:23:50.186013 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:50.185966 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8"] Oct 30 14:23:50.202789 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:50.202753 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-9kfm8"] Oct 30 14:23:50.247127 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:50.247091 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:50.442669 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:50.437913 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="115b91e0-f78d-4edf-9f99-8a497776c921" path="/var/lib/kubelet/pods/115b91e0-f78d-4edf-9f99-8a497776c921/volumes" Oct 30 14:23:50.452497 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:50.438626 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f717a7d7-7930-4224-9872-0fc167f1f391" path="/var/lib/kubelet/pods/f717a7d7-7930-4224-9872-0fc167f1f391/volumes" Oct 30 14:23:50.452497 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:50.451020 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:50.625858 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:50.622741 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:50.685186 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:50.685155 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ntlqq_dc967875-1546-40bb-8376-fde09416c204/console-operator/2.log" Oct 30 14:23:50.704708 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:50.704589 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ntlqq_dc967875-1546-40bb-8376-fde09416c204/console-operator/2.log" Oct 30 14:23:50.706236 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:50.706210 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhxjf_aca47fe8-7068-4743-b6ab-283ee0874f66/ovn-acl-logging/0.log" Oct 30 14:23:50.710749 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:50.710728 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhxjf_aca47fe8-7068-4743-b6ab-283ee0874f66/ovn-acl-logging/0.log" Oct 30 14:23:50.896992 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:50.896955 4982 scope.go:122] "RemoveContainer" containerID="d3a8c712cd8a3e327fc14d04987c71e04245084a562c3535f366ce37d4850fbb" Oct 30 14:23:51.371716 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:51.369626 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:51.493661 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:51.493159 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn"] Oct 30 14:23:51.494191 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:51.494152 4982 cpu_manager.go:414] "containerMap: removing container" podUID="f717a7d7-7930-4224-9872-0fc167f1f391" containerName="istio-proxy" Oct 30 14:23:51.494191 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:51.494185 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f717a7d7-7930-4224-9872-0fc167f1f391" containerName="istio-proxy" Oct 30 14:23:51.494348 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:51.494209 4982 cpu_manager.go:414] "containerMap: removing container" podUID="115b91e0-f78d-4edf-9f99-8a497776c921" containerName="istio-proxy" Oct 30 14:23:51.494348 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:51.494217 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="115b91e0-f78d-4edf-9f99-8a497776c921" containerName="istio-proxy" Oct 30 14:23:51.494413 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:51.494354 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="f717a7d7-7930-4224-9872-0fc167f1f391" containerName="istio-proxy" Oct 30 14:23:51.494413 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:51.494372 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="115b91e0-f78d-4edf-9f99-8a497776c921" containerName="istio-proxy" Oct 30 14:23:51.500632 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:51.500610 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn" Oct 30 14:23:51.528242 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:51.528211 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f3c580-7bff98f799-c6d7q"] Oct 30 14:23:51.535308 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:51.535277 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3c580-7bff98f799-c6d7q" Oct 30 14:23:51.541797 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:51.541754 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn"] Oct 30 14:23:51.559053 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:51.559022 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-f3c580-dockercfg-mzp7x\"" Oct 30 14:23:51.559277 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:51.559250 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-f3c580\"" Oct 30 14:23:51.559375 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:51.559252 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f3c580-7bff98f799-c6d7q"] Oct 30 14:23:51.569087 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:51.569052 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:51.663712 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:51.659780 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f3c580-6d587b47f8-zthxw"] Oct 30 14:23:51.682561 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:51.681746 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb9065d6-3192-463a-bfb3-bda7753fefe2-workload-socket\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn\" (UID: \"fb9065d6-3192-463a-bfb3-bda7753fefe2\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn" Oct 30 14:23:51.682561 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:51.681891 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb9065d6-3192-463a-bfb3-bda7753fefe2-istio-envoy\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn\" (UID: \"fb9065d6-3192-463a-bfb3-bda7753fefe2\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn" Oct 30 14:23:51.682561 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:51.681920 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz7b7\" (UniqueName: \"kubernetes.io/projected/a2fc2354-7034-40dd-9086-ef67d615ec96-kube-api-access-zz7b7\") pod \"payload-pre-processing-e2e-inf-a-f3c580-7bff98f799-c6d7q\" (UID: \"a2fc2354-7034-40dd-9086-ef67d615ec96\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3c580-7bff98f799-c6d7q" Oct 30 14:23:51.682561 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:51.681966 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb9065d6-3192-463a-bfb3-bda7753fefe2-credential-socket\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn\" (UID: \"fb9065d6-3192-463a-bfb3-bda7753fefe2\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn" Oct 30 14:23:51.682561 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:51.682007 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f57l\" (UniqueName: \"kubernetes.io/projected/fb9065d6-3192-463a-bfb3-bda7753fefe2-kube-api-access-6f57l\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn\" (UID: \"fb9065d6-3192-463a-bfb3-bda7753fefe2\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn" Oct 30 14:23:51.682561 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:51.682051 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb9065d6-3192-463a-bfb3-bda7753fefe2-workload-certs\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn\" (UID: \"fb9065d6-3192-463a-bfb3-bda7753fefe2\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn" Oct 30 14:23:51.682561 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:51.682111 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2fc2354-7034-40dd-9086-ef67d615ec96-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f3c580-7bff98f799-c6d7q\" (UID: \"a2fc2354-7034-40dd-9086-ef67d615ec96\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3c580-7bff98f799-c6d7q" Oct 30 14:23:51.682561 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:51.682157 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb9065d6-3192-463a-bfb3-bda7753fefe2-istio-podinfo\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn\" (UID: \"fb9065d6-3192-463a-bfb3-bda7753fefe2\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn" Oct 30 14:23:51.682561 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:51.682215 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb9065d6-3192-463a-bfb3-bda7753fefe2-istio-token\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn\" (UID: \"fb9065d6-3192-463a-bfb3-bda7753fefe2\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn" Oct 30 14:23:51.682561 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:51.682318 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb9065d6-3192-463a-bfb3-bda7753fefe2-istio-data\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn\" (UID: \"fb9065d6-3192-463a-bfb3-bda7753fefe2\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn" Oct 30 14:23:51.682561 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:51.682348 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb9065d6-3192-463a-bfb3-bda7753fefe2-istiod-ca-cert\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn\" (UID: \"fb9065d6-3192-463a-bfb3-bda7753fefe2\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn" Oct 30 14:23:51.684244 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:51.683543 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f3c580-6d587b47f8-zthxw" Oct 30 14:23:51.809665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:51.790257 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb9065d6-3192-463a-bfb3-bda7753fefe2-istiod-ca-cert\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn\" (UID: \"fb9065d6-3192-463a-bfb3-bda7753fefe2\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn" Oct 30 14:23:51.809665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:51.790324 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb9065d6-3192-463a-bfb3-bda7753fefe2-workload-socket\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn\" (UID: \"fb9065d6-3192-463a-bfb3-bda7753fefe2\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn" Oct 30 14:23:51.809665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:51.790448 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb9065d6-3192-463a-bfb3-bda7753fefe2-istio-envoy\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn\" (UID: \"fb9065d6-3192-463a-bfb3-bda7753fefe2\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn" Oct 30 14:23:51.809665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:51.790475 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz7b7\" (UniqueName: \"kubernetes.io/projected/a2fc2354-7034-40dd-9086-ef67d615ec96-kube-api-access-zz7b7\") pod \"payload-pre-processing-e2e-inf-a-f3c580-7bff98f799-c6d7q\" (UID: \"a2fc2354-7034-40dd-9086-ef67d615ec96\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3c580-7bff98f799-c6d7q" Oct 30 14:23:51.809665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:51.790522 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb9065d6-3192-463a-bfb3-bda7753fefe2-credential-socket\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn\" (UID: \"fb9065d6-3192-463a-bfb3-bda7753fefe2\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn" Oct 30 14:23:51.809665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:51.790564 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f57l\" (UniqueName: \"kubernetes.io/projected/fb9065d6-3192-463a-bfb3-bda7753fefe2-kube-api-access-6f57l\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn\" (UID: \"fb9065d6-3192-463a-bfb3-bda7753fefe2\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn" Oct 30 14:23:51.809665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:51.790604 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wrfq\" (UniqueName: \"kubernetes.io/projected/522b3312-a38d-4253-998d-226d3bb2ae62-kube-api-access-8wrfq\") pod \"payload-processing-e2e-inf-a-f3c580-6d587b47f8-zthxw\" (UID: \"522b3312-a38d-4253-998d-226d3bb2ae62\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f3c580-6d587b47f8-zthxw" Oct 30 14:23:51.809665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:51.790660 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb9065d6-3192-463a-bfb3-bda7753fefe2-workload-certs\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn\" (UID: \"fb9065d6-3192-463a-bfb3-bda7753fefe2\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn" Oct 30 14:23:51.809665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:51.790725 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2fc2354-7034-40dd-9086-ef67d615ec96-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f3c580-7bff98f799-c6d7q\" (UID: \"a2fc2354-7034-40dd-9086-ef67d615ec96\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3c580-7bff98f799-c6d7q" Oct 30 14:23:51.809665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:51.790775 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb9065d6-3192-463a-bfb3-bda7753fefe2-istio-podinfo\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn\" (UID: \"fb9065d6-3192-463a-bfb3-bda7753fefe2\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn" Oct 30 14:23:51.809665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:51.790830 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb9065d6-3192-463a-bfb3-bda7753fefe2-istio-token\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn\" (UID: \"fb9065d6-3192-463a-bfb3-bda7753fefe2\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn" Oct 30 14:23:51.809665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:51.790869 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/522b3312-a38d-4253-998d-226d3bb2ae62-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f3c580-6d587b47f8-zthxw\" (UID: \"522b3312-a38d-4253-998d-226d3bb2ae62\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f3c580-6d587b47f8-zthxw" Oct 30 14:23:51.809665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:51.790970 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb9065d6-3192-463a-bfb3-bda7753fefe2-istio-data\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn\" (UID: \"fb9065d6-3192-463a-bfb3-bda7753fefe2\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn" Oct 30 14:23:51.809665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:51.791405 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb9065d6-3192-463a-bfb3-bda7753fefe2-istio-data\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn\" (UID: \"fb9065d6-3192-463a-bfb3-bda7753fefe2\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn" Oct 30 14:23:51.824682 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:51.818183 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb9065d6-3192-463a-bfb3-bda7753fefe2-istiod-ca-cert\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn\" (UID: \"fb9065d6-3192-463a-bfb3-bda7753fefe2\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn" Oct 30 14:23:51.825843 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:51.825594 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f3c580-6d587b47f8-zthxw"] Oct 30 14:23:51.837088 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:51.829076 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2fc2354-7034-40dd-9086-ef67d615ec96-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f3c580-7bff98f799-c6d7q\" (UID: \"a2fc2354-7034-40dd-9086-ef67d615ec96\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3c580-7bff98f799-c6d7q" Oct 30 14:23:51.841166 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:51.840980 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb9065d6-3192-463a-bfb3-bda7753fefe2-workload-socket\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn\" (UID: \"fb9065d6-3192-463a-bfb3-bda7753fefe2\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn" Oct 30 14:23:51.841476 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:51.841426 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb9065d6-3192-463a-bfb3-bda7753fefe2-workload-certs\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn\" (UID: \"fb9065d6-3192-463a-bfb3-bda7753fefe2\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn" Oct 30 14:23:51.851675 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:51.851022 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb9065d6-3192-463a-bfb3-bda7753fefe2-credential-socket\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn\" (UID: \"fb9065d6-3192-463a-bfb3-bda7753fefe2\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn" Oct 30 14:23:51.856916 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:51.856889 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f57l\" (UniqueName: \"kubernetes.io/projected/fb9065d6-3192-463a-bfb3-bda7753fefe2-kube-api-access-6f57l\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn\" (UID: \"fb9065d6-3192-463a-bfb3-bda7753fefe2\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn" Oct 30 14:23:51.859950 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:51.859924 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb9065d6-3192-463a-bfb3-bda7753fefe2-istio-token\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn\" (UID: \"fb9065d6-3192-463a-bfb3-bda7753fefe2\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn" Oct 30 14:23:51.866581 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:51.866517 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb9065d6-3192-463a-bfb3-bda7753fefe2-istio-podinfo\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn\" (UID: \"fb9065d6-3192-463a-bfb3-bda7753fefe2\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn" Oct 30 14:23:51.867609 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:51.867548 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz7b7\" (UniqueName: \"kubernetes.io/projected/a2fc2354-7034-40dd-9086-ef67d615ec96-kube-api-access-zz7b7\") pod \"payload-pre-processing-e2e-inf-a-f3c580-7bff98f799-c6d7q\" (UID: \"a2fc2354-7034-40dd-9086-ef67d615ec96\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3c580-7bff98f799-c6d7q" Oct 30 14:23:51.872433 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:51.872149 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb9065d6-3192-463a-bfb3-bda7753fefe2-istio-envoy\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn\" (UID: \"fb9065d6-3192-463a-bfb3-bda7753fefe2\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn" Oct 30 14:23:51.872433 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:51.872197 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3c580-7bff98f799-c6d7q" Oct 30 14:23:51.895792 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:51.895754 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wrfq\" (UniqueName: \"kubernetes.io/projected/522b3312-a38d-4253-998d-226d3bb2ae62-kube-api-access-8wrfq\") pod \"payload-processing-e2e-inf-a-f3c580-6d587b47f8-zthxw\" (UID: \"522b3312-a38d-4253-998d-226d3bb2ae62\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f3c580-6d587b47f8-zthxw" Oct 30 14:23:51.896096 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:51.896074 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/522b3312-a38d-4253-998d-226d3bb2ae62-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f3c580-6d587b47f8-zthxw\" (UID: \"522b3312-a38d-4253-998d-226d3bb2ae62\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f3c580-6d587b47f8-zthxw" Oct 30 14:23:51.897200 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:51.897178 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/522b3312-a38d-4253-998d-226d3bb2ae62-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f3c580-6d587b47f8-zthxw\" (UID: \"522b3312-a38d-4253-998d-226d3bb2ae62\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f3c580-6d587b47f8-zthxw" Oct 30 14:23:51.926061 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:51.925962 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wrfq\" (UniqueName: \"kubernetes.io/projected/522b3312-a38d-4253-998d-226d3bb2ae62-kube-api-access-8wrfq\") pod \"payload-processing-e2e-inf-a-f3c580-6d587b47f8-zthxw\" (UID: \"522b3312-a38d-4253-998d-226d3bb2ae62\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f3c580-6d587b47f8-zthxw" Oct 30 14:23:52.132669 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:52.130232 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f3c580-6d587b47f8-zthxw" Oct 30 14:23:52.132669 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:52.130849 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn" Oct 30 14:23:52.212077 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:23:52.212030 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2fc2354_7034_40dd_9086_ef67d615ec96.slice/crio-25d17f35fee99ed05cd0f1038dc2aebcb02702f1129dce2771b04979ad4fcd52 WatchSource:0}: Error finding container 25d17f35fee99ed05cd0f1038dc2aebcb02702f1129dce2771b04979ad4fcd52: Status 404 returned error can't find the container with id 25d17f35fee99ed05cd0f1038dc2aebcb02702f1129dce2771b04979ad4fcd52 Oct 30 14:23:52.228609 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:52.228562 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f3c580-7bff98f799-c6d7q"] Oct 30 14:23:52.458564 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:52.458533 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f3c580-6d587b47f8-zthxw"] Oct 30 14:23:52.458850 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:23:52.458826 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod522b3312_a38d_4253_998d_226d3bb2ae62.slice/crio-f0814caeb11905efd1df4526886f890f609424276ae5367c3821db361b4464a2 WatchSource:0}: Error finding container f0814caeb11905efd1df4526886f890f609424276ae5367c3821db361b4464a2: Status 404 returned error can't find the container with id f0814caeb11905efd1df4526886f890f609424276ae5367c3821db361b4464a2 Oct 30 14:23:52.495786 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:52.495761 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn"] Oct 30 14:23:52.503086 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:52.503033 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:23:52.503262 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:52.503239 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:23:52.503354 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:52.503337 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:23:52.873177 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:52.873130 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:53.151819 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:53.151713 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn" event={"ID":"fb9065d6-3192-463a-bfb3-bda7753fefe2","Type":"ContainerStarted","Data":"84d90f3062ee12186fa43cb855201bd800ded5f1f48baa3821ff9ac6284630bb"} Oct 30 14:23:53.151819 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:53.151812 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn" event={"ID":"fb9065d6-3192-463a-bfb3-bda7753fefe2","Type":"ContainerStarted","Data":"5d74b0f53a9eebaebe050ee045373f6f45a11a32817e55bbfdb034dc55029033"} Oct 30 14:23:53.153474 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:53.153438 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f3c580-6d587b47f8-zthxw" event={"ID":"522b3312-a38d-4253-998d-226d3bb2ae62","Type":"ContainerStarted","Data":"55cacd2b97162e09df600d8d9bd9fe12e9403dab34763ede166dfbec82c36827"} Oct 30 14:23:53.153609 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:53.153490 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f3c580-6d587b47f8-zthxw" event={"ID":"522b3312-a38d-4253-998d-226d3bb2ae62","Type":"ContainerStarted","Data":"f0814caeb11905efd1df4526886f890f609424276ae5367c3821db361b4464a2"} Oct 30 14:23:53.153609 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:53.153513 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-f3c580-6d587b47f8-zthxw" Oct 30 14:23:53.154966 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:53.154944 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3c580-7bff98f799-c6d7q" event={"ID":"a2fc2354-7034-40dd-9086-ef67d615ec96","Type":"ContainerStarted","Data":"d6aa943ac56d42cbd55cfa730e0da33f617931fcf5a68a1d5a8f69ea68526731"} Oct 30 14:23:53.155050 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:53.154981 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3c580-7bff98f799-c6d7q" event={"ID":"a2fc2354-7034-40dd-9086-ef67d615ec96","Type":"ContainerStarted","Data":"25d17f35fee99ed05cd0f1038dc2aebcb02702f1129dce2771b04979ad4fcd52"} Oct 30 14:23:53.155091 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:53.155067 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3c580-7bff98f799-c6d7q" Oct 30 14:23:53.204562 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:53.204491 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn" podStartSLOduration=2.204473394 podStartE2EDuration="2.204473394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:23:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:23:53.203417075 +0000 UTC m=+2102.947551118" watchObservedRunningTime="2026-10-30 14:23:53.204473394 +0000 UTC m=+2102.948607436" Oct 30 14:23:53.247070 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:53.247012 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-f3c580-6d587b47f8-zthxw" podStartSLOduration=2.246999651 podStartE2EDuration="2.246999651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:23:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:23:53.24665417 +0000 UTC m=+2102.990788203" watchObservedRunningTime="2026-10-30 14:23:53.246999651 +0000 UTC m=+2102.991133691" Oct 30 14:23:53.291414 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:53.291353 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3c580-7bff98f799-c6d7q" podStartSLOduration=2.291336327 podStartE2EDuration="2.291336327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:23:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:23:53.28946642 +0000 UTC m=+2103.033600460" watchObservedRunningTime="2026-10-30 14:23:53.291336327 +0000 UTC m=+2103.035470371" Oct 30 14:23:54.131790 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:54.131068 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn" Oct 30 14:23:54.132709 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:54.132682 4982 patch_prober.go:29] interesting pod/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 14:23:54.132806 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:54.132737 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn" podUID="fb9065d6-3192-463a-bfb3-bda7753fefe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 14:23:54.261129 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:54.261095 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:54.263258 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:54.263234 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-409364-58bb5dbb9b-rdzd5"] Oct 30 14:23:54.267757 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:54.267737 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-409364-58bb5dbb9b-rdzd5" Oct 30 14:23:54.313671 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:54.310834 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-409364-58bb5dbb9b-rdzd5"] Oct 30 14:23:54.322039 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:54.322008 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-409364\"" Oct 30 14:23:54.470716 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:54.469634 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft5ds\" (UniqueName: \"kubernetes.io/projected/ef5aa554-9820-4315-9d99-e7143e52f56a-kube-api-access-ft5ds\") pod \"payload-processing-e2e-inf-a-409364-58bb5dbb9b-rdzd5\" (UID: \"ef5aa554-9820-4315-9d99-e7143e52f56a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-409364-58bb5dbb9b-rdzd5" Oct 30 14:23:54.470716 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:54.469730 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef5aa554-9820-4315-9d99-e7143e52f56a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-409364-58bb5dbb9b-rdzd5\" (UID: \"ef5aa554-9820-4315-9d99-e7143e52f56a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-409364-58bb5dbb9b-rdzd5" Oct 30 14:23:54.535662 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:54.528159 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:54.578674 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:54.577497 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft5ds\" (UniqueName: \"kubernetes.io/projected/ef5aa554-9820-4315-9d99-e7143e52f56a-kube-api-access-ft5ds\") pod \"payload-processing-e2e-inf-a-409364-58bb5dbb9b-rdzd5\" (UID: \"ef5aa554-9820-4315-9d99-e7143e52f56a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-409364-58bb5dbb9b-rdzd5" Oct 30 14:23:54.578674 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:54.577559 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef5aa554-9820-4315-9d99-e7143e52f56a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-409364-58bb5dbb9b-rdzd5\" (UID: \"ef5aa554-9820-4315-9d99-e7143e52f56a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-409364-58bb5dbb9b-rdzd5" Oct 30 14:23:54.578674 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:54.578613 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef5aa554-9820-4315-9d99-e7143e52f56a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-409364-58bb5dbb9b-rdzd5\" (UID: \"ef5aa554-9820-4315-9d99-e7143e52f56a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-409364-58bb5dbb9b-rdzd5" Oct 30 14:23:54.654670 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:54.653857 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft5ds\" (UniqueName: \"kubernetes.io/projected/ef5aa554-9820-4315-9d99-e7143e52f56a-kube-api-access-ft5ds\") pod \"payload-processing-e2e-inf-a-409364-58bb5dbb9b-rdzd5\" (UID: \"ef5aa554-9820-4315-9d99-e7143e52f56a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-409364-58bb5dbb9b-rdzd5" Oct 30 14:23:54.913663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:54.912729 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-409364-58bb5dbb9b-rdzd5" Oct 30 14:23:55.131286 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:55.131255 4982 patch_prober.go:29] interesting pod/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 14:23:55.131494 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:55.131467 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn" podUID="fb9065d6-3192-463a-bfb3-bda7753fefe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 14:23:55.231099 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:55.231068 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-409364-58bb5dbb9b-rdzd5"] Oct 30 14:23:55.231947 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:23:55.231917 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef5aa554_9820_4315_9d99_e7143e52f56a.slice/crio-1e973af9327d74155d6298f21262800e701e23d945596604c459feee71ee58d3 WatchSource:0}: Error finding container 1e973af9327d74155d6298f21262800e701e23d945596604c459feee71ee58d3: Status 404 returned error can't find the container with id 1e973af9327d74155d6298f21262800e701e23d945596604c459feee71ee58d3 Oct 30 14:23:55.407665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:55.406895 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:56.131398 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:56.131361 4982 patch_prober.go:29] interesting pod/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 14:23:56.131621 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:56.131425 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn" podUID="fb9065d6-3192-463a-bfb3-bda7753fefe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 14:23:56.195991 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:56.195949 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-409364-58bb5dbb9b-rdzd5" event={"ID":"ef5aa554-9820-4315-9d99-e7143e52f56a","Type":"ContainerStarted","Data":"e657bc211163ef3bb11b85680d1341f9ceb55a0712ed499718b59d0c75767cae"} Oct 30 14:23:56.196248 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:56.196226 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-409364-58bb5dbb9b-rdzd5" event={"ID":"ef5aa554-9820-4315-9d99-e7143e52f56a","Type":"ContainerStarted","Data":"1e973af9327d74155d6298f21262800e701e23d945596604c459feee71ee58d3"} Oct 30 14:23:56.213666 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:56.197425 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-409364-58bb5dbb9b-rdzd5" Oct 30 14:23:56.247566 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:56.247512 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-409364-58bb5dbb9b-rdzd5" podStartSLOduration=2.247500102 podStartE2EDuration="2.247500102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:23:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:23:56.243988011 +0000 UTC m=+2105.988122052" watchObservedRunningTime="2026-10-30 14:23:56.247500102 +0000 UTC m=+2105.991634143" Oct 30 14:23:57.131861 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:57.131815 4982 patch_prober.go:29] interesting pod/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 14:23:57.132045 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:57.131887 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn" podUID="fb9065d6-3192-463a-bfb3-bda7753fefe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 14:23:57.618814 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:57.618775 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb"] Oct 30 14:23:57.619254 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:57.619029 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb" podUID="9f29ce71-fed6-4c16-b200-b61362a9a60f" containerName="istio-proxy" containerID="cri-o://20125e17829caf31e07b7400dceec76b4f8ee75f9a5b2bba7237cf1049be710f" gracePeriod=30 Oct 30 14:23:57.765555 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:57.765515 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:57.970757 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:57.969400 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:23:58.050536 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:23:58.050501 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:23:58.131531 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:58.131494 4982 patch_prober.go:29] interesting pod/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 14:23:58.131717 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:58.131554 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn" podUID="fb9065d6-3192-463a-bfb3-bda7753fefe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 14:23:59.207898 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:59.207869 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn" Oct 30 14:23:59.209849 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:59.209824 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn" Oct 30 14:23:59.214081 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:23:59.214025 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn" Oct 30 14:24:00.040853 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:00.040812 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:24:01.556939 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:01.556898 4982 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 14:24:01.557347 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:01.556968 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb" podUID="9f29ce71-fed6-4c16-b200-b61362a9a60f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 14:24:01.834351 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:01.834244 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:24:02.907847 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:02.907811 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb" Oct 30 14:24:02.962329 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:02.962293 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f29ce71-fed6-4c16-b200-b61362a9a60f-kube-api-access-z5hrc\" (UniqueName: \"kubernetes.io/projected/9f29ce71-fed6-4c16-b200-b61362a9a60f-kube-api-access-z5hrc\") pod \"9f29ce71-fed6-4c16-b200-b61362a9a60f\" (UID: \"9f29ce71-fed6-4c16-b200-b61362a9a60f\") " Oct 30 14:24:02.962527 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:02.962374 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f29ce71-fed6-4c16-b200-b61362a9a60f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f29ce71-fed6-4c16-b200-b61362a9a60f-credential-socket\") pod \"9f29ce71-fed6-4c16-b200-b61362a9a60f\" (UID: \"9f29ce71-fed6-4c16-b200-b61362a9a60f\") " Oct 30 14:24:02.962527 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:02.962473 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f29ce71-fed6-4c16-b200-b61362a9a60f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f29ce71-fed6-4c16-b200-b61362a9a60f-workload-certs\") pod \"9f29ce71-fed6-4c16-b200-b61362a9a60f\" (UID: \"9f29ce71-fed6-4c16-b200-b61362a9a60f\") " Oct 30 14:24:02.962527 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:02.962509 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f29ce71-fed6-4c16-b200-b61362a9a60f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f29ce71-fed6-4c16-b200-b61362a9a60f-istiod-ca-cert\") pod \"9f29ce71-fed6-4c16-b200-b61362a9a60f\" (UID: \"9f29ce71-fed6-4c16-b200-b61362a9a60f\") " Oct 30 14:24:02.962764 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:02.962597 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f29ce71-fed6-4c16-b200-b61362a9a60f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f29ce71-fed6-4c16-b200-b61362a9a60f-istio-envoy\") pod \"9f29ce71-fed6-4c16-b200-b61362a9a60f\" (UID: \"9f29ce71-fed6-4c16-b200-b61362a9a60f\") " Oct 30 14:24:02.962764 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:02.962696 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f29ce71-fed6-4c16-b200-b61362a9a60f-credential-socket" pod "9f29ce71-fed6-4c16-b200-b61362a9a60f" (UID: "9f29ce71-fed6-4c16-b200-b61362a9a60f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:24:02.962764 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:02.962726 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f29ce71-fed6-4c16-b200-b61362a9a60f-istio-token\" (UniqueName: \"kubernetes.io/projected/9f29ce71-fed6-4c16-b200-b61362a9a60f-istio-token\") pod \"9f29ce71-fed6-4c16-b200-b61362a9a60f\" (UID: \"9f29ce71-fed6-4c16-b200-b61362a9a60f\") " Oct 30 14:24:02.962931 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:02.962735 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f29ce71-fed6-4c16-b200-b61362a9a60f-workload-certs" pod "9f29ce71-fed6-4c16-b200-b61362a9a60f" (UID: "9f29ce71-fed6-4c16-b200-b61362a9a60f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:24:02.962931 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:02.962892 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f29ce71-fed6-4c16-b200-b61362a9a60f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f29ce71-fed6-4c16-b200-b61362a9a60f-istio-data\") pod \"9f29ce71-fed6-4c16-b200-b61362a9a60f\" (UID: \"9f29ce71-fed6-4c16-b200-b61362a9a60f\") " Oct 30 14:24:02.963020 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:02.962932 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f29ce71-fed6-4c16-b200-b61362a9a60f-istiod-ca-cert" pod "9f29ce71-fed6-4c16-b200-b61362a9a60f" (UID: "9f29ce71-fed6-4c16-b200-b61362a9a60f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:24:02.963020 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:02.962938 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f29ce71-fed6-4c16-b200-b61362a9a60f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f29ce71-fed6-4c16-b200-b61362a9a60f-istio-podinfo\") pod \"9f29ce71-fed6-4c16-b200-b61362a9a60f\" (UID: \"9f29ce71-fed6-4c16-b200-b61362a9a60f\") " Oct 30 14:24:02.963020 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:02.963012 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f29ce71-fed6-4c16-b200-b61362a9a60f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f29ce71-fed6-4c16-b200-b61362a9a60f-workload-socket\") pod \"9f29ce71-fed6-4c16-b200-b61362a9a60f\" (UID: \"9f29ce71-fed6-4c16-b200-b61362a9a60f\") " Oct 30 14:24:02.963175 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:02.963155 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f29ce71-fed6-4c16-b200-b61362a9a60f-istio-data" pod "9f29ce71-fed6-4c16-b200-b61362a9a60f" (UID: "9f29ce71-fed6-4c16-b200-b61362a9a60f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:24:02.963281 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:02.963253 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f29ce71-fed6-4c16-b200-b61362a9a60f-workload-socket" pod "9f29ce71-fed6-4c16-b200-b61362a9a60f" (UID: "9f29ce71-fed6-4c16-b200-b61362a9a60f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:24:02.963682 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:02.963662 4982 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f29ce71-fed6-4c16-b200-b61362a9a60f-credential-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:24:02.963770 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:02.963685 4982 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f29ce71-fed6-4c16-b200-b61362a9a60f-workload-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:24:02.963770 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:02.963700 4982 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f29ce71-fed6-4c16-b200-b61362a9a60f-istiod-ca-cert\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:24:02.963770 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:02.963712 4982 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f29ce71-fed6-4c16-b200-b61362a9a60f-istio-data\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:24:02.963770 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:02.963725 4982 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f29ce71-fed6-4c16-b200-b61362a9a60f-workload-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:24:02.964810 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:02.964784 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f29ce71-fed6-4c16-b200-b61362a9a60f-kube-api-access-z5hrc" pod "9f29ce71-fed6-4c16-b200-b61362a9a60f" (UID: "9f29ce71-fed6-4c16-b200-b61362a9a60f"). InnerVolumeSpecName "kube-api-access-z5hrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:24:02.965101 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:02.965079 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f29ce71-fed6-4c16-b200-b61362a9a60f-istio-podinfo" pod "9f29ce71-fed6-4c16-b200-b61362a9a60f" (UID: "9f29ce71-fed6-4c16-b200-b61362a9a60f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:24:02.965188 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:02.965156 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f29ce71-fed6-4c16-b200-b61362a9a60f-istio-envoy" pod "9f29ce71-fed6-4c16-b200-b61362a9a60f" (UID: "9f29ce71-fed6-4c16-b200-b61362a9a60f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:24:02.965304 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:02.965285 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f29ce71-fed6-4c16-b200-b61362a9a60f-istio-token" pod "9f29ce71-fed6-4c16-b200-b61362a9a60f" (UID: "9f29ce71-fed6-4c16-b200-b61362a9a60f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:24:03.065767 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:03.065732 4982 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f29ce71-fed6-4c16-b200-b61362a9a60f-istio-podinfo\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:24:03.065767 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:03.065765 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5hrc\" (UniqueName: \"kubernetes.io/projected/9f29ce71-fed6-4c16-b200-b61362a9a60f-kube-api-access-z5hrc\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:24:03.065971 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:03.065781 4982 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f29ce71-fed6-4c16-b200-b61362a9a60f-istio-envoy\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:24:03.065971 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:03.065794 4982 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f29ce71-fed6-4c16-b200-b61362a9a60f-istio-token\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:24:03.312791 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:03.312736 4982 generic.go:356] "Generic (PLEG): container finished" podID="9f29ce71-fed6-4c16-b200-b61362a9a60f" containerID="20125e17829caf31e07b7400dceec76b4f8ee75f9a5b2bba7237cf1049be710f" exitCode=0 Oct 30 14:24:03.313358 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:03.313333 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb" event={"ID":"9f29ce71-fed6-4c16-b200-b61362a9a60f","Type":"ContainerDied","Data":"20125e17829caf31e07b7400dceec76b4f8ee75f9a5b2bba7237cf1049be710f"} Oct 30 14:24:03.313675 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:03.313651 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb" event={"ID":"9f29ce71-fed6-4c16-b200-b61362a9a60f","Type":"ContainerDied","Data":"f4a141503f1e20fa1f00aceaf68c6caf7fa763c4ddb83ed41d47716331036d58"} Oct 30 14:24:03.313848 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:03.313825 4982 scope.go:122] "RemoveContainer" containerID="20125e17829caf31e07b7400dceec76b4f8ee75f9a5b2bba7237cf1049be710f" Oct 30 14:24:03.314032 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:03.314017 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb" Oct 30 14:24:03.346127 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:03.346103 4982 scope.go:122] "RemoveContainer" containerID="20125e17829caf31e07b7400dceec76b4f8ee75f9a5b2bba7237cf1049be710f" Oct 30 14:24:03.346715 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:24:03.346524 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20125e17829caf31e07b7400dceec76b4f8ee75f9a5b2bba7237cf1049be710f\": container with ID starting with 20125e17829caf31e07b7400dceec76b4f8ee75f9a5b2bba7237cf1049be710f not found: ID does not exist" containerID="20125e17829caf31e07b7400dceec76b4f8ee75f9a5b2bba7237cf1049be710f" Oct 30 14:24:03.346715 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:03.346588 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20125e17829caf31e07b7400dceec76b4f8ee75f9a5b2bba7237cf1049be710f"} err="failed to get container status \"20125e17829caf31e07b7400dceec76b4f8ee75f9a5b2bba7237cf1049be710f\": rpc error: code = NotFound desc = could not find container \"20125e17829caf31e07b7400dceec76b4f8ee75f9a5b2bba7237cf1049be710f\": container with ID starting with 20125e17829caf31e07b7400dceec76b4f8ee75f9a5b2bba7237cf1049be710f not found: ID does not exist" Oct 30 14:24:03.412664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:03.402377 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb"] Oct 30 14:24:03.428277 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:03.427341 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-292pb"] Oct 30 14:24:03.886555 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:03.886509 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:24:04.166742 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:04.166630 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-f3c580-6d587b47f8-zthxw" Oct 30 14:24:04.167122 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:04.166930 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3c580-7bff98f799-c6d7q" Oct 30 14:24:04.400783 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:04.400752 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f29ce71-fed6-4c16-b200-b61362a9a60f" path="/var/lib/kubelet/pods/9f29ce71-fed6-4c16-b200-b61362a9a60f/volumes" Oct 30 14:24:05.740451 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:05.740409 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:24:06.579246 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:06.579204 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm"] Oct 30 14:24:06.580240 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:06.580169 4982 cpu_manager.go:414] "containerMap: removing container" podUID="9f29ce71-fed6-4c16-b200-b61362a9a60f" containerName="istio-proxy" Oct 30 14:24:06.580350 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:06.580248 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f29ce71-fed6-4c16-b200-b61362a9a60f" containerName="istio-proxy" Oct 30 14:24:06.580463 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:06.580450 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f29ce71-fed6-4c16-b200-b61362a9a60f" containerName="istio-proxy" Oct 30 14:24:06.590463 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:06.590435 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm" Oct 30 14:24:06.611728 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:06.611700 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm"] Oct 30 14:24:06.718105 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:06.718066 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20041b11-f00c-4853-acf3-7c210f742f6d-istio-envoy\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm\" (UID: \"20041b11-f00c-4853-acf3-7c210f742f6d\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm" Oct 30 14:24:06.718105 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:06.718099 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20041b11-f00c-4853-acf3-7c210f742f6d-istio-data\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm\" (UID: \"20041b11-f00c-4853-acf3-7c210f742f6d\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm" Oct 30 14:24:06.718368 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:06.718141 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20041b11-f00c-4853-acf3-7c210f742f6d-workload-certs\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm\" (UID: \"20041b11-f00c-4853-acf3-7c210f742f6d\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm" Oct 30 14:24:06.718368 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:06.718261 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20041b11-f00c-4853-acf3-7c210f742f6d-istiod-ca-cert\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm\" (UID: \"20041b11-f00c-4853-acf3-7c210f742f6d\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm" Oct 30 14:24:06.718368 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:06.718287 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qpsv\" (UniqueName: \"kubernetes.io/projected/20041b11-f00c-4853-acf3-7c210f742f6d-kube-api-access-5qpsv\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm\" (UID: \"20041b11-f00c-4853-acf3-7c210f742f6d\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm" Oct 30 14:24:06.718368 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:06.718330 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20041b11-f00c-4853-acf3-7c210f742f6d-istio-podinfo\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm\" (UID: \"20041b11-f00c-4853-acf3-7c210f742f6d\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm" Oct 30 14:24:06.718583 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:06.718407 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20041b11-f00c-4853-acf3-7c210f742f6d-credential-socket\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm\" (UID: \"20041b11-f00c-4853-acf3-7c210f742f6d\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm" Oct 30 14:24:06.718583 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:06.718557 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20041b11-f00c-4853-acf3-7c210f742f6d-istio-token\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm\" (UID: \"20041b11-f00c-4853-acf3-7c210f742f6d\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm" Oct 30 14:24:06.718583 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:06.718579 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20041b11-f00c-4853-acf3-7c210f742f6d-workload-socket\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm\" (UID: \"20041b11-f00c-4853-acf3-7c210f742f6d\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm" Oct 30 14:24:06.819515 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:06.819482 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20041b11-f00c-4853-acf3-7c210f742f6d-istio-envoy\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm\" (UID: \"20041b11-f00c-4853-acf3-7c210f742f6d\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm" Oct 30 14:24:06.819515 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:06.819516 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20041b11-f00c-4853-acf3-7c210f742f6d-istio-data\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm\" (UID: \"20041b11-f00c-4853-acf3-7c210f742f6d\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm" Oct 30 14:24:06.819982 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:06.819554 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20041b11-f00c-4853-acf3-7c210f742f6d-workload-certs\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm\" (UID: \"20041b11-f00c-4853-acf3-7c210f742f6d\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm" Oct 30 14:24:06.819982 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:06.819604 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20041b11-f00c-4853-acf3-7c210f742f6d-istiod-ca-cert\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm\" (UID: \"20041b11-f00c-4853-acf3-7c210f742f6d\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm" Oct 30 14:24:06.819982 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:06.819621 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qpsv\" (UniqueName: \"kubernetes.io/projected/20041b11-f00c-4853-acf3-7c210f742f6d-kube-api-access-5qpsv\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm\" (UID: \"20041b11-f00c-4853-acf3-7c210f742f6d\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm" Oct 30 14:24:06.819982 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:06.819670 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20041b11-f00c-4853-acf3-7c210f742f6d-istio-podinfo\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm\" (UID: \"20041b11-f00c-4853-acf3-7c210f742f6d\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm" Oct 30 14:24:06.819982 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:06.819711 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20041b11-f00c-4853-acf3-7c210f742f6d-credential-socket\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm\" (UID: \"20041b11-f00c-4853-acf3-7c210f742f6d\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm" Oct 30 14:24:06.819982 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:06.819870 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20041b11-f00c-4853-acf3-7c210f742f6d-istio-token\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm\" (UID: \"20041b11-f00c-4853-acf3-7c210f742f6d\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm" Oct 30 14:24:06.819982 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:06.819896 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20041b11-f00c-4853-acf3-7c210f742f6d-workload-socket\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm\" (UID: \"20041b11-f00c-4853-acf3-7c210f742f6d\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm" Oct 30 14:24:06.820423 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:06.820396 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20041b11-f00c-4853-acf3-7c210f742f6d-istio-data\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm\" (UID: \"20041b11-f00c-4853-acf3-7c210f742f6d\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm" Oct 30 14:24:06.820738 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:06.820705 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20041b11-f00c-4853-acf3-7c210f742f6d-workload-certs\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm\" (UID: \"20041b11-f00c-4853-acf3-7c210f742f6d\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm" Oct 30 14:24:06.820868 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:06.820744 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20041b11-f00c-4853-acf3-7c210f742f6d-credential-socket\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm\" (UID: \"20041b11-f00c-4853-acf3-7c210f742f6d\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm" Oct 30 14:24:06.820868 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:06.820811 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20041b11-f00c-4853-acf3-7c210f742f6d-istiod-ca-cert\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm\" (UID: \"20041b11-f00c-4853-acf3-7c210f742f6d\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm" Oct 30 14:24:06.820984 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:06.820944 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20041b11-f00c-4853-acf3-7c210f742f6d-workload-socket\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm\" (UID: \"20041b11-f00c-4853-acf3-7c210f742f6d\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm" Oct 30 14:24:06.822420 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:06.822397 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20041b11-f00c-4853-acf3-7c210f742f6d-istio-podinfo\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm\" (UID: \"20041b11-f00c-4853-acf3-7c210f742f6d\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm" Oct 30 14:24:06.822901 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:06.822871 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20041b11-f00c-4853-acf3-7c210f742f6d-istio-envoy\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm\" (UID: \"20041b11-f00c-4853-acf3-7c210f742f6d\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm" Oct 30 14:24:06.846955 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:06.846897 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qpsv\" (UniqueName: \"kubernetes.io/projected/20041b11-f00c-4853-acf3-7c210f742f6d-kube-api-access-5qpsv\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm\" (UID: \"20041b11-f00c-4853-acf3-7c210f742f6d\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm" Oct 30 14:24:06.847467 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:06.847449 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20041b11-f00c-4853-acf3-7c210f742f6d-istio-token\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm\" (UID: \"20041b11-f00c-4853-acf3-7c210f742f6d\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm" Oct 30 14:24:06.904882 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:06.904840 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm" Oct 30 14:24:07.238578 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:24:07.238543 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20041b11_f00c_4853_acf3_7c210f742f6d.slice/crio-95768511b953bcf6a1005603a5e18d11454442d738eb85db178a457e968eedca WatchSource:0}: Error finding container 95768511b953bcf6a1005603a5e18d11454442d738eb85db178a457e968eedca: Status 404 returned error can't find the container with id 95768511b953bcf6a1005603a5e18d11454442d738eb85db178a457e968eedca Oct 30 14:24:07.240795 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:07.240771 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm"] Oct 30 14:24:07.241901 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:07.241863 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:24:07.241967 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:07.241948 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:24:07.242005 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:07.241986 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:24:07.346619 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:07.346585 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm" event={"ID":"20041b11-f00c-4853-acf3-7c210f742f6d","Type":"ContainerStarted","Data":"95768511b953bcf6a1005603a5e18d11454442d738eb85db178a457e968eedca"} Oct 30 14:24:07.390683 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:07.390117 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:24:07.943633 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:07.943592 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:24:08.058850 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:08.058819 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4bf33a-sgtxq"] Oct 30 14:24:08.071443 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:08.071410 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4bf33a-sgtxq"] Oct 30 14:24:08.084658 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:08.084617 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:24:08.230523 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:08.230430 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:24:08.231408 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:08.231390 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-409364-58bb5dbb9b-rdzd5" Oct 30 14:24:08.355082 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:08.355044 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm" event={"ID":"20041b11-f00c-4853-acf3-7c210f742f6d","Type":"ContainerStarted","Data":"53034c6d7382e2872770a780f003f76a50435ecb7a012daab23a5c6c68c50249"} Oct 30 14:24:08.407435 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:08.407386 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm" podStartSLOduration=2.407371484 podStartE2EDuration="2.407371484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:24:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:24:08.402613255 +0000 UTC m=+2118.146747295" watchObservedRunningTime="2026-10-30 14:24:08.407371484 +0000 UTC m=+2118.151505525" Oct 30 14:24:08.409167 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:08.409137 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69ad8491-34f8-494b-89ba-a9ddd63cac3d" path="/var/lib/kubelet/pods/69ad8491-34f8-494b-89ba-a9ddd63cac3d/volumes" Oct 30 14:24:08.606663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:08.606080 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:24:08.911036 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:08.910930 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm" Oct 30 14:24:08.913516 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:08.913481 4982 patch_prober.go:29] interesting pod/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Oct 30 14:24:08.913688 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:08.913571 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm" podUID="20041b11-f00c-4853-acf3-7c210f742f6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Oct 30 14:24:09.908663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:09.906847 4982 patch_prober.go:29] interesting pod/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Oct 30 14:24:09.908663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:09.906913 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm" podUID="20041b11-f00c-4853-acf3-7c210f742f6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Oct 30 14:24:10.322790 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:10.322750 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:24:10.475231 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:10.475188 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:24:10.912665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:10.911909 4982 patch_prober.go:29] interesting pod/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Oct 30 14:24:10.912665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:10.911978 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm" podUID="20041b11-f00c-4853-acf3-7c210f742f6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Oct 30 14:24:11.909104 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:11.909063 4982 patch_prober.go:29] interesting pod/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Oct 30 14:24:11.909387 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:11.909359 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm" podUID="20041b11-f00c-4853-acf3-7c210f742f6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Oct 30 14:24:12.454928 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:12.454894 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:24:12.913687 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:12.913501 4982 patch_prober.go:29] interesting pod/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Oct 30 14:24:12.913687 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:12.913560 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm" podUID="20041b11-f00c-4853-acf3-7c210f742f6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Oct 30 14:24:13.905730 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:13.905682 4982 patch_prober.go:29] interesting pod/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Oct 30 14:24:13.906262 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:13.905749 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm" podUID="20041b11-f00c-4853-acf3-7c210f742f6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Oct 30 14:24:14.421735 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:14.421700 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:24:14.912942 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:14.912903 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm" Oct 30 14:24:14.913513 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:14.913487 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm" Oct 30 14:24:14.914159 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:14.914134 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm" Oct 30 14:24:16.061700 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:16.061664 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:24:16.252686 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:16.252629 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:24:16.510258 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:16.510162 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:24:18.447775 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:18.447741 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:24:20.382239 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:20.382199 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:24:22.008221 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:22.008185 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:24:22.211917 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:22.211883 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:24:22.407562 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:22.407523 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:24:24.326678 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:24.326625 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:24:26.261857 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:26.261818 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:24:26.420316 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:26.420273 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:24:26.866607 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:26.866571 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea9ed2-nrct6"] Oct 30 14:24:26.872185 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:26.872167 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea9ed2-nrct6" Oct 30 14:24:26.893794 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:26.893747 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea9ed2-nrct6"] Oct 30 14:24:26.902309 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:26.902129 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:24:26.902309 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:26.902277 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:24:27.003099 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:27.003063 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e145e1fb-ad70-46eb-bba7-adbf5cab95b8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ea9ed2-nrct6\" (UID: \"e145e1fb-ad70-46eb-bba7-adbf5cab95b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea9ed2-nrct6" Oct 30 14:24:27.003360 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:27.003332 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qgww\" (UniqueName: \"kubernetes.io/projected/e145e1fb-ad70-46eb-bba7-adbf5cab95b8-kube-api-access-6qgww\") pod \"maas-api-revoke-keys-e2e-api-b-ea9ed2-nrct6\" (UID: \"e145e1fb-ad70-46eb-bba7-adbf5cab95b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea9ed2-nrct6" Oct 30 14:24:27.003414 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:27.003378 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e145e1fb-ad70-46eb-bba7-adbf5cab95b8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ea9ed2-nrct6\" (UID: \"e145e1fb-ad70-46eb-bba7-adbf5cab95b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea9ed2-nrct6" Oct 30 14:24:27.105174 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:27.105138 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qgww\" (UniqueName: \"kubernetes.io/projected/e145e1fb-ad70-46eb-bba7-adbf5cab95b8-kube-api-access-6qgww\") pod \"maas-api-revoke-keys-e2e-api-b-ea9ed2-nrct6\" (UID: \"e145e1fb-ad70-46eb-bba7-adbf5cab95b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea9ed2-nrct6" Oct 30 14:24:27.105342 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:27.105188 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e145e1fb-ad70-46eb-bba7-adbf5cab95b8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ea9ed2-nrct6\" (UID: \"e145e1fb-ad70-46eb-bba7-adbf5cab95b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea9ed2-nrct6" Oct 30 14:24:27.105391 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:27.105358 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e145e1fb-ad70-46eb-bba7-adbf5cab95b8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ea9ed2-nrct6\" (UID: \"e145e1fb-ad70-46eb-bba7-adbf5cab95b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea9ed2-nrct6" Oct 30 14:24:27.105896 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:27.105870 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e145e1fb-ad70-46eb-bba7-adbf5cab95b8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ea9ed2-nrct6\" (UID: \"e145e1fb-ad70-46eb-bba7-adbf5cab95b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea9ed2-nrct6" Oct 30 14:24:27.107722 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:27.107687 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e145e1fb-ad70-46eb-bba7-adbf5cab95b8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ea9ed2-nrct6\" (UID: \"e145e1fb-ad70-46eb-bba7-adbf5cab95b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea9ed2-nrct6" Oct 30 14:24:27.133749 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:27.133676 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qgww\" (UniqueName: \"kubernetes.io/projected/e145e1fb-ad70-46eb-bba7-adbf5cab95b8-kube-api-access-6qgww\") pod \"maas-api-revoke-keys-e2e-api-b-ea9ed2-nrct6\" (UID: \"e145e1fb-ad70-46eb-bba7-adbf5cab95b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea9ed2-nrct6" Oct 30 14:24:27.186580 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:27.186548 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea9ed2-nrct6" Oct 30 14:24:27.508829 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:24:27.508797 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode145e1fb_ad70_46eb_bba7_adbf5cab95b8.slice/crio-39d0d80623631c70ab8619dab493bf7f6ba282ff8fc132b4c01f4dbc032ebdc5 WatchSource:0}: Error finding container 39d0d80623631c70ab8619dab493bf7f6ba282ff8fc132b4c01f4dbc032ebdc5: Status 404 returned error can't find the container with id 39d0d80623631c70ab8619dab493bf7f6ba282ff8fc132b4c01f4dbc032ebdc5 Oct 30 14:24:27.510055 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:27.510024 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea9ed2-nrct6"] Oct 30 14:24:27.597048 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:27.597006 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea9ed2-nrct6" event={"ID":"e145e1fb-ad70-46eb-bba7-adbf5cab95b8","Type":"ContainerStarted","Data":"39d0d80623631c70ab8619dab493bf7f6ba282ff8fc132b4c01f4dbc032ebdc5"} Oct 30 14:24:28.014969 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:24:28.014934 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:24:28.342447 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:28.342407 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:24:28.605802 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:28.605715 4982 generic.go:356] "Generic (PLEG): container finished" podID="e145e1fb-ad70-46eb-bba7-adbf5cab95b8" containerID="5f89b70cf548c575dd647c7763ab5fcc62977089edeef38dfe057dc7acf230bc" exitCode=0 Oct 30 14:24:28.606211 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:28.605801 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea9ed2-nrct6" event={"ID":"e145e1fb-ad70-46eb-bba7-adbf5cab95b8","Type":"ContainerDied","Data":"5f89b70cf548c575dd647c7763ab5fcc62977089edeef38dfe057dc7acf230bc"} Oct 30 14:24:29.785348 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:29.785321 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea9ed2-nrct6" Oct 30 14:24:29.946083 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:29.946004 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e145e1fb-ad70-46eb-bba7-adbf5cab95b8-kube-api-access-6qgww\" (UniqueName: \"kubernetes.io/projected/e145e1fb-ad70-46eb-bba7-adbf5cab95b8-kube-api-access-6qgww\") pod \"e145e1fb-ad70-46eb-bba7-adbf5cab95b8\" (UID: \"e145e1fb-ad70-46eb-bba7-adbf5cab95b8\") " Oct 30 14:24:29.946083 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:29.946067 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e145e1fb-ad70-46eb-bba7-adbf5cab95b8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e145e1fb-ad70-46eb-bba7-adbf5cab95b8-maas-api-openshift-ca\") pod \"e145e1fb-ad70-46eb-bba7-adbf5cab95b8\" (UID: \"e145e1fb-ad70-46eb-bba7-adbf5cab95b8\") " Oct 30 14:24:29.946263 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:29.946111 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e145e1fb-ad70-46eb-bba7-adbf5cab95b8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e145e1fb-ad70-46eb-bba7-adbf5cab95b8-maas-api-xks-ca\") pod \"e145e1fb-ad70-46eb-bba7-adbf5cab95b8\" (UID: \"e145e1fb-ad70-46eb-bba7-adbf5cab95b8\") " Oct 30 14:24:29.946490 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:29.946459 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e145e1fb-ad70-46eb-bba7-adbf5cab95b8-maas-api-openshift-ca" pod "e145e1fb-ad70-46eb-bba7-adbf5cab95b8" (UID: "e145e1fb-ad70-46eb-bba7-adbf5cab95b8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:24:29.946755 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:29.946735 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e145e1fb-ad70-46eb-bba7-adbf5cab95b8-maas-api-openshift-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:24:29.948156 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:29.948135 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e145e1fb-ad70-46eb-bba7-adbf5cab95b8-kube-api-access-6qgww" pod "e145e1fb-ad70-46eb-bba7-adbf5cab95b8" (UID: "e145e1fb-ad70-46eb-bba7-adbf5cab95b8"). InnerVolumeSpecName "kube-api-access-6qgww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:24:29.948243 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:29.948159 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e145e1fb-ad70-46eb-bba7-adbf5cab95b8-maas-api-xks-ca" pod "e145e1fb-ad70-46eb-bba7-adbf5cab95b8" (UID: "e145e1fb-ad70-46eb-bba7-adbf5cab95b8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:24:30.047917 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:30.047884 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qgww\" (UniqueName: \"kubernetes.io/projected/e145e1fb-ad70-46eb-bba7-adbf5cab95b8-kube-api-access-6qgww\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:24:30.047917 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:30.047915 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e145e1fb-ad70-46eb-bba7-adbf5cab95b8-maas-api-xks-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:24:30.531982 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:30.531947 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:24:30.623615 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:30.623585 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea9ed2-nrct6" Oct 30 14:24:30.623826 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:30.623625 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea9ed2-nrct6" event={"ID":"e145e1fb-ad70-46eb-bba7-adbf5cab95b8","Type":"ContainerDied","Data":"39d0d80623631c70ab8619dab493bf7f6ba282ff8fc132b4c01f4dbc032ebdc5"} Oct 30 14:24:30.623826 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:30.623700 4982 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39d0d80623631c70ab8619dab493bf7f6ba282ff8fc132b4c01f4dbc032ebdc5" Oct 30 14:24:32.560185 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:32.560148 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:24:32.710238 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:32.710171 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:24:34.746633 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:34.746596 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:24:36.675848 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:36.675817 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:24:36.827384 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:36.827342 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:24:37.063315 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:37.063282 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:24:37.633717 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:37.633672 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ea9ed2-78fc9fb9fb-ftnvk"] Oct 30 14:24:37.703675 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:37.700705 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ea9ed2-849b7b59dc-vp88x"] Oct 30 14:24:37.879751 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:37.879717 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:24:38.029785 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:38.029631 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:24:39.057116 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:39.057082 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:24:41.459370 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:41.459319 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:24:41.742625 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:41.742536 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:24:41.914363 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:41.914325 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:24:42.079537 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:42.079504 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:24:42.634366 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:42.634328 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-ea9ed2-78fc9fb9fb-ftnvk" podUID="a754700e-df12-456f-af9e-2b5f408ad14e" containerName="payload-processing" containerID="cri-o://bd5444b7bf193899d9e657d596afe6d7f20d5e6f39deb148455132df5a6c2685" gracePeriod=25 Oct 30 14:24:42.701998 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:42.701960 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ea9ed2-849b7b59dc-vp88x" podUID="56fc4cc3-42d8-4e38-bd33-a7ab94c7dea8" containerName="payload-pre-processing" containerID="cri-o://d7d09adf67e7fae45155b06f6403ef983d73f87f97336255bc67ba5785ea0606" gracePeriod=25 Oct 30 14:24:42.778106 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:42.778064 4982 generic.go:356] "Generic (PLEG): container finished" podID="a754700e-df12-456f-af9e-2b5f408ad14e" containerID="bd5444b7bf193899d9e657d596afe6d7f20d5e6f39deb148455132df5a6c2685" exitCode=0 Oct 30 14:24:42.778288 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:42.778133 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ea9ed2-78fc9fb9fb-ftnvk" event={"ID":"a754700e-df12-456f-af9e-2b5f408ad14e","Type":"ContainerDied","Data":"bd5444b7bf193899d9e657d596afe6d7f20d5e6f39deb148455132df5a6c2685"} Oct 30 14:24:43.035295 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:43.035259 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ea9ed2-78fc9fb9fb-ftnvk" Oct 30 14:24:43.065218 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:43.065189 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ea9ed2-849b7b59dc-vp88x" Oct 30 14:24:43.149616 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:43.149535 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56fc4cc3-42d8-4e38-bd33-a7ab94c7dea8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56fc4cc3-42d8-4e38-bd33-a7ab94c7dea8-plugins-config-volume\") pod \"56fc4cc3-42d8-4e38-bd33-a7ab94c7dea8\" (UID: \"56fc4cc3-42d8-4e38-bd33-a7ab94c7dea8\") " Oct 30 14:24:43.149822 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:43.149673 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56fc4cc3-42d8-4e38-bd33-a7ab94c7dea8-kube-api-access-4zkwd\" (UniqueName: \"kubernetes.io/projected/56fc4cc3-42d8-4e38-bd33-a7ab94c7dea8-kube-api-access-4zkwd\") pod \"56fc4cc3-42d8-4e38-bd33-a7ab94c7dea8\" (UID: \"56fc4cc3-42d8-4e38-bd33-a7ab94c7dea8\") " Oct 30 14:24:43.149822 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:43.149753 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a754700e-df12-456f-af9e-2b5f408ad14e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a754700e-df12-456f-af9e-2b5f408ad14e-plugins-config-volume\") pod \"a754700e-df12-456f-af9e-2b5f408ad14e\" (UID: \"a754700e-df12-456f-af9e-2b5f408ad14e\") " Oct 30 14:24:43.149948 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:43.149847 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a754700e-df12-456f-af9e-2b5f408ad14e-kube-api-access-dtkqr\" (UniqueName: \"kubernetes.io/projected/a754700e-df12-456f-af9e-2b5f408ad14e-kube-api-access-dtkqr\") pod \"a754700e-df12-456f-af9e-2b5f408ad14e\" (UID: \"a754700e-df12-456f-af9e-2b5f408ad14e\") " Oct 30 14:24:43.150465 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:43.150136 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56fc4cc3-42d8-4e38-bd33-a7ab94c7dea8-plugins-config-volume" pod "56fc4cc3-42d8-4e38-bd33-a7ab94c7dea8" (UID: "56fc4cc3-42d8-4e38-bd33-a7ab94c7dea8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:24:43.150465 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:43.150164 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a754700e-df12-456f-af9e-2b5f408ad14e-plugins-config-volume" pod "a754700e-df12-456f-af9e-2b5f408ad14e" (UID: "a754700e-df12-456f-af9e-2b5f408ad14e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:24:43.150673 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:43.150481 4982 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56fc4cc3-42d8-4e38-bd33-a7ab94c7dea8-plugins-config-volume\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:24:43.150673 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:43.150504 4982 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a754700e-df12-456f-af9e-2b5f408ad14e-plugins-config-volume\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:24:43.155537 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:43.152588 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a754700e-df12-456f-af9e-2b5f408ad14e-kube-api-access-dtkqr" pod "a754700e-df12-456f-af9e-2b5f408ad14e" (UID: "a754700e-df12-456f-af9e-2b5f408ad14e"). InnerVolumeSpecName "kube-api-access-dtkqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:24:43.155537 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:43.152704 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56fc4cc3-42d8-4e38-bd33-a7ab94c7dea8-kube-api-access-4zkwd" pod "56fc4cc3-42d8-4e38-bd33-a7ab94c7dea8" (UID: "56fc4cc3-42d8-4e38-bd33-a7ab94c7dea8"). InnerVolumeSpecName "kube-api-access-4zkwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:24:43.251740 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:43.251706 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zkwd\" (UniqueName: \"kubernetes.io/projected/56fc4cc3-42d8-4e38-bd33-a7ab94c7dea8-kube-api-access-4zkwd\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:24:43.251740 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:43.251737 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtkqr\" (UniqueName: \"kubernetes.io/projected/a754700e-df12-456f-af9e-2b5f408ad14e-kube-api-access-dtkqr\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:24:43.569323 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:43.569287 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:24:43.788604 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:43.788568 4982 generic.go:356] "Generic (PLEG): container finished" podID="56fc4cc3-42d8-4e38-bd33-a7ab94c7dea8" containerID="d7d09adf67e7fae45155b06f6403ef983d73f87f97336255bc67ba5785ea0606" exitCode=0 Oct 30 14:24:43.789095 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:43.788624 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ea9ed2-849b7b59dc-vp88x" Oct 30 14:24:43.789095 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:43.788666 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ea9ed2-849b7b59dc-vp88x" event={"ID":"56fc4cc3-42d8-4e38-bd33-a7ab94c7dea8","Type":"ContainerDied","Data":"d7d09adf67e7fae45155b06f6403ef983d73f87f97336255bc67ba5785ea0606"} Oct 30 14:24:43.789095 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:43.788727 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ea9ed2-849b7b59dc-vp88x" event={"ID":"56fc4cc3-42d8-4e38-bd33-a7ab94c7dea8","Type":"ContainerDied","Data":"83d68674cc0cfb94085b3bd1cffdcd93ad97e378292b5e61ab44b5e38f177139"} Oct 30 14:24:43.789095 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:43.788758 4982 scope.go:122] "RemoveContainer" containerID="d7d09adf67e7fae45155b06f6403ef983d73f87f97336255bc67ba5785ea0606" Oct 30 14:24:43.790281 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:43.790263 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ea9ed2-78fc9fb9fb-ftnvk" Oct 30 14:24:43.790380 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:43.790309 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ea9ed2-78fc9fb9fb-ftnvk" event={"ID":"a754700e-df12-456f-af9e-2b5f408ad14e","Type":"ContainerDied","Data":"53b7f965eb49c9730aa47a86442e0fe11ac12619124165e7da23702765fc6a13"} Oct 30 14:24:43.800688 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:43.800668 4982 scope.go:122] "RemoveContainer" containerID="d7d09adf67e7fae45155b06f6403ef983d73f87f97336255bc67ba5785ea0606" Oct 30 14:24:43.800998 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:24:43.800979 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7d09adf67e7fae45155b06f6403ef983d73f87f97336255bc67ba5785ea0606\": container with ID starting with d7d09adf67e7fae45155b06f6403ef983d73f87f97336255bc67ba5785ea0606 not found: ID does not exist" containerID="d7d09adf67e7fae45155b06f6403ef983d73f87f97336255bc67ba5785ea0606" Oct 30 14:24:43.801083 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:43.801006 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7d09adf67e7fae45155b06f6403ef983d73f87f97336255bc67ba5785ea0606"} err="failed to get container status \"d7d09adf67e7fae45155b06f6403ef983d73f87f97336255bc67ba5785ea0606\": rpc error: code = NotFound desc = could not find container \"d7d09adf67e7fae45155b06f6403ef983d73f87f97336255bc67ba5785ea0606\": container with ID starting with d7d09adf67e7fae45155b06f6403ef983d73f87f97336255bc67ba5785ea0606 not found: ID does not exist" Oct 30 14:24:43.801083 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:43.801022 4982 scope.go:122] "RemoveContainer" containerID="bd5444b7bf193899d9e657d596afe6d7f20d5e6f39deb148455132df5a6c2685" Oct 30 14:24:43.849838 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:43.849779 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ea9ed2-849b7b59dc-vp88x"] Oct 30 14:24:43.862786 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:43.862763 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ea9ed2-849b7b59dc-vp88x"] Oct 30 14:24:43.892556 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:43.892528 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ea9ed2-78fc9fb9fb-ftnvk"] Oct 30 14:24:43.898082 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:43.898056 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ea9ed2-78fc9fb9fb-ftnvk"] Oct 30 14:24:44.403258 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:44.403216 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56fc4cc3-42d8-4e38-bd33-a7ab94c7dea8" path="/var/lib/kubelet/pods/56fc4cc3-42d8-4e38-bd33-a7ab94c7dea8/volumes" Oct 30 14:24:44.403860 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:44.403835 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a754700e-df12-456f-af9e-2b5f408ad14e" path="/var/lib/kubelet/pods/a754700e-df12-456f-af9e-2b5f408ad14e/volumes" Oct 30 14:24:44.563585 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:44.563548 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:24:45.369174 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:45.368283 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:24:45.707388 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:45.707308 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:24:46.263550 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:46.263514 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:24:46.739328 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:46.739287 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:24:47.277809 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:47.277516 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:24:47.453594 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:47.453548 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:24:49.660838 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:49.660287 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:24:50.897542 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:50.897506 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:24:50.963302 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:50.963258 4982 scope.go:122] "RemoveContainer" containerID="28c94a2f3d776edd7687532a918b289a2bb2781ef0347f66683229b2745aa27b" Oct 30 14:24:51.529714 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:51.529671 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:24:53.447012 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:53.446974 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:24:55.240316 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:55.240274 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:24:57.103669 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:57.103607 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:24:58.023070 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:24:58.023027 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:24:59.211751 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:24:59.211708 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:25:00.669620 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:00.669589 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:25:00.818736 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:00.818703 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:25:01.242096 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:01.242057 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:25:02.678318 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:02.678270 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:25:02.829599 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:02.829562 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-93549e-56ff978bf6-qh2c5"] Oct 30 14:25:02.830195 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:02.830180 4982 cpu_manager.go:414] "containerMap: removing container" podUID="56fc4cc3-42d8-4e38-bd33-a7ab94c7dea8" containerName="payload-pre-processing" Oct 30 14:25:02.830240 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:02.830196 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56fc4cc3-42d8-4e38-bd33-a7ab94c7dea8" containerName="payload-pre-processing" Oct 30 14:25:02.830240 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:02.830212 4982 cpu_manager.go:414] "containerMap: removing container" podUID="a754700e-df12-456f-af9e-2b5f408ad14e" containerName="payload-processing" Oct 30 14:25:02.830240 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:02.830218 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a754700e-df12-456f-af9e-2b5f408ad14e" containerName="payload-processing" Oct 30 14:25:02.830341 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:02.830253 4982 cpu_manager.go:414] "containerMap: removing container" podUID="e145e1fb-ad70-46eb-bba7-adbf5cab95b8" containerName="revoke-keys" Oct 30 14:25:02.830341 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:02.830258 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e145e1fb-ad70-46eb-bba7-adbf5cab95b8" containerName="revoke-keys" Oct 30 14:25:02.830398 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:02.830375 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="e145e1fb-ad70-46eb-bba7-adbf5cab95b8" containerName="revoke-keys" Oct 30 14:25:02.830398 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:02.830388 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="56fc4cc3-42d8-4e38-bd33-a7ab94c7dea8" containerName="payload-pre-processing" Oct 30 14:25:02.830398 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:02.830395 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="a754700e-df12-456f-af9e-2b5f408ad14e" containerName="payload-processing" Oct 30 14:25:02.835478 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:02.835458 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-93549e-56ff978bf6-qh2c5" Oct 30 14:25:02.857233 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:02.857203 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-93549e-56ff978bf6-qh2c5"] Oct 30 14:25:02.866386 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:02.866361 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-93549e\"" Oct 30 14:25:02.866524 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:02.866455 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-93549e-dockercfg-l4kgf\"" Oct 30 14:25:02.878033 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:02.877962 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:25:02.894037 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:02.893689 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wb7p\" (UniqueName: \"kubernetes.io/projected/8d73bb58-560d-4c40-aa66-1ad183d82e20-kube-api-access-6wb7p\") pod \"payload-processing-e2e-inf-b-93549e-56ff978bf6-qh2c5\" (UID: \"8d73bb58-560d-4c40-aa66-1ad183d82e20\") " pod="openshift-ingress/payload-processing-e2e-inf-b-93549e-56ff978bf6-qh2c5" Oct 30 14:25:02.913662 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:02.894576 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d73bb58-560d-4c40-aa66-1ad183d82e20-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-93549e-56ff978bf6-qh2c5\" (UID: \"8d73bb58-560d-4c40-aa66-1ad183d82e20\") " pod="openshift-ingress/payload-processing-e2e-inf-b-93549e-56ff978bf6-qh2c5" Oct 30 14:25:03.027726 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:03.015675 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d73bb58-560d-4c40-aa66-1ad183d82e20-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-93549e-56ff978bf6-qh2c5\" (UID: \"8d73bb58-560d-4c40-aa66-1ad183d82e20\") " pod="openshift-ingress/payload-processing-e2e-inf-b-93549e-56ff978bf6-qh2c5" Oct 30 14:25:03.051665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:03.028068 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wb7p\" (UniqueName: \"kubernetes.io/projected/8d73bb58-560d-4c40-aa66-1ad183d82e20-kube-api-access-6wb7p\") pod \"payload-processing-e2e-inf-b-93549e-56ff978bf6-qh2c5\" (UID: \"8d73bb58-560d-4c40-aa66-1ad183d82e20\") " pod="openshift-ingress/payload-processing-e2e-inf-b-93549e-56ff978bf6-qh2c5" Oct 30 14:25:03.052004 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:03.037184 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d73bb58-560d-4c40-aa66-1ad183d82e20-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-93549e-56ff978bf6-qh2c5\" (UID: \"8d73bb58-560d-4c40-aa66-1ad183d82e20\") " pod="openshift-ingress/payload-processing-e2e-inf-b-93549e-56ff978bf6-qh2c5" Oct 30 14:25:03.105277 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:03.105248 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wb7p\" (UniqueName: \"kubernetes.io/projected/8d73bb58-560d-4c40-aa66-1ad183d82e20-kube-api-access-6wb7p\") pod \"payload-processing-e2e-inf-b-93549e-56ff978bf6-qh2c5\" (UID: \"8d73bb58-560d-4c40-aa66-1ad183d82e20\") " pod="openshift-ingress/payload-processing-e2e-inf-b-93549e-56ff978bf6-qh2c5" Oct 30 14:25:03.149282 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:03.149248 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-93549e-56ff978bf6-qh2c5" Oct 30 14:25:03.491566 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:03.491529 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-93549e-56ff978bf6-qh2c5"] Oct 30 14:25:03.492065 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:25:03.492041 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d73bb58_560d_4c40_aa66_1ad183d82e20.slice/crio-2787b1ab5c3f605612f100f588355664682b499b1580ae737b1468c6cca91c59 WatchSource:0}: Error finding container 2787b1ab5c3f605612f100f588355664682b499b1580ae737b1468c6cca91c59: Status 404 returned error can't find the container with id 2787b1ab5c3f605612f100f588355664682b499b1580ae737b1468c6cca91c59 Oct 30 14:25:03.497264 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:03.497222 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:25:03.675712 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:03.670834 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:25:04.070290 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:04.070254 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-93549e-56ff978bf6-qh2c5" event={"ID":"8d73bb58-560d-4c40-aa66-1ad183d82e20","Type":"ContainerStarted","Data":"c512940230e6f18ca851a3cde9666937dc400546657ad985cbdba37e1eb42399"} Oct 30 14:25:04.070804 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:04.070307 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-93549e-56ff978bf6-qh2c5" event={"ID":"8d73bb58-560d-4c40-aa66-1ad183d82e20","Type":"ContainerStarted","Data":"2787b1ab5c3f605612f100f588355664682b499b1580ae737b1468c6cca91c59"} Oct 30 14:25:04.070804 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:04.070331 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-93549e-56ff978bf6-qh2c5" Oct 30 14:25:04.118597 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:04.118542 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-93549e-56ff978bf6-qh2c5" podStartSLOduration=2.118523929 podStartE2EDuration="2.118523929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:25:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:25:04.115991819 +0000 UTC m=+2173.860125859" watchObservedRunningTime="2026-10-30 14:25:04.118523929 +0000 UTC m=+2173.862658022" Oct 30 14:25:05.671397 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:05.671335 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:25:05.868669 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:05.868473 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:25:08.047532 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:08.047496 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:25:09.930513 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:09.930406 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:25:11.362132 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:11.362092 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:25:11.935981 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:11.935939 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:25:12.795777 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:12.795741 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld"] Oct 30 14:25:12.796290 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:12.795980 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld" podUID="5716f397-1e25-4fd2-b1e3-2b289d1f67c9" containerName="istio-proxy" containerID="cri-o://cc7962755abca63077edcc967775e41df63354e8f23e03df114fc9fd38ce88e7" gracePeriod=30 Oct 30 14:25:13.964996 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:13.964954 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:25:15.088294 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:15.088258 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-93549e-56ff978bf6-qh2c5" Oct 30 14:25:16.099115 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:16.099076 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:25:17.951147 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:17.948280 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:25:18.114863 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:18.114835 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld" Oct 30 14:25:18.123159 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:18.123122 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5716f397-1e25-4fd2-b1e3-2b289d1f67c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5716f397-1e25-4fd2-b1e3-2b289d1f67c9-istio-envoy\") pod \"5716f397-1e25-4fd2-b1e3-2b289d1f67c9\" (UID: \"5716f397-1e25-4fd2-b1e3-2b289d1f67c9\") " Oct 30 14:25:18.123325 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:18.123178 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5716f397-1e25-4fd2-b1e3-2b289d1f67c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5716f397-1e25-4fd2-b1e3-2b289d1f67c9-credential-socket\") pod \"5716f397-1e25-4fd2-b1e3-2b289d1f67c9\" (UID: \"5716f397-1e25-4fd2-b1e3-2b289d1f67c9\") " Oct 30 14:25:18.123325 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:18.123205 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5716f397-1e25-4fd2-b1e3-2b289d1f67c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5716f397-1e25-4fd2-b1e3-2b289d1f67c9-istiod-ca-cert\") pod \"5716f397-1e25-4fd2-b1e3-2b289d1f67c9\" (UID: \"5716f397-1e25-4fd2-b1e3-2b289d1f67c9\") " Oct 30 14:25:18.123325 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:18.123305 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5716f397-1e25-4fd2-b1e3-2b289d1f67c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5716f397-1e25-4fd2-b1e3-2b289d1f67c9-istio-podinfo\") pod \"5716f397-1e25-4fd2-b1e3-2b289d1f67c9\" (UID: \"5716f397-1e25-4fd2-b1e3-2b289d1f67c9\") " Oct 30 14:25:18.123507 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:18.123346 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5716f397-1e25-4fd2-b1e3-2b289d1f67c9-kube-api-access-vkbw4\" (UniqueName: \"kubernetes.io/projected/5716f397-1e25-4fd2-b1e3-2b289d1f67c9-kube-api-access-vkbw4\") pod \"5716f397-1e25-4fd2-b1e3-2b289d1f67c9\" (UID: \"5716f397-1e25-4fd2-b1e3-2b289d1f67c9\") " Oct 30 14:25:18.123507 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:18.123387 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5716f397-1e25-4fd2-b1e3-2b289d1f67c9-istio-token\" (UniqueName: \"kubernetes.io/projected/5716f397-1e25-4fd2-b1e3-2b289d1f67c9-istio-token\") pod \"5716f397-1e25-4fd2-b1e3-2b289d1f67c9\" (UID: \"5716f397-1e25-4fd2-b1e3-2b289d1f67c9\") " Oct 30 14:25:18.123507 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:18.123465 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5716f397-1e25-4fd2-b1e3-2b289d1f67c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5716f397-1e25-4fd2-b1e3-2b289d1f67c9-workload-socket\") pod \"5716f397-1e25-4fd2-b1e3-2b289d1f67c9\" (UID: \"5716f397-1e25-4fd2-b1e3-2b289d1f67c9\") " Oct 30 14:25:18.123507 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:18.123498 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5716f397-1e25-4fd2-b1e3-2b289d1f67c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5716f397-1e25-4fd2-b1e3-2b289d1f67c9-istio-data\") pod \"5716f397-1e25-4fd2-b1e3-2b289d1f67c9\" (UID: \"5716f397-1e25-4fd2-b1e3-2b289d1f67c9\") " Oct 30 14:25:18.123735 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:18.123712 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5716f397-1e25-4fd2-b1e3-2b289d1f67c9-credential-socket" pod "5716f397-1e25-4fd2-b1e3-2b289d1f67c9" (UID: "5716f397-1e25-4fd2-b1e3-2b289d1f67c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:25:18.123792 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:18.123725 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5716f397-1e25-4fd2-b1e3-2b289d1f67c9-istio-data" pod "5716f397-1e25-4fd2-b1e3-2b289d1f67c9" (UID: "5716f397-1e25-4fd2-b1e3-2b289d1f67c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:25:18.123792 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:18.123738 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5716f397-1e25-4fd2-b1e3-2b289d1f67c9-workload-socket" pod "5716f397-1e25-4fd2-b1e3-2b289d1f67c9" (UID: "5716f397-1e25-4fd2-b1e3-2b289d1f67c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:25:18.123792 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:18.123756 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5716f397-1e25-4fd2-b1e3-2b289d1f67c9-istiod-ca-cert" pod "5716f397-1e25-4fd2-b1e3-2b289d1f67c9" (UID: "5716f397-1e25-4fd2-b1e3-2b289d1f67c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:25:18.123944 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:18.123851 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5716f397-1e25-4fd2-b1e3-2b289d1f67c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5716f397-1e25-4fd2-b1e3-2b289d1f67c9-workload-certs\") pod \"5716f397-1e25-4fd2-b1e3-2b289d1f67c9\" (UID: \"5716f397-1e25-4fd2-b1e3-2b289d1f67c9\") " Oct 30 14:25:18.124128 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:18.124107 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5716f397-1e25-4fd2-b1e3-2b289d1f67c9-workload-certs" pod "5716f397-1e25-4fd2-b1e3-2b289d1f67c9" (UID: "5716f397-1e25-4fd2-b1e3-2b289d1f67c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:25:18.124424 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:18.124405 4982 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5716f397-1e25-4fd2-b1e3-2b289d1f67c9-credential-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:25:18.124512 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:18.124424 4982 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5716f397-1e25-4fd2-b1e3-2b289d1f67c9-istiod-ca-cert\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:25:18.124512 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:18.124434 4982 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5716f397-1e25-4fd2-b1e3-2b289d1f67c9-workload-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:25:18.124512 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:18.124442 4982 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5716f397-1e25-4fd2-b1e3-2b289d1f67c9-istio-data\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:25:18.124512 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:18.124451 4982 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5716f397-1e25-4fd2-b1e3-2b289d1f67c9-workload-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:25:18.125562 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:18.125538 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5716f397-1e25-4fd2-b1e3-2b289d1f67c9-istio-envoy" pod "5716f397-1e25-4fd2-b1e3-2b289d1f67c9" (UID: "5716f397-1e25-4fd2-b1e3-2b289d1f67c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:25:18.125673 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:18.125618 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5716f397-1e25-4fd2-b1e3-2b289d1f67c9-istio-podinfo" pod "5716f397-1e25-4fd2-b1e3-2b289d1f67c9" (UID: "5716f397-1e25-4fd2-b1e3-2b289d1f67c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:25:18.126053 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:18.126023 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5716f397-1e25-4fd2-b1e3-2b289d1f67c9-kube-api-access-vkbw4" pod "5716f397-1e25-4fd2-b1e3-2b289d1f67c9" (UID: "5716f397-1e25-4fd2-b1e3-2b289d1f67c9"). InnerVolumeSpecName "kube-api-access-vkbw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:25:18.126146 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:18.126048 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5716f397-1e25-4fd2-b1e3-2b289d1f67c9-istio-token" pod "5716f397-1e25-4fd2-b1e3-2b289d1f67c9" (UID: "5716f397-1e25-4fd2-b1e3-2b289d1f67c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:25:18.187437 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:18.186997 4982 generic.go:356] "Generic (PLEG): container finished" podID="5716f397-1e25-4fd2-b1e3-2b289d1f67c9" containerID="cc7962755abca63077edcc967775e41df63354e8f23e03df114fc9fd38ce88e7" exitCode=0 Oct 30 14:25:18.187437 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:18.187053 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld" Oct 30 14:25:18.187437 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:18.187203 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld" event={"ID":"5716f397-1e25-4fd2-b1e3-2b289d1f67c9","Type":"ContainerDied","Data":"cc7962755abca63077edcc967775e41df63354e8f23e03df114fc9fd38ce88e7"} Oct 30 14:25:18.187437 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:18.187271 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld" event={"ID":"5716f397-1e25-4fd2-b1e3-2b289d1f67c9","Type":"ContainerDied","Data":"336c8f46f8dd36bad6249d725611b8be8840fa645d2f9d882f79826d7d04a484"} Oct 30 14:25:18.187437 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:18.187300 4982 scope.go:122] "RemoveContainer" containerID="cc7962755abca63077edcc967775e41df63354e8f23e03df114fc9fd38ce88e7" Oct 30 14:25:18.201187 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:18.201163 4982 scope.go:122] "RemoveContainer" containerID="cc7962755abca63077edcc967775e41df63354e8f23e03df114fc9fd38ce88e7" Oct 30 14:25:18.201538 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:25:18.201512 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc7962755abca63077edcc967775e41df63354e8f23e03df114fc9fd38ce88e7\": container with ID starting with cc7962755abca63077edcc967775e41df63354e8f23e03df114fc9fd38ce88e7 not found: ID does not exist" containerID="cc7962755abca63077edcc967775e41df63354e8f23e03df114fc9fd38ce88e7" Oct 30 14:25:18.201661 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:18.201548 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc7962755abca63077edcc967775e41df63354e8f23e03df114fc9fd38ce88e7"} err="failed to get container status \"cc7962755abca63077edcc967775e41df63354e8f23e03df114fc9fd38ce88e7\": rpc error: code = NotFound desc = could not find container \"cc7962755abca63077edcc967775e41df63354e8f23e03df114fc9fd38ce88e7\": container with ID starting with cc7962755abca63077edcc967775e41df63354e8f23e03df114fc9fd38ce88e7 not found: ID does not exist" Oct 30 14:25:18.226034 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:18.226008 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkbw4\" (UniqueName: \"kubernetes.io/projected/5716f397-1e25-4fd2-b1e3-2b289d1f67c9-kube-api-access-vkbw4\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:25:18.226034 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:18.226034 4982 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5716f397-1e25-4fd2-b1e3-2b289d1f67c9-istio-token\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:25:18.226217 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:18.226044 4982 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5716f397-1e25-4fd2-b1e3-2b289d1f67c9-istio-envoy\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:25:18.226217 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:18.226052 4982 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5716f397-1e25-4fd2-b1e3-2b289d1f67c9-istio-podinfo\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:25:18.257884 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:18.257855 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld"] Oct 30 14:25:18.265490 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:18.265466 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-hswld"] Oct 30 14:25:18.402933 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:18.402860 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5716f397-1e25-4fd2-b1e3-2b289d1f67c9" path="/var/lib/kubelet/pods/5716f397-1e25-4fd2-b1e3-2b289d1f67c9/volumes" Oct 30 14:25:19.975052 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:19.974998 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:25:20.553019 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:20.552971 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:25:21.031020 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:21.030985 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:25:21.483998 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:21.483951 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh"] Oct 30 14:25:21.485494 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:21.485464 4982 cpu_manager.go:414] "containerMap: removing container" podUID="5716f397-1e25-4fd2-b1e3-2b289d1f67c9" containerName="istio-proxy" Oct 30 14:25:21.485494 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:21.485493 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5716f397-1e25-4fd2-b1e3-2b289d1f67c9" containerName="istio-proxy" Oct 30 14:25:21.485800 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:21.485779 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="5716f397-1e25-4fd2-b1e3-2b289d1f67c9" containerName="istio-proxy" Oct 30 14:25:21.491324 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:21.491297 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh" Oct 30 14:25:21.514605 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:21.514575 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh"] Oct 30 14:25:21.568762 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:21.568724 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:25:21.593714 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:21.579688 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-0b1c15-openshift-default-dockercfg-bmcps\"" Oct 30 14:25:21.593714 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:21.589072 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d57125b3-c6b9-4d3b-b04c-7f6b30094325-istio-data\") pod \"e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh\" (UID: \"d57125b3-c6b9-4d3b-b04c-7f6b30094325\") " pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh" Oct 30 14:25:21.593714 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:21.589177 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d57125b3-c6b9-4d3b-b04c-7f6b30094325-istio-podinfo\") pod \"e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh\" (UID: \"d57125b3-c6b9-4d3b-b04c-7f6b30094325\") " pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh" Oct 30 14:25:21.593714 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:21.589254 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d57125b3-c6b9-4d3b-b04c-7f6b30094325-istiod-ca-cert\") pod \"e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh\" (UID: \"d57125b3-c6b9-4d3b-b04c-7f6b30094325\") " pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh" Oct 30 14:25:21.593714 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:21.589295 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mlrq\" (UniqueName: \"kubernetes.io/projected/d57125b3-c6b9-4d3b-b04c-7f6b30094325-kube-api-access-6mlrq\") pod \"e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh\" (UID: \"d57125b3-c6b9-4d3b-b04c-7f6b30094325\") " pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh" Oct 30 14:25:21.593714 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:21.589342 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d57125b3-c6b9-4d3b-b04c-7f6b30094325-istio-token\") pod \"e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh\" (UID: \"d57125b3-c6b9-4d3b-b04c-7f6b30094325\") " pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh" Oct 30 14:25:21.593714 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:21.589406 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d57125b3-c6b9-4d3b-b04c-7f6b30094325-workload-certs\") pod \"e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh\" (UID: \"d57125b3-c6b9-4d3b-b04c-7f6b30094325\") " pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh" Oct 30 14:25:21.593714 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:21.589503 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d57125b3-c6b9-4d3b-b04c-7f6b30094325-credential-socket\") pod \"e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh\" (UID: \"d57125b3-c6b9-4d3b-b04c-7f6b30094325\") " pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh" Oct 30 14:25:21.593714 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:21.589579 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d57125b3-c6b9-4d3b-b04c-7f6b30094325-workload-socket\") pod \"e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh\" (UID: \"d57125b3-c6b9-4d3b-b04c-7f6b30094325\") " pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh" Oct 30 14:25:21.593714 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:21.589681 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d57125b3-c6b9-4d3b-b04c-7f6b30094325-istio-envoy\") pod \"e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh\" (UID: \"d57125b3-c6b9-4d3b-b04c-7f6b30094325\") " pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh" Oct 30 14:25:21.697578 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:21.697532 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d57125b3-c6b9-4d3b-b04c-7f6b30094325-istio-envoy\") pod \"e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh\" (UID: \"d57125b3-c6b9-4d3b-b04c-7f6b30094325\") " pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh" Oct 30 14:25:21.697964 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:21.697932 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d57125b3-c6b9-4d3b-b04c-7f6b30094325-istio-data\") pod \"e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh\" (UID: \"d57125b3-c6b9-4d3b-b04c-7f6b30094325\") " pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh" Oct 30 14:25:21.698170 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:21.698150 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d57125b3-c6b9-4d3b-b04c-7f6b30094325-istio-podinfo\") pod \"e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh\" (UID: \"d57125b3-c6b9-4d3b-b04c-7f6b30094325\") " pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh" Oct 30 14:25:21.698337 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:21.698318 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d57125b3-c6b9-4d3b-b04c-7f6b30094325-istiod-ca-cert\") pod \"e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh\" (UID: \"d57125b3-c6b9-4d3b-b04c-7f6b30094325\") " pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh" Oct 30 14:25:21.698471 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:21.698454 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mlrq\" (UniqueName: \"kubernetes.io/projected/d57125b3-c6b9-4d3b-b04c-7f6b30094325-kube-api-access-6mlrq\") pod \"e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh\" (UID: \"d57125b3-c6b9-4d3b-b04c-7f6b30094325\") " pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh" Oct 30 14:25:21.698598 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:21.698583 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d57125b3-c6b9-4d3b-b04c-7f6b30094325-istio-token\") pod \"e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh\" (UID: \"d57125b3-c6b9-4d3b-b04c-7f6b30094325\") " pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh" Oct 30 14:25:21.698764 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:21.698747 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d57125b3-c6b9-4d3b-b04c-7f6b30094325-workload-certs\") pod \"e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh\" (UID: \"d57125b3-c6b9-4d3b-b04c-7f6b30094325\") " pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh" Oct 30 14:25:21.698942 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:21.698926 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d57125b3-c6b9-4d3b-b04c-7f6b30094325-credential-socket\") pod \"e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh\" (UID: \"d57125b3-c6b9-4d3b-b04c-7f6b30094325\") " pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh" Oct 30 14:25:21.699081 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:21.699068 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d57125b3-c6b9-4d3b-b04c-7f6b30094325-workload-socket\") pod \"e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh\" (UID: \"d57125b3-c6b9-4d3b-b04c-7f6b30094325\") " pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh" Oct 30 14:25:21.707680 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:21.702749 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d57125b3-c6b9-4d3b-b04c-7f6b30094325-workload-socket\") pod \"e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh\" (UID: \"d57125b3-c6b9-4d3b-b04c-7f6b30094325\") " pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh" Oct 30 14:25:21.734822 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:21.704471 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d57125b3-c6b9-4d3b-b04c-7f6b30094325-istiod-ca-cert\") pod \"e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh\" (UID: \"d57125b3-c6b9-4d3b-b04c-7f6b30094325\") " pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh" Oct 30 14:25:21.735042 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:21.708621 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d57125b3-c6b9-4d3b-b04c-7f6b30094325-workload-certs\") pod \"e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh\" (UID: \"d57125b3-c6b9-4d3b-b04c-7f6b30094325\") " pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh" Oct 30 14:25:21.735183 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:21.708893 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d57125b3-c6b9-4d3b-b04c-7f6b30094325-credential-socket\") pod \"e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh\" (UID: \"d57125b3-c6b9-4d3b-b04c-7f6b30094325\") " pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh" Oct 30 14:25:21.735313 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:21.709132 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d57125b3-c6b9-4d3b-b04c-7f6b30094325-istio-data\") pod \"e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh\" (UID: \"d57125b3-c6b9-4d3b-b04c-7f6b30094325\") " pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh" Oct 30 14:25:21.759686 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:21.755961 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:25:21.766667 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:21.763246 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d57125b3-c6b9-4d3b-b04c-7f6b30094325-istio-podinfo\") pod \"e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh\" (UID: \"d57125b3-c6b9-4d3b-b04c-7f6b30094325\") " pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh" Oct 30 14:25:21.792841 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:21.785316 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mlrq\" (UniqueName: \"kubernetes.io/projected/d57125b3-c6b9-4d3b-b04c-7f6b30094325-kube-api-access-6mlrq\") pod \"e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh\" (UID: \"d57125b3-c6b9-4d3b-b04c-7f6b30094325\") " pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh" Oct 30 14:25:21.792841 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:21.792777 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d57125b3-c6b9-4d3b-b04c-7f6b30094325-istio-token\") pod \"e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh\" (UID: \"d57125b3-c6b9-4d3b-b04c-7f6b30094325\") " pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh" Oct 30 14:25:21.802665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:21.798954 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d57125b3-c6b9-4d3b-b04c-7f6b30094325-istio-envoy\") pod \"e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh\" (UID: \"d57125b3-c6b9-4d3b-b04c-7f6b30094325\") " pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh" Oct 30 14:25:21.817664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:21.811912 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh" Oct 30 14:25:22.277162 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:22.276451 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:25:22.460212 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:22.460172 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:25:22.491030 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:25:22.490762 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd57125b3_c6b9_4d3b_b04c_7f6b30094325.slice/crio-90a6976e769b7ec63f22ec95bed16c5235f76f73e299a00c09c11e45561594d7 WatchSource:0}: Error finding container 90a6976e769b7ec63f22ec95bed16c5235f76f73e299a00c09c11e45561594d7: Status 404 returned error can't find the container with id 90a6976e769b7ec63f22ec95bed16c5235f76f73e299a00c09c11e45561594d7 Oct 30 14:25:22.494132 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:22.494101 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh"] Oct 30 14:25:22.501917 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:22.500489 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:25:22.501917 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:22.500579 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:25:22.501917 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:22.500625 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:25:22.751296 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:22.751258 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:25:23.287198 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:23.287144 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh" event={"ID":"d57125b3-c6b9-4d3b-b04c-7f6b30094325","Type":"ContainerStarted","Data":"d2f5f4d9ec23ca09019f62b706839660f2af42d297cc465750ba89ed7c052cd8"} Oct 30 14:25:23.287756 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:23.287224 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh" event={"ID":"d57125b3-c6b9-4d3b-b04c-7f6b30094325","Type":"ContainerStarted","Data":"90a6976e769b7ec63f22ec95bed16c5235f76f73e299a00c09c11e45561594d7"} Oct 30 14:25:23.361064 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:23.361005 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh" podStartSLOduration=2.360990462 podStartE2EDuration="2.360990462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:25:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:25:23.360260462 +0000 UTC m=+2193.104394502" watchObservedRunningTime="2026-10-30 14:25:23.360990462 +0000 UTC m=+2193.105124501" Oct 30 14:25:23.812881 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:23.812843 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh" Oct 30 14:25:23.819535 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:23.819503 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh" Oct 30 14:25:24.297774 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:24.297688 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh" Oct 30 14:25:24.298814 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:24.298789 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh" Oct 30 14:25:24.397402 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:24.397121 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:25:25.807308 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:25.807275 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:25:25.997438 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:25.997379 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:25:26.415657 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:26.415610 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:25:26.622523 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:26.622480 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:25:26.803975 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:26.803885 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:25:27.389888 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:27.389832 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:25:27.544681 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:27.544633 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:25:27.694224 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:27.694135 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:25:27.873931 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:27.873894 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:25:28.025801 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:25:28.025718 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:25:28.597232 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:28.597198 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:25:28.755070 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:28.755033 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:25:29.131933 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:29.131895 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:25:29.614039 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:29.614002 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:25:29.689072 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:29.689035 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fde75d-75cd788695-5n8hq"] Oct 30 14:25:29.694989 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:29.694959 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fde75d-75cd788695-5n8hq" Oct 30 14:25:29.718078 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:29.718048 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fde75d-75cd788695-5n8hq"] Oct 30 14:25:29.725725 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:29.725699 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-fde75d-dockercfg-44fvc\"" Oct 30 14:25:29.725927 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:29.725888 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-fde75d\"" Oct 30 14:25:29.768542 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:29.768497 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:25:29.809676 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:29.801134 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl4rj\" (UniqueName: \"kubernetes.io/projected/a7b9a748-bba1-45e2-9612-0fd47cc5f8a6-kube-api-access-wl4rj\") pod \"payload-pre-processing-e2e-inf-b-fde75d-75cd788695-5n8hq\" (UID: \"a7b9a748-bba1-45e2-9612-0fd47cc5f8a6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fde75d-75cd788695-5n8hq" Oct 30 14:25:29.809676 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:29.801263 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7b9a748-bba1-45e2-9612-0fd47cc5f8a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-fde75d-75cd788695-5n8hq\" (UID: \"a7b9a748-bba1-45e2-9612-0fd47cc5f8a6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fde75d-75cd788695-5n8hq" Oct 30 14:25:29.868729 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:29.868577 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fde75d-7d8d99ffcb-79hw8"] Oct 30 14:25:29.890605 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:29.874565 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-fde75d-7d8d99ffcb-79hw8" Oct 30 14:25:29.918955 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:29.906569 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl4rj\" (UniqueName: \"kubernetes.io/projected/a7b9a748-bba1-45e2-9612-0fd47cc5f8a6-kube-api-access-wl4rj\") pod \"payload-pre-processing-e2e-inf-b-fde75d-75cd788695-5n8hq\" (UID: \"a7b9a748-bba1-45e2-9612-0fd47cc5f8a6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fde75d-75cd788695-5n8hq" Oct 30 14:25:29.918955 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:29.906700 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlr6l\" (UniqueName: \"kubernetes.io/projected/b473ec20-3b31-418d-87ac-bc1269c41744-kube-api-access-hlr6l\") pod \"payload-processing-e2e-inf-b-fde75d-7d8d99ffcb-79hw8\" (UID: \"b473ec20-3b31-418d-87ac-bc1269c41744\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fde75d-7d8d99ffcb-79hw8" Oct 30 14:25:29.918955 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:29.906765 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7b9a748-bba1-45e2-9612-0fd47cc5f8a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-fde75d-75cd788695-5n8hq\" (UID: \"a7b9a748-bba1-45e2-9612-0fd47cc5f8a6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fde75d-75cd788695-5n8hq" Oct 30 14:25:29.918955 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:29.906858 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b473ec20-3b31-418d-87ac-bc1269c41744-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-fde75d-7d8d99ffcb-79hw8\" (UID: \"b473ec20-3b31-418d-87ac-bc1269c41744\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fde75d-7d8d99ffcb-79hw8" Oct 30 14:25:29.918955 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:29.908142 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7b9a748-bba1-45e2-9612-0fd47cc5f8a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-fde75d-75cd788695-5n8hq\" (UID: \"a7b9a748-bba1-45e2-9612-0fd47cc5f8a6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fde75d-75cd788695-5n8hq" Oct 30 14:25:29.999663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:29.973429 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl4rj\" (UniqueName: \"kubernetes.io/projected/a7b9a748-bba1-45e2-9612-0fd47cc5f8a6-kube-api-access-wl4rj\") pod \"payload-pre-processing-e2e-inf-b-fde75d-75cd788695-5n8hq\" (UID: \"a7b9a748-bba1-45e2-9612-0fd47cc5f8a6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fde75d-75cd788695-5n8hq" Oct 30 14:25:30.034664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:30.010169 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlr6l\" (UniqueName: \"kubernetes.io/projected/b473ec20-3b31-418d-87ac-bc1269c41744-kube-api-access-hlr6l\") pod \"payload-processing-e2e-inf-b-fde75d-7d8d99ffcb-79hw8\" (UID: \"b473ec20-3b31-418d-87ac-bc1269c41744\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fde75d-7d8d99ffcb-79hw8" Oct 30 14:25:30.034664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:30.010326 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b473ec20-3b31-418d-87ac-bc1269c41744-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-fde75d-7d8d99ffcb-79hw8\" (UID: \"b473ec20-3b31-418d-87ac-bc1269c41744\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fde75d-7d8d99ffcb-79hw8" Oct 30 14:25:30.034664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:30.011386 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b473ec20-3b31-418d-87ac-bc1269c41744-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-fde75d-7d8d99ffcb-79hw8\" (UID: \"b473ec20-3b31-418d-87ac-bc1269c41744\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fde75d-7d8d99ffcb-79hw8" Oct 30 14:25:30.034664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:30.011568 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fde75d-75cd788695-5n8hq" Oct 30 14:25:30.050698 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:30.050379 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlr6l\" (UniqueName: \"kubernetes.io/projected/b473ec20-3b31-418d-87ac-bc1269c41744-kube-api-access-hlr6l\") pod \"payload-processing-e2e-inf-b-fde75d-7d8d99ffcb-79hw8\" (UID: \"b473ec20-3b31-418d-87ac-bc1269c41744\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fde75d-7d8d99ffcb-79hw8" Oct 30 14:25:30.084027 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:30.083987 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fde75d-7d8d99ffcb-79hw8"] Oct 30 14:25:30.247400 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:30.247362 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20ce18b4-prz95"] Oct 30 14:25:30.253387 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:30.253361 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ce18b4-prz95" Oct 30 14:25:30.296787 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:30.291191 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20ce18b4-prz95"] Oct 30 14:25:30.318920 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:30.318834 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:25:30.319140 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:30.319119 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:25:30.366911 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:30.366869 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-fde75d-7d8d99ffcb-79hw8" Oct 30 14:25:30.368767 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:30.368732 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0255e279-3912-48f3-8710-7ba964b2af0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20ce18b4-prz95\" (UID: \"0255e279-3912-48f3-8710-7ba964b2af0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ce18b4-prz95" Oct 30 14:25:30.368954 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:30.368922 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22bbs\" (UniqueName: \"kubernetes.io/projected/0255e279-3912-48f3-8710-7ba964b2af0a-kube-api-access-22bbs\") pod \"maas-api-revoke-sub-20ce18b4-prz95\" (UID: \"0255e279-3912-48f3-8710-7ba964b2af0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ce18b4-prz95" Oct 30 14:25:30.369029 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:30.369002 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0255e279-3912-48f3-8710-7ba964b2af0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20ce18b4-prz95\" (UID: \"0255e279-3912-48f3-8710-7ba964b2af0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ce18b4-prz95" Oct 30 14:25:30.422839 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:25:30.422803 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7b9a748_bba1_45e2_9612_0fd47cc5f8a6.slice/crio-3e17098b66d5c45c98e6c0d6d260aa7c8d4a69741923d46d45613a600798b4f4 WatchSource:0}: Error finding container 3e17098b66d5c45c98e6c0d6d260aa7c8d4a69741923d46d45613a600798b4f4: Status 404 returned error can't find the container with id 3e17098b66d5c45c98e6c0d6d260aa7c8d4a69741923d46d45613a600798b4f4 Oct 30 14:25:30.426423 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:30.426397 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fde75d-75cd788695-5n8hq"] Oct 30 14:25:30.472135 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:30.472083 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0255e279-3912-48f3-8710-7ba964b2af0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20ce18b4-prz95\" (UID: \"0255e279-3912-48f3-8710-7ba964b2af0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ce18b4-prz95" Oct 30 14:25:30.472277 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:30.472260 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22bbs\" (UniqueName: \"kubernetes.io/projected/0255e279-3912-48f3-8710-7ba964b2af0a-kube-api-access-22bbs\") pod \"maas-api-revoke-sub-20ce18b4-prz95\" (UID: \"0255e279-3912-48f3-8710-7ba964b2af0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ce18b4-prz95" Oct 30 14:25:30.472368 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:30.472348 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0255e279-3912-48f3-8710-7ba964b2af0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20ce18b4-prz95\" (UID: \"0255e279-3912-48f3-8710-7ba964b2af0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ce18b4-prz95" Oct 30 14:25:30.474776 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:30.474712 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0255e279-3912-48f3-8710-7ba964b2af0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20ce18b4-prz95\" (UID: \"0255e279-3912-48f3-8710-7ba964b2af0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ce18b4-prz95" Oct 30 14:25:30.476504 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:30.476483 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0255e279-3912-48f3-8710-7ba964b2af0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20ce18b4-prz95\" (UID: \"0255e279-3912-48f3-8710-7ba964b2af0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ce18b4-prz95" Oct 30 14:25:30.513761 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:30.512715 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22bbs\" (UniqueName: \"kubernetes.io/projected/0255e279-3912-48f3-8710-7ba964b2af0a-kube-api-access-22bbs\") pod \"maas-api-revoke-sub-20ce18b4-prz95\" (UID: \"0255e279-3912-48f3-8710-7ba964b2af0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ce18b4-prz95" Oct 30 14:25:30.598359 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:30.594911 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ce18b4-prz95" Oct 30 14:25:30.727505 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:25:30.727471 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb473ec20_3b31_418d_87ac_bc1269c41744.slice/crio-c6b03550b21a1d631c47441206a40be8999b21376f1ce8081ccbd8fbc005c3fc WatchSource:0}: Error finding container c6b03550b21a1d631c47441206a40be8999b21376f1ce8081ccbd8fbc005c3fc: Status 404 returned error can't find the container with id c6b03550b21a1d631c47441206a40be8999b21376f1ce8081ccbd8fbc005c3fc Oct 30 14:25:30.727891 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:30.727499 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fde75d-7d8d99ffcb-79hw8"] Oct 30 14:25:30.916736 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:30.916428 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20ce18b4-prz95"] Oct 30 14:25:30.916736 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:25:30.916478 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0255e279_3912_48f3_8710_7ba964b2af0a.slice/crio-a5edc0eeb3de2d19d957b551f4e31cd811e9701186a1fd3817eb28f0366bc6b1 WatchSource:0}: Error finding container a5edc0eeb3de2d19d957b551f4e31cd811e9701186a1fd3817eb28f0366bc6b1: Status 404 returned error can't find the container with id a5edc0eeb3de2d19d957b551f4e31cd811e9701186a1fd3817eb28f0366bc6b1 Oct 30 14:25:31.192761 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:31.192726 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:25:31.391269 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:31.391227 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-fde75d-7d8d99ffcb-79hw8" event={"ID":"b473ec20-3b31-418d-87ac-bc1269c41744","Type":"ContainerStarted","Data":"75bd369161fed518765352e20b7d7514b67e39becfd19fb6bd46ca7de3a10bc6"} Oct 30 14:25:31.391461 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:31.391279 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-fde75d-7d8d99ffcb-79hw8" event={"ID":"b473ec20-3b31-418d-87ac-bc1269c41744","Type":"ContainerStarted","Data":"c6b03550b21a1d631c47441206a40be8999b21376f1ce8081ccbd8fbc005c3fc"} Oct 30 14:25:31.393757 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:31.393717 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fde75d-75cd788695-5n8hq" event={"ID":"a7b9a748-bba1-45e2-9612-0fd47cc5f8a6","Type":"ContainerStarted","Data":"c6944f8c7ad6bb968408d0640d78188f7d98c042c5806245ac4c254bcad24437"} Oct 30 14:25:31.393921 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:31.393786 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fde75d-75cd788695-5n8hq" Oct 30 14:25:31.393921 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:31.393811 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fde75d-75cd788695-5n8hq" event={"ID":"a7b9a748-bba1-45e2-9612-0fd47cc5f8a6","Type":"ContainerStarted","Data":"3e17098b66d5c45c98e6c0d6d260aa7c8d4a69741923d46d45613a600798b4f4"} Oct 30 14:25:31.395769 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:31.395741 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ce18b4-prz95" event={"ID":"0255e279-3912-48f3-8710-7ba964b2af0a","Type":"ContainerStarted","Data":"d07ee4ced56696441a46a377d411b21ff950ca3710aeea3786850c25463b0d7f"} Oct 30 14:25:31.395906 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:31.395799 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ce18b4-prz95" event={"ID":"0255e279-3912-48f3-8710-7ba964b2af0a","Type":"ContainerStarted","Data":"a5edc0eeb3de2d19d957b551f4e31cd811e9701186a1fd3817eb28f0366bc6b1"} Oct 30 14:25:31.438354 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:31.438296 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-fde75d-7d8d99ffcb-79hw8" podStartSLOduration=2.438282474 podStartE2EDuration="2.438282474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:25:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:25:31.435205605 +0000 UTC m=+2201.179339646" watchObservedRunningTime="2026-10-30 14:25:31.438282474 +0000 UTC m=+2201.182416515" Oct 30 14:25:31.486014 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:31.485894 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fde75d-75cd788695-5n8hq" podStartSLOduration=2.485874923 podStartE2EDuration="2.485874923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:25:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:25:31.482553843 +0000 UTC m=+2201.226687884" watchObservedRunningTime="2026-10-30 14:25:31.485874923 +0000 UTC m=+2201.230008966" Oct 30 14:25:31.526711 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:31.526623 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ce18b4-prz95" podStartSLOduration=1.526609401 podStartE2EDuration="1.526609401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:25:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:25:31.525957436 +0000 UTC m=+2201.270091483" watchObservedRunningTime="2026-10-30 14:25:31.526609401 +0000 UTC m=+2201.270743460" Oct 30 14:25:32.406470 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:32.406434 4982 generic.go:356] "Generic (PLEG): container finished" podID="0255e279-3912-48f3-8710-7ba964b2af0a" containerID="d07ee4ced56696441a46a377d411b21ff950ca3710aeea3786850c25463b0d7f" exitCode=0 Oct 30 14:25:32.406855 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:32.406519 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ce18b4-prz95" event={"ID":"0255e279-3912-48f3-8710-7ba964b2af0a","Type":"ContainerDied","Data":"d07ee4ced56696441a46a377d411b21ff950ca3710aeea3786850c25463b0d7f"} Oct 30 14:25:32.406935 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:32.406860 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-fde75d-7d8d99ffcb-79hw8" Oct 30 14:25:33.092592 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:33.092555 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:25:33.587005 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:33.586976 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ce18b4-prz95" Oct 30 14:25:33.721121 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:33.721031 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0255e279-3912-48f3-8710-7ba964b2af0a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0255e279-3912-48f3-8710-7ba964b2af0a-maas-api-xks-ca\") pod \"0255e279-3912-48f3-8710-7ba964b2af0a\" (UID: \"0255e279-3912-48f3-8710-7ba964b2af0a\") " Oct 30 14:25:33.721302 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:33.721205 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0255e279-3912-48f3-8710-7ba964b2af0a-kube-api-access-22bbs\" (UniqueName: \"kubernetes.io/projected/0255e279-3912-48f3-8710-7ba964b2af0a-kube-api-access-22bbs\") pod \"0255e279-3912-48f3-8710-7ba964b2af0a\" (UID: \"0255e279-3912-48f3-8710-7ba964b2af0a\") " Oct 30 14:25:33.721302 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:33.721255 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0255e279-3912-48f3-8710-7ba964b2af0a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0255e279-3912-48f3-8710-7ba964b2af0a-maas-api-openshift-ca\") pod \"0255e279-3912-48f3-8710-7ba964b2af0a\" (UID: \"0255e279-3912-48f3-8710-7ba964b2af0a\") " Oct 30 14:25:33.721714 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:33.721685 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0255e279-3912-48f3-8710-7ba964b2af0a-maas-api-openshift-ca" pod "0255e279-3912-48f3-8710-7ba964b2af0a" (UID: "0255e279-3912-48f3-8710-7ba964b2af0a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:25:33.721945 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:33.721918 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0255e279-3912-48f3-8710-7ba964b2af0a-maas-api-openshift-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:25:33.723367 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:33.723342 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0255e279-3912-48f3-8710-7ba964b2af0a-maas-api-xks-ca" pod "0255e279-3912-48f3-8710-7ba964b2af0a" (UID: "0255e279-3912-48f3-8710-7ba964b2af0a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:25:33.723591 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:33.723564 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0255e279-3912-48f3-8710-7ba964b2af0a-kube-api-access-22bbs" pod "0255e279-3912-48f3-8710-7ba964b2af0a" (UID: "0255e279-3912-48f3-8710-7ba964b2af0a"). InnerVolumeSpecName "kube-api-access-22bbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:25:33.823094 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:33.823064 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0255e279-3912-48f3-8710-7ba964b2af0a-maas-api-xks-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:25:33.823094 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:33.823092 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22bbs\" (UniqueName: \"kubernetes.io/projected/0255e279-3912-48f3-8710-7ba964b2af0a-kube-api-access-22bbs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:25:34.427433 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:34.427404 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ce18b4-prz95" Oct 30 14:25:34.427577 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:34.427405 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ce18b4-prz95" event={"ID":"0255e279-3912-48f3-8710-7ba964b2af0a","Type":"ContainerDied","Data":"a5edc0eeb3de2d19d957b551f4e31cd811e9701186a1fd3817eb28f0366bc6b1"} Oct 30 14:25:34.427577 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:34.427541 4982 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5edc0eeb3de2d19d957b551f4e31cd811e9701186a1fd3817eb28f0366bc6b1" Oct 30 14:25:35.091980 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:35.091939 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:25:35.857073 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:35.857033 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:25:37.212303 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:37.212263 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:25:37.365596 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:37.365559 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:25:39.140848 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:39.140814 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:25:40.654152 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:40.654118 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:25:40.814762 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:40.814728 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:25:41.384431 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:41.384388 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:25:42.330124 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:42.330079 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:25:42.410313 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:42.410285 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fde75d-75cd788695-5n8hq" Oct 30 14:25:42.489823 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:42.489787 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b9c4b4-6498846c67-q52pq"] Oct 30 14:25:42.490829 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:42.490800 4982 cpu_manager.go:414] "containerMap: removing container" podUID="0255e279-3912-48f3-8710-7ba964b2af0a" containerName="revoke-keys" Oct 30 14:25:42.491123 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:42.491103 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0255e279-3912-48f3-8710-7ba964b2af0a" containerName="revoke-keys" Oct 30 14:25:42.497245 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:42.497221 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="0255e279-3912-48f3-8710-7ba964b2af0a" containerName="revoke-keys" Oct 30 14:25:42.506518 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:42.506494 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b9c4b4-6498846c67-q52pq" Oct 30 14:25:42.525130 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:42.525101 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:25:42.525771 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:42.525718 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-b9c4b4-dockercfg-z6x8p\"" Oct 30 14:25:42.532692 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:42.530838 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-b9c4b4\"" Oct 30 14:25:42.536292 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:42.536271 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b9c4b4-6498846c67-q52pq"] Oct 30 14:25:42.570417 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:42.570377 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b87e2746-abcd-4e8a-bda1-cf9badd05b04-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b9c4b4-6498846c67-q52pq\" (UID: \"b87e2746-abcd-4e8a-bda1-cf9badd05b04\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b9c4b4-6498846c67-q52pq" Oct 30 14:25:42.570580 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:42.570460 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26gw5\" (UniqueName: \"kubernetes.io/projected/b87e2746-abcd-4e8a-bda1-cf9badd05b04-kube-api-access-26gw5\") pod \"payload-pre-processing-e2e-inf-b-b9c4b4-6498846c67-q52pq\" (UID: \"b87e2746-abcd-4e8a-bda1-cf9badd05b04\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b9c4b4-6498846c67-q52pq" Oct 30 14:25:42.573384 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:42.573355 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b9c4b4-76948497d-7vhrd"] Oct 30 14:25:42.579373 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:42.579348 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b9c4b4-76948497d-7vhrd" Oct 30 14:25:42.629628 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:42.629570 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b9c4b4-76948497d-7vhrd"] Oct 30 14:25:42.690731 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:42.679071 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26gw5\" (UniqueName: \"kubernetes.io/projected/b87e2746-abcd-4e8a-bda1-cf9badd05b04-kube-api-access-26gw5\") pod \"payload-pre-processing-e2e-inf-b-b9c4b4-6498846c67-q52pq\" (UID: \"b87e2746-abcd-4e8a-bda1-cf9badd05b04\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b9c4b4-6498846c67-q52pq" Oct 30 14:25:42.690731 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:42.679178 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjclk\" (UniqueName: \"kubernetes.io/projected/68c95c68-83ae-4ea1-8ef2-b90f99c077e3-kube-api-access-bjclk\") pod \"payload-processing-e2e-inf-b-b9c4b4-76948497d-7vhrd\" (UID: \"68c95c68-83ae-4ea1-8ef2-b90f99c077e3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b9c4b4-76948497d-7vhrd" Oct 30 14:25:42.690731 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:42.679370 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68c95c68-83ae-4ea1-8ef2-b90f99c077e3-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b9c4b4-76948497d-7vhrd\" (UID: \"68c95c68-83ae-4ea1-8ef2-b90f99c077e3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b9c4b4-76948497d-7vhrd" Oct 30 14:25:42.690731 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:42.679577 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b87e2746-abcd-4e8a-bda1-cf9badd05b04-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b9c4b4-6498846c67-q52pq\" (UID: \"b87e2746-abcd-4e8a-bda1-cf9badd05b04\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b9c4b4-6498846c67-q52pq" Oct 30 14:25:42.752714 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:42.736946 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b87e2746-abcd-4e8a-bda1-cf9badd05b04-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b9c4b4-6498846c67-q52pq\" (UID: \"b87e2746-abcd-4e8a-bda1-cf9badd05b04\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b9c4b4-6498846c67-q52pq" Oct 30 14:25:42.811734 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:42.811696 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjclk\" (UniqueName: \"kubernetes.io/projected/68c95c68-83ae-4ea1-8ef2-b90f99c077e3-kube-api-access-bjclk\") pod \"payload-processing-e2e-inf-b-b9c4b4-76948497d-7vhrd\" (UID: \"68c95c68-83ae-4ea1-8ef2-b90f99c077e3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b9c4b4-76948497d-7vhrd" Oct 30 14:25:42.812090 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:42.812068 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68c95c68-83ae-4ea1-8ef2-b90f99c077e3-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b9c4b4-76948497d-7vhrd\" (UID: \"68c95c68-83ae-4ea1-8ef2-b90f99c077e3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b9c4b4-76948497d-7vhrd" Oct 30 14:25:42.842320 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:42.813235 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68c95c68-83ae-4ea1-8ef2-b90f99c077e3-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b9c4b4-76948497d-7vhrd\" (UID: \"68c95c68-83ae-4ea1-8ef2-b90f99c077e3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b9c4b4-76948497d-7vhrd" Oct 30 14:25:42.851705 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:42.835959 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26gw5\" (UniqueName: \"kubernetes.io/projected/b87e2746-abcd-4e8a-bda1-cf9badd05b04-kube-api-access-26gw5\") pod \"payload-pre-processing-e2e-inf-b-b9c4b4-6498846c67-q52pq\" (UID: \"b87e2746-abcd-4e8a-bda1-cf9badd05b04\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b9c4b4-6498846c67-q52pq" Oct 30 14:25:42.870720 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:42.870673 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjclk\" (UniqueName: \"kubernetes.io/projected/68c95c68-83ae-4ea1-8ef2-b90f99c077e3-kube-api-access-bjclk\") pod \"payload-processing-e2e-inf-b-b9c4b4-76948497d-7vhrd\" (UID: \"68c95c68-83ae-4ea1-8ef2-b90f99c077e3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b9c4b4-76948497d-7vhrd" Oct 30 14:25:42.926855 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:42.926818 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b9c4b4-76948497d-7vhrd" Oct 30 14:25:43.017477 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:43.017443 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:25:43.020856 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:43.020833 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d46a90e-mr46c"] Oct 30 14:25:43.027239 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:43.027219 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d46a90e-mr46c" Oct 30 14:25:43.043469 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:43.043448 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d46a90e-mr46c"] Oct 30 14:25:43.066513 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:43.066477 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:25:43.067331 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:43.066768 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:25:43.118206 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:43.118141 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjjpt\" (UniqueName: \"kubernetes.io/projected/6cceaa42-4daf-4c92-b4d8-1a0bdefc49e8-kube-api-access-qjjpt\") pod \"maas-api-revoke-sub-0d46a90e-mr46c\" (UID: \"6cceaa42-4daf-4c92-b4d8-1a0bdefc49e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d46a90e-mr46c" Oct 30 14:25:43.118206 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:43.118188 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cceaa42-4daf-4c92-b4d8-1a0bdefc49e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d46a90e-mr46c\" (UID: \"6cceaa42-4daf-4c92-b4d8-1a0bdefc49e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d46a90e-mr46c" Oct 30 14:25:43.118379 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:43.118318 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cceaa42-4daf-4c92-b4d8-1a0bdefc49e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d46a90e-mr46c\" (UID: \"6cceaa42-4daf-4c92-b4d8-1a0bdefc49e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d46a90e-mr46c" Oct 30 14:25:43.144939 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:43.144913 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b9c4b4-6498846c67-q52pq" Oct 30 14:25:43.220703 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:43.219839 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjjpt\" (UniqueName: \"kubernetes.io/projected/6cceaa42-4daf-4c92-b4d8-1a0bdefc49e8-kube-api-access-qjjpt\") pod \"maas-api-revoke-sub-0d46a90e-mr46c\" (UID: \"6cceaa42-4daf-4c92-b4d8-1a0bdefc49e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d46a90e-mr46c" Oct 30 14:25:43.220703 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:43.219905 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cceaa42-4daf-4c92-b4d8-1a0bdefc49e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d46a90e-mr46c\" (UID: \"6cceaa42-4daf-4c92-b4d8-1a0bdefc49e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d46a90e-mr46c" Oct 30 14:25:43.220703 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:43.220019 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cceaa42-4daf-4c92-b4d8-1a0bdefc49e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d46a90e-mr46c\" (UID: \"6cceaa42-4daf-4c92-b4d8-1a0bdefc49e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d46a90e-mr46c" Oct 30 14:25:43.223862 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:43.223440 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cceaa42-4daf-4c92-b4d8-1a0bdefc49e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d46a90e-mr46c\" (UID: \"6cceaa42-4daf-4c92-b4d8-1a0bdefc49e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d46a90e-mr46c" Oct 30 14:25:43.224459 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:43.224429 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cceaa42-4daf-4c92-b4d8-1a0bdefc49e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d46a90e-mr46c\" (UID: \"6cceaa42-4daf-4c92-b4d8-1a0bdefc49e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d46a90e-mr46c" Oct 30 14:25:43.250379 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:43.250328 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjjpt\" (UniqueName: \"kubernetes.io/projected/6cceaa42-4daf-4c92-b4d8-1a0bdefc49e8-kube-api-access-qjjpt\") pod \"maas-api-revoke-sub-0d46a90e-mr46c\" (UID: \"6cceaa42-4daf-4c92-b4d8-1a0bdefc49e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d46a90e-mr46c" Oct 30 14:25:43.258744 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:43.258517 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b9c4b4-76948497d-7vhrd"] Oct 30 14:25:43.346218 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:43.346196 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d46a90e-mr46c" Oct 30 14:25:43.420896 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:43.420873 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-fde75d-7d8d99ffcb-79hw8" Oct 30 14:25:43.463680 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:43.463652 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b9c4b4-6498846c67-q52pq"] Oct 30 14:25:43.512860 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:43.512805 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b9c4b4-76948497d-7vhrd" event={"ID":"68c95c68-83ae-4ea1-8ef2-b90f99c077e3","Type":"ContainerStarted","Data":"fe7c21f76782ed21ccc3b1d710ec55d95e4a06820290bf20109672b4a006e11e"} Oct 30 14:25:43.513030 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:43.513004 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b9c4b4-76948497d-7vhrd" event={"ID":"68c95c68-83ae-4ea1-8ef2-b90f99c077e3","Type":"ContainerStarted","Data":"c40ff919cbef033f9acda46ba4ed48c33ec75b757f4b1e544ef5ba5344e195c1"} Oct 30 14:25:43.514478 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:43.514456 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-b9c4b4-76948497d-7vhrd" Oct 30 14:25:43.516542 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:43.516514 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b9c4b4-6498846c67-q52pq" event={"ID":"b87e2746-abcd-4e8a-bda1-cf9badd05b04","Type":"ContainerStarted","Data":"37ab85dbbe9f8073f69faf585cc18eca80b229ee00cb91a234b72e0f288af9fc"} Oct 30 14:25:43.563819 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:43.563773 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-b9c4b4-76948497d-7vhrd" podStartSLOduration=1.563760899 podStartE2EDuration="1.563760899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:25:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:25:43.560754798 +0000 UTC m=+2213.304888838" watchObservedRunningTime="2026-10-30 14:25:43.563760899 +0000 UTC m=+2213.307894964" Oct 30 14:25:43.656107 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:25:43.656068 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cceaa42_4daf_4c92_b4d8_1a0bdefc49e8.slice/crio-c55190049f7c9419f9a45afe7bce8dcc0cf00a22e1dbf712313f35981bd054d0 WatchSource:0}: Error finding container c55190049f7c9419f9a45afe7bce8dcc0cf00a22e1dbf712313f35981bd054d0: Status 404 returned error can't find the container with id c55190049f7c9419f9a45afe7bce8dcc0cf00a22e1dbf712313f35981bd054d0 Oct 30 14:25:43.658216 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:43.658185 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d46a90e-mr46c"] Oct 30 14:25:43.884507 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:43.882699 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:25:44.526710 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:44.526602 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b9c4b4-6498846c67-q52pq" event={"ID":"b87e2746-abcd-4e8a-bda1-cf9badd05b04","Type":"ContainerStarted","Data":"7daa8053c072645197f3bec6f19ffed89e19e22e92a3eacdf53da4fb3788d231"} Oct 30 14:25:44.526710 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:44.526700 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b9c4b4-6498846c67-q52pq" Oct 30 14:25:44.528780 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:44.528753 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d46a90e-mr46c" event={"ID":"6cceaa42-4daf-4c92-b4d8-1a0bdefc49e8","Type":"ContainerStarted","Data":"c2c12278bc9458bfed71997cac45483f53606815bd4729e933eedec7e73994ac"} Oct 30 14:25:44.528931 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:44.528799 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d46a90e-mr46c" event={"ID":"6cceaa42-4daf-4c92-b4d8-1a0bdefc49e8","Type":"ContainerStarted","Data":"c55190049f7c9419f9a45afe7bce8dcc0cf00a22e1dbf712313f35981bd054d0"} Oct 30 14:25:44.576676 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:44.576596 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b9c4b4-6498846c67-q52pq" podStartSLOduration=2.576578162 podStartE2EDuration="2.576578162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:25:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:25:44.570961416 +0000 UTC m=+2214.315095457" watchObservedRunningTime="2026-10-30 14:25:44.576578162 +0000 UTC m=+2214.320712203" Oct 30 14:25:44.621759 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:44.621710 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d46a90e-mr46c" podStartSLOduration=2.621694988 podStartE2EDuration="2.621694988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:25:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:25:44.619230867 +0000 UTC m=+2214.363364909" watchObservedRunningTime="2026-10-30 14:25:44.621694988 +0000 UTC m=+2214.365829025" Oct 30 14:25:44.677917 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:44.677881 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:25:45.538317 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:45.538281 4982 generic.go:356] "Generic (PLEG): container finished" podID="6cceaa42-4daf-4c92-b4d8-1a0bdefc49e8" containerID="c2c12278bc9458bfed71997cac45483f53606815bd4729e933eedec7e73994ac" exitCode=0 Oct 30 14:25:45.538792 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:45.538363 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d46a90e-mr46c" event={"ID":"6cceaa42-4daf-4c92-b4d8-1a0bdefc49e8","Type":"ContainerDied","Data":"c2c12278bc9458bfed71997cac45483f53606815bd4729e933eedec7e73994ac"} Oct 30 14:25:46.126210 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:46.125544 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:25:46.730238 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:46.730205 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d46a90e-mr46c" Oct 30 14:25:46.775127 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:46.775098 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cceaa42-4daf-4c92-b4d8-1a0bdefc49e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cceaa42-4daf-4c92-b4d8-1a0bdefc49e8-maas-api-openshift-ca\") pod \"6cceaa42-4daf-4c92-b4d8-1a0bdefc49e8\" (UID: \"6cceaa42-4daf-4c92-b4d8-1a0bdefc49e8\") " Oct 30 14:25:46.775302 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:46.775173 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cceaa42-4daf-4c92-b4d8-1a0bdefc49e8-kube-api-access-qjjpt\" (UniqueName: \"kubernetes.io/projected/6cceaa42-4daf-4c92-b4d8-1a0bdefc49e8-kube-api-access-qjjpt\") pod \"6cceaa42-4daf-4c92-b4d8-1a0bdefc49e8\" (UID: \"6cceaa42-4daf-4c92-b4d8-1a0bdefc49e8\") " Oct 30 14:25:46.775478 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:46.775452 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cceaa42-4daf-4c92-b4d8-1a0bdefc49e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cceaa42-4daf-4c92-b4d8-1a0bdefc49e8-maas-api-xks-ca\") pod \"6cceaa42-4daf-4c92-b4d8-1a0bdefc49e8\" (UID: \"6cceaa42-4daf-4c92-b4d8-1a0bdefc49e8\") " Oct 30 14:25:46.777049 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:46.775476 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cceaa42-4daf-4c92-b4d8-1a0bdefc49e8-maas-api-openshift-ca" pod "6cceaa42-4daf-4c92-b4d8-1a0bdefc49e8" (UID: "6cceaa42-4daf-4c92-b4d8-1a0bdefc49e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:25:46.777958 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:46.777930 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cceaa42-4daf-4c92-b4d8-1a0bdefc49e8-maas-api-xks-ca" pod "6cceaa42-4daf-4c92-b4d8-1a0bdefc49e8" (UID: "6cceaa42-4daf-4c92-b4d8-1a0bdefc49e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:25:46.793849 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:46.793823 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cceaa42-4daf-4c92-b4d8-1a0bdefc49e8-kube-api-access-qjjpt" pod "6cceaa42-4daf-4c92-b4d8-1a0bdefc49e8" (UID: "6cceaa42-4daf-4c92-b4d8-1a0bdefc49e8"). InnerVolumeSpecName "kube-api-access-qjjpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:25:46.878251 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:46.878214 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cceaa42-4daf-4c92-b4d8-1a0bdefc49e8-maas-api-openshift-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:25:46.878251 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:46.878245 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjjpt\" (UniqueName: \"kubernetes.io/projected/6cceaa42-4daf-4c92-b4d8-1a0bdefc49e8-kube-api-access-qjjpt\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:25:46.878251 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:46.878255 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cceaa42-4daf-4c92-b4d8-1a0bdefc49e8-maas-api-xks-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:25:47.556756 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:47.556728 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d46a90e-mr46c" Oct 30 14:25:47.556919 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:47.556765 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d46a90e-mr46c" event={"ID":"6cceaa42-4daf-4c92-b4d8-1a0bdefc49e8","Type":"ContainerDied","Data":"c55190049f7c9419f9a45afe7bce8dcc0cf00a22e1dbf712313f35981bd054d0"} Oct 30 14:25:47.556919 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:47.556817 4982 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c55190049f7c9419f9a45afe7bce8dcc0cf00a22e1dbf712313f35981bd054d0" Oct 30 14:25:48.143042 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:48.143002 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-93549e-65ffc5485c-wqqd4"] Oct 30 14:25:48.144086 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:48.144059 4982 cpu_manager.go:414] "containerMap: removing container" podUID="6cceaa42-4daf-4c92-b4d8-1a0bdefc49e8" containerName="revoke-keys" Oct 30 14:25:48.144086 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:48.144084 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cceaa42-4daf-4c92-b4d8-1a0bdefc49e8" containerName="revoke-keys" Oct 30 14:25:48.144274 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:48.144220 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cceaa42-4daf-4c92-b4d8-1a0bdefc49e8" containerName="revoke-keys" Oct 30 14:25:48.148674 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:48.148652 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-93549e-65ffc5485c-wqqd4" Oct 30 14:25:48.203197 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:48.203155 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-93549e-serving-cert\"" Oct 30 14:25:48.209371 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:48.209339 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-93549e-65ffc5485c-wqqd4"] Oct 30 14:25:48.299598 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:48.299552 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3ef6149-023a-44f7-a1d7-ac0acd7de745-openshift-service-ca\") pod \"maas-api-e2e-inf-b-93549e-65ffc5485c-wqqd4\" (UID: \"f3ef6149-023a-44f7-a1d7-ac0acd7de745\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-93549e-65ffc5485c-wqqd4" Oct 30 14:25:48.299788 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:48.299617 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzc5h\" (UniqueName: \"kubernetes.io/projected/f3ef6149-023a-44f7-a1d7-ac0acd7de745-kube-api-access-wzc5h\") pod \"maas-api-e2e-inf-b-93549e-65ffc5485c-wqqd4\" (UID: \"f3ef6149-023a-44f7-a1d7-ac0acd7de745\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-93549e-65ffc5485c-wqqd4" Oct 30 14:25:48.299788 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:48.299749 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3ef6149-023a-44f7-a1d7-ac0acd7de745-metrics-certs\") pod \"maas-api-e2e-inf-b-93549e-65ffc5485c-wqqd4\" (UID: \"f3ef6149-023a-44f7-a1d7-ac0acd7de745\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-93549e-65ffc5485c-wqqd4" Oct 30 14:25:48.300047 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:48.300031 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3ef6149-023a-44f7-a1d7-ac0acd7de745-maas-api-tls\") pod \"maas-api-e2e-inf-b-93549e-65ffc5485c-wqqd4\" (UID: \"f3ef6149-023a-44f7-a1d7-ac0acd7de745\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-93549e-65ffc5485c-wqqd4" Oct 30 14:25:48.308533 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:48.308502 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:25:48.408108 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:48.407567 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3ef6149-023a-44f7-a1d7-ac0acd7de745-maas-api-tls\") pod \"maas-api-e2e-inf-b-93549e-65ffc5485c-wqqd4\" (UID: \"f3ef6149-023a-44f7-a1d7-ac0acd7de745\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-93549e-65ffc5485c-wqqd4" Oct 30 14:25:48.408108 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:48.407694 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3ef6149-023a-44f7-a1d7-ac0acd7de745-openshift-service-ca\") pod \"maas-api-e2e-inf-b-93549e-65ffc5485c-wqqd4\" (UID: \"f3ef6149-023a-44f7-a1d7-ac0acd7de745\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-93549e-65ffc5485c-wqqd4" Oct 30 14:25:48.408108 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:48.407766 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzc5h\" (UniqueName: \"kubernetes.io/projected/f3ef6149-023a-44f7-a1d7-ac0acd7de745-kube-api-access-wzc5h\") pod \"maas-api-e2e-inf-b-93549e-65ffc5485c-wqqd4\" (UID: \"f3ef6149-023a-44f7-a1d7-ac0acd7de745\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-93549e-65ffc5485c-wqqd4" Oct 30 14:25:48.408108 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:48.407806 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3ef6149-023a-44f7-a1d7-ac0acd7de745-metrics-certs\") pod \"maas-api-e2e-inf-b-93549e-65ffc5485c-wqqd4\" (UID: \"f3ef6149-023a-44f7-a1d7-ac0acd7de745\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-93549e-65ffc5485c-wqqd4" Oct 30 14:25:48.409008 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:25:48.408981 4982 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-93549e-serving-cert: secret "maas-api-e2e-inf-b-93549e-serving-cert" not found Oct 30 14:25:48.409149 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:25:48.409057 4982 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3ef6149-023a-44f7-a1d7-ac0acd7de745-maas-api-tls podName:f3ef6149-023a-44f7-a1d7-ac0acd7de745 nodeName:}" failed. No retries permitted until 2026-10-30 14:25:48.909038161 +0000 UTC m=+2218.653172180 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f3ef6149-023a-44f7-a1d7-ac0acd7de745-maas-api-tls") pod "maas-api-e2e-inf-b-93549e-65ffc5485c-wqqd4" (UID: "f3ef6149-023a-44f7-a1d7-ac0acd7de745") : secret "maas-api-e2e-inf-b-93549e-serving-cert" not found Oct 30 14:25:48.410202 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:48.409998 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3ef6149-023a-44f7-a1d7-ac0acd7de745-openshift-service-ca\") pod \"maas-api-e2e-inf-b-93549e-65ffc5485c-wqqd4\" (UID: \"f3ef6149-023a-44f7-a1d7-ac0acd7de745\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-93549e-65ffc5485c-wqqd4" Oct 30 14:25:48.430461 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:48.430344 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3ef6149-023a-44f7-a1d7-ac0acd7de745-metrics-certs\") pod \"maas-api-e2e-inf-b-93549e-65ffc5485c-wqqd4\" (UID: \"f3ef6149-023a-44f7-a1d7-ac0acd7de745\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-93549e-65ffc5485c-wqqd4" Oct 30 14:25:48.438544 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:48.438515 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzc5h\" (UniqueName: \"kubernetes.io/projected/f3ef6149-023a-44f7-a1d7-ac0acd7de745-kube-api-access-wzc5h\") pod \"maas-api-e2e-inf-b-93549e-65ffc5485c-wqqd4\" (UID: \"f3ef6149-023a-44f7-a1d7-ac0acd7de745\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-93549e-65ffc5485c-wqqd4" Oct 30 14:25:48.455421 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:48.455389 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:25:48.632704 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:48.632664 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:25:48.916245 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:48.916205 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3ef6149-023a-44f7-a1d7-ac0acd7de745-maas-api-tls\") pod \"maas-api-e2e-inf-b-93549e-65ffc5485c-wqqd4\" (UID: \"f3ef6149-023a-44f7-a1d7-ac0acd7de745\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-93549e-65ffc5485c-wqqd4" Oct 30 14:25:48.918794 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:48.918770 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3ef6149-023a-44f7-a1d7-ac0acd7de745-maas-api-tls\") pod \"maas-api-e2e-inf-b-93549e-65ffc5485c-wqqd4\" (UID: \"f3ef6149-023a-44f7-a1d7-ac0acd7de745\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-93549e-65ffc5485c-wqqd4" Oct 30 14:25:49.063240 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:49.063206 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-93549e-65ffc5485c-wqqd4" Oct 30 14:25:49.394046 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:25:49.394017 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3ef6149_023a_44f7_a1d7_ac0acd7de745.slice/crio-2900a84fe2192570e18275d72a75957c015f8623709bb2ee87cea26147544497 WatchSource:0}: Error finding container 2900a84fe2192570e18275d72a75957c015f8623709bb2ee87cea26147544497: Status 404 returned error can't find the container with id 2900a84fe2192570e18275d72a75957c015f8623709bb2ee87cea26147544497 Oct 30 14:25:49.394469 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:49.394270 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-93549e-65ffc5485c-wqqd4"] Oct 30 14:25:49.396077 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:49.396054 4982 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 14:25:49.573048 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:49.573012 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-93549e-65ffc5485c-wqqd4" event={"ID":"f3ef6149-023a-44f7-a1d7-ac0acd7de745","Type":"ContainerStarted","Data":"2900a84fe2192570e18275d72a75957c015f8623709bb2ee87cea26147544497"} Oct 30 14:25:50.376664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:50.374603 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:25:50.583222 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:50.583173 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-93549e-65ffc5485c-wqqd4" event={"ID":"f3ef6149-023a-44f7-a1d7-ac0acd7de745","Type":"ContainerStarted","Data":"a5b64e11026c8677183555d580dd1cbe7180e04ddb14a0cb8d7034df3880fb69"} Oct 30 14:25:50.583604 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:50.583248 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-93549e-65ffc5485c-wqqd4" Oct 30 14:25:50.633308 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:50.633219 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-93549e-65ffc5485c-wqqd4" podStartSLOduration=2.30168001 podStartE2EDuration="2.633204818s" totalImagesPullingTime="331.524808ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:25:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:25:50.632846287 +0000 UTC m=+2220.376980328" watchObservedRunningTime="2026-10-30 14:25:50.633204818 +0000 UTC m=+2220.377338858" Oct 30 14:25:52.357997 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:52.356030 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:25:54.239399 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:54.239334 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:25:55.541922 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:55.541884 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-b9c4b4-76948497d-7vhrd" Oct 30 14:25:55.542341 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:55.541963 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b9c4b4-6498846c67-q52pq" Oct 30 14:25:56.275070 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:56.275024 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:25:56.598739 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:56.598705 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-93549e-65ffc5485c-wqqd4" Oct 30 14:25:58.016739 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:25:58.016697 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:25:58.176178 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:25:58.176068 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:26:00.031879 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:00.031841 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:26:02.021197 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:02.021156 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:26:03.672112 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:03.672072 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:26:03.872508 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:03.872471 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:26:04.051854 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:04.051755 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:26:05.526947 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:05.526909 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:26:06.132779 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:06.124415 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:26:06.268881 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:06.268847 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea9ed2-nrct6"] Oct 30 14:26:06.281019 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:06.280985 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea9ed2-nrct6"] Oct 30 14:26:06.283796 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:06.283756 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:26:06.455713 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:06.451709 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e145e1fb-ad70-46eb-bba7-adbf5cab95b8" path="/var/lib/kubelet/pods/e145e1fb-ad70-46eb-bba7-adbf5cab95b8/volumes" Oct 30 14:26:06.455713 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:06.452524 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:26:06.788187 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:06.788071 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:26:08.229513 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:08.229482 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:26:08.519984 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:08.519892 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:26:10.410487 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:10.410453 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:26:12.280035 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:12.279988 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:26:12.904979 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:12.904935 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:26:13.050499 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:13.050464 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:26:14.097210 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:14.097160 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:26:14.247663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:14.247265 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:26:14.415709 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:14.414995 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:26:14.618235 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:14.618196 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn"] Oct 30 14:26:14.619049 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:14.619022 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn" podUID="fb9065d6-3192-463a-bfb3-bda7753fefe2" containerName="istio-proxy" containerID="cri-o://84d90f3062ee12186fa43cb855201bd800ded5f1f48baa3821ff9ac6284630bb" gracePeriod=30 Oct 30 14:26:14.625543 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:14.625519 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm"] Oct 30 14:26:14.625912 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:14.625889 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm" podUID="20041b11-f00c-4853-acf3-7c210f742f6d" containerName="istio-proxy" containerID="cri-o://53034c6d7382e2872770a780f003f76a50435ecb7a012daab23a5c6c68c50249" gracePeriod=30 Oct 30 14:26:14.633215 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:14.633190 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5"] Oct 30 14:26:14.633404 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:14.633382 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5" podUID="d85ef9d1-c82a-4957-a041-6995e428f4a1" containerName="istio-proxy" containerID="cri-o://c9e6f9a5bc9e3be4dba76192a17491b9b76115568e29499bf60b91ee0ce47016" gracePeriod=30 Oct 30 14:26:14.656628 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:14.656598 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:26:14.922159 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:14.920790 4982 patch_prober.go:29] interesting pod/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 14:26:14.922159 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:14.920885 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm" podUID="20041b11-f00c-4853-acf3-7c210f742f6d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 14:26:15.505377 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:15.505340 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:26:16.185712 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:16.184516 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:26:16.420664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:16.420415 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:26:18.244023 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:18.243974 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:26:18.797486 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:18.797451 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5a9d7b-9bb744df9-2qrjh"] Oct 30 14:26:18.936361 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:18.936310 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:26:19.090440 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:19.090394 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:26:19.916199 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:19.915870 4982 generic.go:356] "Generic (PLEG): container finished" podID="d85ef9d1-c82a-4957-a041-6995e428f4a1" containerID="c9e6f9a5bc9e3be4dba76192a17491b9b76115568e29499bf60b91ee0ce47016" exitCode=0 Oct 30 14:26:19.916199 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:19.916080 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5" event={"ID":"d85ef9d1-c82a-4957-a041-6995e428f4a1","Type":"ContainerDied","Data":"c9e6f9a5bc9e3be4dba76192a17491b9b76115568e29499bf60b91ee0ce47016"} Oct 30 14:26:19.919233 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:19.919029 4982 generic.go:356] "Generic (PLEG): container finished" podID="20041b11-f00c-4853-acf3-7c210f742f6d" containerID="53034c6d7382e2872770a780f003f76a50435ecb7a012daab23a5c6c68c50249" exitCode=0 Oct 30 14:26:19.919233 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:19.919141 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm" event={"ID":"20041b11-f00c-4853-acf3-7c210f742f6d","Type":"ContainerDied","Data":"53034c6d7382e2872770a780f003f76a50435ecb7a012daab23a5c6c68c50249"} Oct 30 14:26:19.922458 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:19.922325 4982 generic.go:356] "Generic (PLEG): container finished" podID="fb9065d6-3192-463a-bfb3-bda7753fefe2" containerID="84d90f3062ee12186fa43cb855201bd800ded5f1f48baa3821ff9ac6284630bb" exitCode=0 Oct 30 14:26:19.922458 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:19.922362 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn" event={"ID":"fb9065d6-3192-463a-bfb3-bda7753fefe2","Type":"ContainerDied","Data":"84d90f3062ee12186fa43cb855201bd800ded5f1f48baa3821ff9ac6284630bb"} Oct 30 14:26:19.978460 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:19.978278 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm" Oct 30 14:26:20.007152 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.007114 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn" Oct 30 14:26:20.036007 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.035981 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5" Oct 30 14:26:20.086052 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.086015 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20041b11-f00c-4853-acf3-7c210f742f6d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20041b11-f00c-4853-acf3-7c210f742f6d-workload-certs\") pod \"20041b11-f00c-4853-acf3-7c210f742f6d\" (UID: \"20041b11-f00c-4853-acf3-7c210f742f6d\") " Oct 30 14:26:20.086239 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.086065 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20041b11-f00c-4853-acf3-7c210f742f6d-kube-api-access-5qpsv\" (UniqueName: \"kubernetes.io/projected/20041b11-f00c-4853-acf3-7c210f742f6d-kube-api-access-5qpsv\") pod \"20041b11-f00c-4853-acf3-7c210f742f6d\" (UID: \"20041b11-f00c-4853-acf3-7c210f742f6d\") " Oct 30 14:26:20.086239 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.086115 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20041b11-f00c-4853-acf3-7c210f742f6d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20041b11-f00c-4853-acf3-7c210f742f6d-istio-data\") pod \"20041b11-f00c-4853-acf3-7c210f742f6d\" (UID: \"20041b11-f00c-4853-acf3-7c210f742f6d\") " Oct 30 14:26:20.086239 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.086154 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20041b11-f00c-4853-acf3-7c210f742f6d-istio-token\" (UniqueName: \"kubernetes.io/projected/20041b11-f00c-4853-acf3-7c210f742f6d-istio-token\") pod \"20041b11-f00c-4853-acf3-7c210f742f6d\" (UID: \"20041b11-f00c-4853-acf3-7c210f742f6d\") " Oct 30 14:26:20.086239 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.086191 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20041b11-f00c-4853-acf3-7c210f742f6d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20041b11-f00c-4853-acf3-7c210f742f6d-workload-socket\") pod \"20041b11-f00c-4853-acf3-7c210f742f6d\" (UID: \"20041b11-f00c-4853-acf3-7c210f742f6d\") " Oct 30 14:26:20.086472 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.086279 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20041b11-f00c-4853-acf3-7c210f742f6d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20041b11-f00c-4853-acf3-7c210f742f6d-istio-envoy\") pod \"20041b11-f00c-4853-acf3-7c210f742f6d\" (UID: \"20041b11-f00c-4853-acf3-7c210f742f6d\") " Oct 30 14:26:20.086472 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.086336 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20041b11-f00c-4853-acf3-7c210f742f6d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20041b11-f00c-4853-acf3-7c210f742f6d-istiod-ca-cert\") pod \"20041b11-f00c-4853-acf3-7c210f742f6d\" (UID: \"20041b11-f00c-4853-acf3-7c210f742f6d\") " Oct 30 14:26:20.086472 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.086364 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/20041b11-f00c-4853-acf3-7c210f742f6d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20041b11-f00c-4853-acf3-7c210f742f6d-istio-podinfo\") pod \"20041b11-f00c-4853-acf3-7c210f742f6d\" (UID: \"20041b11-f00c-4853-acf3-7c210f742f6d\") " Oct 30 14:26:20.086472 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.086395 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20041b11-f00c-4853-acf3-7c210f742f6d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20041b11-f00c-4853-acf3-7c210f742f6d-credential-socket\") pod \"20041b11-f00c-4853-acf3-7c210f742f6d\" (UID: \"20041b11-f00c-4853-acf3-7c210f742f6d\") " Oct 30 14:26:20.087383 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.087350 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20041b11-f00c-4853-acf3-7c210f742f6d-credential-socket" pod "20041b11-f00c-4853-acf3-7c210f742f6d" (UID: "20041b11-f00c-4853-acf3-7c210f742f6d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:20.087589 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.087569 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20041b11-f00c-4853-acf3-7c210f742f6d-workload-certs" pod "20041b11-f00c-4853-acf3-7c210f742f6d" (UID: "20041b11-f00c-4853-acf3-7c210f742f6d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:20.088067 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.088040 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20041b11-f00c-4853-acf3-7c210f742f6d-workload-socket" pod "20041b11-f00c-4853-acf3-7c210f742f6d" (UID: "20041b11-f00c-4853-acf3-7c210f742f6d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:20.089425 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.089396 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20041b11-f00c-4853-acf3-7c210f742f6d-istio-data" pod "20041b11-f00c-4853-acf3-7c210f742f6d" (UID: "20041b11-f00c-4853-acf3-7c210f742f6d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:20.102783 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.097613 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20041b11-f00c-4853-acf3-7c210f742f6d-istio-envoy" pod "20041b11-f00c-4853-acf3-7c210f742f6d" (UID: "20041b11-f00c-4853-acf3-7c210f742f6d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:20.105209 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.105150 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20041b11-f00c-4853-acf3-7c210f742f6d-istio-token" pod "20041b11-f00c-4853-acf3-7c210f742f6d" (UID: "20041b11-f00c-4853-acf3-7c210f742f6d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:26:20.105505 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.105478 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20041b11-f00c-4853-acf3-7c210f742f6d-kube-api-access-5qpsv" pod "20041b11-f00c-4853-acf3-7c210f742f6d" (UID: "20041b11-f00c-4853-acf3-7c210f742f6d"). InnerVolumeSpecName "kube-api-access-5qpsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:26:20.106036 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.106010 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20041b11-f00c-4853-acf3-7c210f742f6d-istiod-ca-cert" pod "20041b11-f00c-4853-acf3-7c210f742f6d" (UID: "20041b11-f00c-4853-acf3-7c210f742f6d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:26:20.108092 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.108061 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20041b11-f00c-4853-acf3-7c210f742f6d-istio-podinfo" pod "20041b11-f00c-4853-acf3-7c210f742f6d" (UID: "20041b11-f00c-4853-acf3-7c210f742f6d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:26:20.166519 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.166362 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:26:20.193884 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.193226 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d85ef9d1-c82a-4957-a041-6995e428f4a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d85ef9d1-c82a-4957-a041-6995e428f4a1-istio-envoy\") pod \"d85ef9d1-c82a-4957-a041-6995e428f4a1\" (UID: \"d85ef9d1-c82a-4957-a041-6995e428f4a1\") " Oct 30 14:26:20.193884 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.193280 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb9065d6-3192-463a-bfb3-bda7753fefe2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb9065d6-3192-463a-bfb3-bda7753fefe2-workload-socket\") pod \"fb9065d6-3192-463a-bfb3-bda7753fefe2\" (UID: \"fb9065d6-3192-463a-bfb3-bda7753fefe2\") " Oct 30 14:26:20.193884 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.193318 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fb9065d6-3192-463a-bfb3-bda7753fefe2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb9065d6-3192-463a-bfb3-bda7753fefe2-istio-podinfo\") pod \"fb9065d6-3192-463a-bfb3-bda7753fefe2\" (UID: \"fb9065d6-3192-463a-bfb3-bda7753fefe2\") " Oct 30 14:26:20.193884 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.193350 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d85ef9d1-c82a-4957-a041-6995e428f4a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d85ef9d1-c82a-4957-a041-6995e428f4a1-workload-certs\") pod \"d85ef9d1-c82a-4957-a041-6995e428f4a1\" (UID: \"d85ef9d1-c82a-4957-a041-6995e428f4a1\") " Oct 30 14:26:20.193884 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.193385 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d85ef9d1-c82a-4957-a041-6995e428f4a1-kube-api-access-278qx\" (UniqueName: \"kubernetes.io/projected/d85ef9d1-c82a-4957-a041-6995e428f4a1-kube-api-access-278qx\") pod \"d85ef9d1-c82a-4957-a041-6995e428f4a1\" (UID: \"d85ef9d1-c82a-4957-a041-6995e428f4a1\") " Oct 30 14:26:20.193884 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.193412 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb9065d6-3192-463a-bfb3-bda7753fefe2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb9065d6-3192-463a-bfb3-bda7753fefe2-istio-data\") pod \"fb9065d6-3192-463a-bfb3-bda7753fefe2\" (UID: \"fb9065d6-3192-463a-bfb3-bda7753fefe2\") " Oct 30 14:26:20.193884 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.193474 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d85ef9d1-c82a-4957-a041-6995e428f4a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85ef9d1-c82a-4957-a041-6995e428f4a1-workload-socket\") pod \"d85ef9d1-c82a-4957-a041-6995e428f4a1\" (UID: \"d85ef9d1-c82a-4957-a041-6995e428f4a1\") " Oct 30 14:26:20.193884 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.193552 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d85ef9d1-c82a-4957-a041-6995e428f4a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d85ef9d1-c82a-4957-a041-6995e428f4a1-istio-data\") pod \"d85ef9d1-c82a-4957-a041-6995e428f4a1\" (UID: \"d85ef9d1-c82a-4957-a041-6995e428f4a1\") " Oct 30 14:26:20.193884 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.193615 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb9065d6-3192-463a-bfb3-bda7753fefe2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb9065d6-3192-463a-bfb3-bda7753fefe2-istiod-ca-cert\") pod \"fb9065d6-3192-463a-bfb3-bda7753fefe2\" (UID: \"fb9065d6-3192-463a-bfb3-bda7753fefe2\") " Oct 30 14:26:20.193884 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.193683 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb9065d6-3192-463a-bfb3-bda7753fefe2-kube-api-access-6f57l\" (UniqueName: \"kubernetes.io/projected/fb9065d6-3192-463a-bfb3-bda7753fefe2-kube-api-access-6f57l\") pod \"fb9065d6-3192-463a-bfb3-bda7753fefe2\" (UID: \"fb9065d6-3192-463a-bfb3-bda7753fefe2\") " Oct 30 14:26:20.193884 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.193752 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d85ef9d1-c82a-4957-a041-6995e428f4a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d85ef9d1-c82a-4957-a041-6995e428f4a1-istiod-ca-cert\") pod \"d85ef9d1-c82a-4957-a041-6995e428f4a1\" (UID: \"d85ef9d1-c82a-4957-a041-6995e428f4a1\") " Oct 30 14:26:20.193884 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.193818 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d85ef9d1-c82a-4957-a041-6995e428f4a1-istio-token\" (UniqueName: \"kubernetes.io/projected/d85ef9d1-c82a-4957-a041-6995e428f4a1-istio-token\") pod \"d85ef9d1-c82a-4957-a041-6995e428f4a1\" (UID: \"d85ef9d1-c82a-4957-a041-6995e428f4a1\") " Oct 30 14:26:20.193884 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.193865 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d85ef9d1-c82a-4957-a041-6995e428f4a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85ef9d1-c82a-4957-a041-6995e428f4a1-credential-socket\") pod \"d85ef9d1-c82a-4957-a041-6995e428f4a1\" (UID: \"d85ef9d1-c82a-4957-a041-6995e428f4a1\") " Oct 30 14:26:20.194792 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.193911 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb9065d6-3192-463a-bfb3-bda7753fefe2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb9065d6-3192-463a-bfb3-bda7753fefe2-workload-certs\") pod \"fb9065d6-3192-463a-bfb3-bda7753fefe2\" (UID: \"fb9065d6-3192-463a-bfb3-bda7753fefe2\") " Oct 30 14:26:20.194792 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.193962 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb9065d6-3192-463a-bfb3-bda7753fefe2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb9065d6-3192-463a-bfb3-bda7753fefe2-credential-socket\") pod \"fb9065d6-3192-463a-bfb3-bda7753fefe2\" (UID: \"fb9065d6-3192-463a-bfb3-bda7753fefe2\") " Oct 30 14:26:20.194792 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.194004 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb9065d6-3192-463a-bfb3-bda7753fefe2-istio-token\" (UniqueName: \"kubernetes.io/projected/fb9065d6-3192-463a-bfb3-bda7753fefe2-istio-token\") pod \"fb9065d6-3192-463a-bfb3-bda7753fefe2\" (UID: \"fb9065d6-3192-463a-bfb3-bda7753fefe2\") " Oct 30 14:26:20.194792 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.194059 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb9065d6-3192-463a-bfb3-bda7753fefe2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb9065d6-3192-463a-bfb3-bda7753fefe2-istio-envoy\") pod \"fb9065d6-3192-463a-bfb3-bda7753fefe2\" (UID: \"fb9065d6-3192-463a-bfb3-bda7753fefe2\") " Oct 30 14:26:20.194792 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.194095 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d85ef9d1-c82a-4957-a041-6995e428f4a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d85ef9d1-c82a-4957-a041-6995e428f4a1-istio-podinfo\") pod \"d85ef9d1-c82a-4957-a041-6995e428f4a1\" (UID: \"d85ef9d1-c82a-4957-a041-6995e428f4a1\") " Oct 30 14:26:20.194792 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.194311 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d85ef9d1-c82a-4957-a041-6995e428f4a1-istio-data" pod "d85ef9d1-c82a-4957-a041-6995e428f4a1" (UID: "d85ef9d1-c82a-4957-a041-6995e428f4a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:20.195113 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.194852 4982 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20041b11-f00c-4853-acf3-7c210f742f6d-istio-envoy\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:26:20.195113 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.194873 4982 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20041b11-f00c-4853-acf3-7c210f742f6d-istiod-ca-cert\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:26:20.195113 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.194887 4982 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20041b11-f00c-4853-acf3-7c210f742f6d-istio-podinfo\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:26:20.195113 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.194902 4982 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20041b11-f00c-4853-acf3-7c210f742f6d-credential-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:26:20.195113 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.194916 4982 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d85ef9d1-c82a-4957-a041-6995e428f4a1-istio-data\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:26:20.195113 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.194929 4982 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20041b11-f00c-4853-acf3-7c210f742f6d-workload-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:26:20.195113 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.194972 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qpsv\" (UniqueName: \"kubernetes.io/projected/20041b11-f00c-4853-acf3-7c210f742f6d-kube-api-access-5qpsv\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:26:20.195113 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.194986 4982 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20041b11-f00c-4853-acf3-7c210f742f6d-istio-data\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:26:20.195113 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.195002 4982 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20041b11-f00c-4853-acf3-7c210f742f6d-istio-token\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:26:20.195113 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.195015 4982 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20041b11-f00c-4853-acf3-7c210f742f6d-workload-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:26:20.195597 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.195195 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb9065d6-3192-463a-bfb3-bda7753fefe2-workload-socket" pod "fb9065d6-3192-463a-bfb3-bda7753fefe2" (UID: "fb9065d6-3192-463a-bfb3-bda7753fefe2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:20.195597 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.195351 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d85ef9d1-c82a-4957-a041-6995e428f4a1-workload-certs" pod "d85ef9d1-c82a-4957-a041-6995e428f4a1" (UID: "d85ef9d1-c82a-4957-a041-6995e428f4a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:20.212749 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.207149 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb9065d6-3192-463a-bfb3-bda7753fefe2-istio-podinfo" pod "fb9065d6-3192-463a-bfb3-bda7753fefe2" (UID: "fb9065d6-3192-463a-bfb3-bda7753fefe2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:26:20.212749 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.207238 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d85ef9d1-c82a-4957-a041-6995e428f4a1-istio-envoy" pod "d85ef9d1-c82a-4957-a041-6995e428f4a1" (UID: "d85ef9d1-c82a-4957-a041-6995e428f4a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:20.212749 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.211588 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb9065d6-3192-463a-bfb3-bda7753fefe2-istio-data" pod "fb9065d6-3192-463a-bfb3-bda7753fefe2" (UID: "fb9065d6-3192-463a-bfb3-bda7753fefe2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:20.212749 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.211902 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb9065d6-3192-463a-bfb3-bda7753fefe2-credential-socket" pod "fb9065d6-3192-463a-bfb3-bda7753fefe2" (UID: "fb9065d6-3192-463a-bfb3-bda7753fefe2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:20.212749 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.212154 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb9065d6-3192-463a-bfb3-bda7753fefe2-workload-certs" pod "fb9065d6-3192-463a-bfb3-bda7753fefe2" (UID: "fb9065d6-3192-463a-bfb3-bda7753fefe2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:20.212749 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.212426 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d85ef9d1-c82a-4957-a041-6995e428f4a1-workload-socket" pod "d85ef9d1-c82a-4957-a041-6995e428f4a1" (UID: "d85ef9d1-c82a-4957-a041-6995e428f4a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:20.213213 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.212794 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d85ef9d1-c82a-4957-a041-6995e428f4a1-credential-socket" pod "d85ef9d1-c82a-4957-a041-6995e428f4a1" (UID: "d85ef9d1-c82a-4957-a041-6995e428f4a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:20.213213 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.213008 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d85ef9d1-c82a-4957-a041-6995e428f4a1-istiod-ca-cert" pod "d85ef9d1-c82a-4957-a041-6995e428f4a1" (UID: "d85ef9d1-c82a-4957-a041-6995e428f4a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:26:20.213549 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.213525 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb9065d6-3192-463a-bfb3-bda7753fefe2-istiod-ca-cert" pod "fb9065d6-3192-463a-bfb3-bda7753fefe2" (UID: "fb9065d6-3192-463a-bfb3-bda7753fefe2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:26:20.213871 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.213842 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d85ef9d1-c82a-4957-a041-6995e428f4a1-kube-api-access-278qx" pod "d85ef9d1-c82a-4957-a041-6995e428f4a1" (UID: "d85ef9d1-c82a-4957-a041-6995e428f4a1"). InnerVolumeSpecName "kube-api-access-278qx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:26:20.216688 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.216619 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d85ef9d1-c82a-4957-a041-6995e428f4a1-istio-token" pod "d85ef9d1-c82a-4957-a041-6995e428f4a1" (UID: "d85ef9d1-c82a-4957-a041-6995e428f4a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:26:20.222977 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.222951 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb9065d6-3192-463a-bfb3-bda7753fefe2-istio-token" pod "fb9065d6-3192-463a-bfb3-bda7753fefe2" (UID: "fb9065d6-3192-463a-bfb3-bda7753fefe2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:26:20.223104 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.223085 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb9065d6-3192-463a-bfb3-bda7753fefe2-kube-api-access-6f57l" pod "fb9065d6-3192-463a-bfb3-bda7753fefe2" (UID: "fb9065d6-3192-463a-bfb3-bda7753fefe2"). InnerVolumeSpecName "kube-api-access-6f57l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:26:20.223257 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.223228 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d85ef9d1-c82a-4957-a041-6995e428f4a1-istio-podinfo" pod "d85ef9d1-c82a-4957-a041-6995e428f4a1" (UID: "d85ef9d1-c82a-4957-a041-6995e428f4a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:26:20.224899 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.224879 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb9065d6-3192-463a-bfb3-bda7753fefe2-istio-envoy" pod "fb9065d6-3192-463a-bfb3-bda7753fefe2" (UID: "fb9065d6-3192-463a-bfb3-bda7753fefe2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:20.296270 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.296234 4982 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d85ef9d1-c82a-4957-a041-6995e428f4a1-istiod-ca-cert\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:26:20.296270 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.296268 4982 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d85ef9d1-c82a-4957-a041-6995e428f4a1-istio-token\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:26:20.296519 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.296280 4982 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85ef9d1-c82a-4957-a041-6995e428f4a1-credential-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:26:20.296519 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.296294 4982 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb9065d6-3192-463a-bfb3-bda7753fefe2-workload-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:26:20.296519 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.296315 4982 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb9065d6-3192-463a-bfb3-bda7753fefe2-credential-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:26:20.296519 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.296329 4982 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb9065d6-3192-463a-bfb3-bda7753fefe2-istio-token\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:26:20.296519 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.296352 4982 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb9065d6-3192-463a-bfb3-bda7753fefe2-istio-envoy\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:26:20.296519 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.296363 4982 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d85ef9d1-c82a-4957-a041-6995e428f4a1-istio-podinfo\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:26:20.296519 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.296374 4982 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d85ef9d1-c82a-4957-a041-6995e428f4a1-istio-envoy\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:26:20.296519 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.296387 4982 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb9065d6-3192-463a-bfb3-bda7753fefe2-workload-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:26:20.296519 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.296399 4982 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb9065d6-3192-463a-bfb3-bda7753fefe2-istio-podinfo\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:26:20.296519 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.296411 4982 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d85ef9d1-c82a-4957-a041-6995e428f4a1-workload-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:26:20.296519 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.296431 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-278qx\" (UniqueName: \"kubernetes.io/projected/d85ef9d1-c82a-4957-a041-6995e428f4a1-kube-api-access-278qx\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:26:20.296519 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.296443 4982 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb9065d6-3192-463a-bfb3-bda7753fefe2-istio-data\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:26:20.296519 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.296456 4982 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85ef9d1-c82a-4957-a041-6995e428f4a1-workload-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:26:20.296519 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.296469 4982 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb9065d6-3192-463a-bfb3-bda7753fefe2-istiod-ca-cert\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:26:20.296519 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.296484 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6f57l\" (UniqueName: \"kubernetes.io/projected/fb9065d6-3192-463a-bfb3-bda7753fefe2-kube-api-access-6f57l\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:26:20.934088 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.934053 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn" Oct 30 14:26:20.934088 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.934071 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn" event={"ID":"fb9065d6-3192-463a-bfb3-bda7753fefe2","Type":"ContainerDied","Data":"5d74b0f53a9eebaebe050ee045373f6f45a11a32817e55bbfdb034dc55029033"} Oct 30 14:26:20.934554 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.934126 4982 scope.go:122] "RemoveContainer" containerID="84d90f3062ee12186fa43cb855201bd800ded5f1f48baa3821ff9ac6284630bb" Oct 30 14:26:20.935795 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.935774 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5" Oct 30 14:26:20.935924 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.935792 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5" event={"ID":"d85ef9d1-c82a-4957-a041-6995e428f4a1","Type":"ContainerDied","Data":"3d39ea8214fb74b6dba10cd08918453ae72fb58c342a43a7e57e7d4a6481620d"} Oct 30 14:26:20.937351 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.937330 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm" event={"ID":"20041b11-f00c-4853-acf3-7c210f742f6d","Type":"ContainerDied","Data":"95768511b953bcf6a1005603a5e18d11454442d738eb85db178a457e968eedca"} Oct 30 14:26:20.937459 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.937357 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm" Oct 30 14:26:20.946938 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.946921 4982 scope.go:122] "RemoveContainer" containerID="c9e6f9a5bc9e3be4dba76192a17491b9b76115568e29499bf60b91ee0ce47016" Oct 30 14:26:20.957158 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.957127 4982 scope.go:122] "RemoveContainer" containerID="53034c6d7382e2872770a780f003f76a50435ecb7a012daab23a5c6c68c50249" Oct 30 14:26:20.994933 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:20.994904 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm"] Oct 30 14:26:21.007983 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:21.007952 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-t8rbm"] Oct 30 14:26:21.037920 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:21.037892 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn"] Oct 30 14:26:21.043912 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:21.043888 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-qp4zn"] Oct 30 14:26:21.082419 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:21.082384 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5"] Oct 30 14:26:21.090678 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:21.090624 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-tbpv5"] Oct 30 14:26:22.049951 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:22.049491 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:26:22.403120 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:22.403087 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20041b11-f00c-4853-acf3-7c210f742f6d" path="/var/lib/kubelet/pods/20041b11-f00c-4853-acf3-7c210f742f6d/volumes" Oct 30 14:26:22.403685 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:22.403665 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d85ef9d1-c82a-4957-a041-6995e428f4a1" path="/var/lib/kubelet/pods/d85ef9d1-c82a-4957-a041-6995e428f4a1/volumes" Oct 30 14:26:22.404108 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:22.404093 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb9065d6-3192-463a-bfb3-bda7753fefe2" path="/var/lib/kubelet/pods/fb9065d6-3192-463a-bfb3-bda7753fefe2/volumes" Oct 30 14:26:23.798369 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:23.798324 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5a9d7b-9bb744df9-2qrjh" podUID="241e6f6a-6c36-4487-b3ba-92941c524787" containerName="payload-pre-processing" containerID="cri-o://c0a244784c6e81328d58cd41e4ec2cddda2d79e0758be3a4cee0363107e3d249" gracePeriod=25 Oct 30 14:26:23.923942 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:23.923911 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:26:24.977796 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:24.977762 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5a9d7b-9bb744df9-2qrjh" podUID="241e6f6a-6c36-4487-b3ba-92941c524787" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:26:25.739595 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:25.739552 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:26:27.632574 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:27.632537 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:26:28.012807 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:26:28.012726 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:26:29.315084 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:29.315041 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:26:30.095788 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:30.095753 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:26:30.233572 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:30.233522 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:26:30.374348 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:30.374250 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:26:30.523167 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:30.523126 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:26:30.678234 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:30.678144 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:26:30.857570 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:30.857535 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:26:32.383950 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:32.383917 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:26:32.549453 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:32.549422 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-019e5b-75f68bd755-f22gh"] Oct 30 14:26:32.550066 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:32.550052 4982 cpu_manager.go:414] "containerMap: removing container" podUID="fb9065d6-3192-463a-bfb3-bda7753fefe2" containerName="istio-proxy" Oct 30 14:26:32.550113 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:32.550068 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb9065d6-3192-463a-bfb3-bda7753fefe2" containerName="istio-proxy" Oct 30 14:26:32.550113 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:32.550089 4982 cpu_manager.go:414] "containerMap: removing container" podUID="d85ef9d1-c82a-4957-a041-6995e428f4a1" containerName="istio-proxy" Oct 30 14:26:32.550113 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:32.550096 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d85ef9d1-c82a-4957-a041-6995e428f4a1" containerName="istio-proxy" Oct 30 14:26:32.550113 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:32.550110 4982 cpu_manager.go:414] "containerMap: removing container" podUID="20041b11-f00c-4853-acf3-7c210f742f6d" containerName="istio-proxy" Oct 30 14:26:32.550238 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:32.550116 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20041b11-f00c-4853-acf3-7c210f742f6d" containerName="istio-proxy" Oct 30 14:26:32.550238 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:32.550219 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb9065d6-3192-463a-bfb3-bda7753fefe2" containerName="istio-proxy" Oct 30 14:26:32.550238 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:32.550228 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="d85ef9d1-c82a-4957-a041-6995e428f4a1" containerName="istio-proxy" Oct 30 14:26:32.550238 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:32.550235 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="20041b11-f00c-4853-acf3-7c210f742f6d" containerName="istio-proxy" Oct 30 14:26:32.555491 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:32.555471 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-019e5b-75f68bd755-f22gh" Oct 30 14:26:32.573606 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:32.573583 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:26:32.577511 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:32.577484 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-019e5b-75f68bd755-f22gh"] Oct 30 14:26:32.585551 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:32.585529 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-019e5b\"" Oct 30 14:26:32.585951 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:32.585912 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-019e5b-dockercfg-r6m5t\"" Oct 30 14:26:32.708711 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:32.704916 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xkm4\" (UniqueName: \"kubernetes.io/projected/60275a20-4dba-4da0-8aff-5150df7abb02-kube-api-access-4xkm4\") pod \"payload-pre-processing-e2e-inf-a-019e5b-75f68bd755-f22gh\" (UID: \"60275a20-4dba-4da0-8aff-5150df7abb02\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-019e5b-75f68bd755-f22gh" Oct 30 14:26:32.708711 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:32.705002 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60275a20-4dba-4da0-8aff-5150df7abb02-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-019e5b-75f68bd755-f22gh\" (UID: \"60275a20-4dba-4da0-8aff-5150df7abb02\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-019e5b-75f68bd755-f22gh" Oct 30 14:26:32.806373 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:32.806327 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60275a20-4dba-4da0-8aff-5150df7abb02-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-019e5b-75f68bd755-f22gh\" (UID: \"60275a20-4dba-4da0-8aff-5150df7abb02\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-019e5b-75f68bd755-f22gh" Oct 30 14:26:32.806695 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:32.806673 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xkm4\" (UniqueName: \"kubernetes.io/projected/60275a20-4dba-4da0-8aff-5150df7abb02-kube-api-access-4xkm4\") pod \"payload-pre-processing-e2e-inf-a-019e5b-75f68bd755-f22gh\" (UID: \"60275a20-4dba-4da0-8aff-5150df7abb02\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-019e5b-75f68bd755-f22gh" Oct 30 14:26:32.807896 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:32.807869 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60275a20-4dba-4da0-8aff-5150df7abb02-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-019e5b-75f68bd755-f22gh\" (UID: \"60275a20-4dba-4da0-8aff-5150df7abb02\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-019e5b-75f68bd755-f22gh" Oct 30 14:26:32.832849 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:32.832784 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xkm4\" (UniqueName: \"kubernetes.io/projected/60275a20-4dba-4da0-8aff-5150df7abb02-kube-api-access-4xkm4\") pod \"payload-pre-processing-e2e-inf-a-019e5b-75f68bd755-f22gh\" (UID: \"60275a20-4dba-4da0-8aff-5150df7abb02\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-019e5b-75f68bd755-f22gh" Oct 30 14:26:32.868800 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:32.868761 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-019e5b-75f68bd755-f22gh" Oct 30 14:26:33.177602 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:26:33.177570 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60275a20_4dba_4da0_8aff_5150df7abb02.slice/crio-c6022ff8241ff8a20bfd01c0910e99f598ef4bd9e78cbb7a5b0d1183c2b36546 WatchSource:0}: Error finding container c6022ff8241ff8a20bfd01c0910e99f598ef4bd9e78cbb7a5b0d1183c2b36546: Status 404 returned error can't find the container with id c6022ff8241ff8a20bfd01c0910e99f598ef4bd9e78cbb7a5b0d1183c2b36546 Oct 30 14:26:33.179301 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:33.179275 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-019e5b-75f68bd755-f22gh"] Oct 30 14:26:34.072471 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:34.072233 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-019e5b-75f68bd755-f22gh" event={"ID":"60275a20-4dba-4da0-8aff-5150df7abb02","Type":"ContainerStarted","Data":"d7d145c412ee52018e2e88fa760757eb6a777e42fbda20876a823f4fde4970d5"} Oct 30 14:26:34.072471 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:34.072317 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-019e5b-75f68bd755-f22gh" event={"ID":"60275a20-4dba-4da0-8aff-5150df7abb02","Type":"ContainerStarted","Data":"c6022ff8241ff8a20bfd01c0910e99f598ef4bd9e78cbb7a5b0d1183c2b36546"} Oct 30 14:26:34.073764 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:34.073350 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-019e5b-75f68bd755-f22gh" Oct 30 14:26:34.126394 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:34.126327 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-019e5b-75f68bd755-f22gh" podStartSLOduration=2.126312178 podStartE2EDuration="2.126312178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:26:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:26:34.117847759 +0000 UTC m=+2263.861981801" watchObservedRunningTime="2026-10-30 14:26:34.126312178 +0000 UTC m=+2263.870446219" Oct 30 14:26:34.655366 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:34.655330 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:26:34.979020 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:34.978919 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5a9d7b-9bb744df9-2qrjh" podUID="241e6f6a-6c36-4487-b3ba-92941c524787" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:26:36.497116 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:36.497070 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:26:37.263707 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:37.263593 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:26:37.403685 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:37.403627 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:26:38.056252 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:38.056214 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:26:38.216330 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:38.216296 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:26:38.366410 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:38.366360 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:26:38.522386 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:38.522351 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:26:38.910811 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:38.910776 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ca02c9-7rvdk"] Oct 30 14:26:38.915616 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:38.915593 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ca02c9-7rvdk" Oct 30 14:26:38.935384 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:38.935356 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ca02c9-7rvdk"] Oct 30 14:26:38.957696 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:38.956951 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:26:38.957696 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:38.957077 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:26:38.994415 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:38.994381 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbs4l\" (UniqueName: \"kubernetes.io/projected/90a4f855-40da-44f6-b482-647247fbee35-kube-api-access-nbs4l\") pod \"maas-api-revoke-keys-e2e-inf-b-ca02c9-7rvdk\" (UID: \"90a4f855-40da-44f6-b482-647247fbee35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ca02c9-7rvdk" Oct 30 14:26:38.994559 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:38.994454 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90a4f855-40da-44f6-b482-647247fbee35-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ca02c9-7rvdk\" (UID: \"90a4f855-40da-44f6-b482-647247fbee35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ca02c9-7rvdk" Oct 30 14:26:38.994617 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:38.994585 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90a4f855-40da-44f6-b482-647247fbee35-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ca02c9-7rvdk\" (UID: \"90a4f855-40da-44f6-b482-647247fbee35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ca02c9-7rvdk" Oct 30 14:26:39.096038 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:39.095993 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbs4l\" (UniqueName: \"kubernetes.io/projected/90a4f855-40da-44f6-b482-647247fbee35-kube-api-access-nbs4l\") pod \"maas-api-revoke-keys-e2e-inf-b-ca02c9-7rvdk\" (UID: \"90a4f855-40da-44f6-b482-647247fbee35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ca02c9-7rvdk" Oct 30 14:26:39.096566 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:39.096120 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90a4f855-40da-44f6-b482-647247fbee35-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ca02c9-7rvdk\" (UID: \"90a4f855-40da-44f6-b482-647247fbee35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ca02c9-7rvdk" Oct 30 14:26:39.096566 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:39.096188 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90a4f855-40da-44f6-b482-647247fbee35-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ca02c9-7rvdk\" (UID: \"90a4f855-40da-44f6-b482-647247fbee35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ca02c9-7rvdk" Oct 30 14:26:39.097002 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:39.096977 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90a4f855-40da-44f6-b482-647247fbee35-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ca02c9-7rvdk\" (UID: \"90a4f855-40da-44f6-b482-647247fbee35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ca02c9-7rvdk" Oct 30 14:26:39.098555 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:39.098521 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90a4f855-40da-44f6-b482-647247fbee35-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ca02c9-7rvdk\" (UID: \"90a4f855-40da-44f6-b482-647247fbee35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ca02c9-7rvdk" Oct 30 14:26:39.121957 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:39.121929 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbs4l\" (UniqueName: \"kubernetes.io/projected/90a4f855-40da-44f6-b482-647247fbee35-kube-api-access-nbs4l\") pod \"maas-api-revoke-keys-e2e-inf-b-ca02c9-7rvdk\" (UID: \"90a4f855-40da-44f6-b482-647247fbee35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ca02c9-7rvdk" Oct 30 14:26:39.229688 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:39.229598 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ca02c9-7rvdk" Oct 30 14:26:39.536328 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:26:39.536292 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90a4f855_40da_44f6_b482_647247fbee35.slice/crio-f0c8a82d1b74d881be4e79675c08be73f1146745583fbe3fde318d8cfd8a0d86 WatchSource:0}: Error finding container f0c8a82d1b74d881be4e79675c08be73f1146745583fbe3fde318d8cfd8a0d86: Status 404 returned error can't find the container with id f0c8a82d1b74d881be4e79675c08be73f1146745583fbe3fde318d8cfd8a0d86 Oct 30 14:26:39.538063 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:39.538036 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ca02c9-7rvdk"] Oct 30 14:26:40.139711 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:40.139677 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ca02c9-7rvdk" event={"ID":"90a4f855-40da-44f6-b482-647247fbee35","Type":"ContainerStarted","Data":"961d8f130b9ada713e709c06b8e5f07b7a67cdb374b7b5233dafce862afcc431"} Oct 30 14:26:40.140125 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:40.139748 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ca02c9-7rvdk" event={"ID":"90a4f855-40da-44f6-b482-647247fbee35","Type":"ContainerStarted","Data":"f0c8a82d1b74d881be4e79675c08be73f1146745583fbe3fde318d8cfd8a0d86"} Oct 30 14:26:40.187335 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:40.187285 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ca02c9-7rvdk" podStartSLOduration=2.187270477 podStartE2EDuration="2.187270477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:26:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:26:40.183909015 +0000 UTC m=+2269.928043055" watchObservedRunningTime="2026-10-30 14:26:40.187270477 +0000 UTC m=+2269.931404574" Oct 30 14:26:40.257114 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:40.257081 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:26:41.157892 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:41.157848 4982 generic.go:356] "Generic (PLEG): container finished" podID="90a4f855-40da-44f6-b482-647247fbee35" containerID="961d8f130b9ada713e709c06b8e5f07b7a67cdb374b7b5233dafce862afcc431" exitCode=0 Oct 30 14:26:41.158294 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:41.157938 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ca02c9-7rvdk" event={"ID":"90a4f855-40da-44f6-b482-647247fbee35","Type":"ContainerDied","Data":"961d8f130b9ada713e709c06b8e5f07b7a67cdb374b7b5233dafce862afcc431"} Oct 30 14:26:42.129978 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:42.129943 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:26:42.353766 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:42.353742 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ca02c9-7rvdk" Oct 30 14:26:42.450901 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:42.450818 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90a4f855-40da-44f6-b482-647247fbee35-kube-api-access-nbs4l\" (UniqueName: \"kubernetes.io/projected/90a4f855-40da-44f6-b482-647247fbee35-kube-api-access-nbs4l\") pod \"90a4f855-40da-44f6-b482-647247fbee35\" (UID: \"90a4f855-40da-44f6-b482-647247fbee35\") " Oct 30 14:26:42.451065 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:42.450942 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90a4f855-40da-44f6-b482-647247fbee35-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90a4f855-40da-44f6-b482-647247fbee35-maas-api-xks-ca\") pod \"90a4f855-40da-44f6-b482-647247fbee35\" (UID: \"90a4f855-40da-44f6-b482-647247fbee35\") " Oct 30 14:26:42.451164 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:42.451148 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90a4f855-40da-44f6-b482-647247fbee35-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90a4f855-40da-44f6-b482-647247fbee35-maas-api-openshift-ca\") pod \"90a4f855-40da-44f6-b482-647247fbee35\" (UID: \"90a4f855-40da-44f6-b482-647247fbee35\") " Oct 30 14:26:42.451521 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:42.451489 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90a4f855-40da-44f6-b482-647247fbee35-maas-api-openshift-ca" pod "90a4f855-40da-44f6-b482-647247fbee35" (UID: "90a4f855-40da-44f6-b482-647247fbee35"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:26:42.452116 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:42.452090 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90a4f855-40da-44f6-b482-647247fbee35-maas-api-openshift-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:26:42.453232 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:42.453209 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90a4f855-40da-44f6-b482-647247fbee35-maas-api-xks-ca" pod "90a4f855-40da-44f6-b482-647247fbee35" (UID: "90a4f855-40da-44f6-b482-647247fbee35"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:26:42.453307 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:42.453257 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90a4f855-40da-44f6-b482-647247fbee35-kube-api-access-nbs4l" pod "90a4f855-40da-44f6-b482-647247fbee35" (UID: "90a4f855-40da-44f6-b482-647247fbee35"). InnerVolumeSpecName "kube-api-access-nbs4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:26:42.553902 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:42.553866 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbs4l\" (UniqueName: \"kubernetes.io/projected/90a4f855-40da-44f6-b482-647247fbee35-kube-api-access-nbs4l\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:26:42.553902 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:42.553896 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90a4f855-40da-44f6-b482-647247fbee35-maas-api-xks-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:26:43.035138 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:43.035103 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dd54ca2-58vcb"] Oct 30 14:26:43.048200 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:43.048171 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dd54ca2-58vcb"] Oct 30 14:26:43.141538 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:43.141505 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:26:43.193759 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:43.193728 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ca02c9-7rvdk" Oct 30 14:26:43.193964 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:43.193800 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ca02c9-7rvdk" event={"ID":"90a4f855-40da-44f6-b482-647247fbee35","Type":"ContainerDied","Data":"f0c8a82d1b74d881be4e79675c08be73f1146745583fbe3fde318d8cfd8a0d86"} Oct 30 14:26:43.193964 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:43.193843 4982 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0c8a82d1b74d881be4e79675c08be73f1146745583fbe3fde318d8cfd8a0d86" Oct 30 14:26:43.341991 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:43.341956 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml"] Oct 30 14:26:43.342185 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:43.342161 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml" podUID="a6687b0d-4a6b-4fe8-9c62-47057042db79" containerName="istio-proxy" containerID="cri-o://f28c75d166598ecf32a358d61a8bcaf00ed385e2d78c966ace93a5c61b896556" gracePeriod=30 Oct 30 14:26:43.954882 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:43.954846 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:26:44.168707 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:44.168676 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5a9d7b-9bb744df9-2qrjh" Oct 30 14:26:44.203926 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:44.203895 4982 generic.go:356] "Generic (PLEG): container finished" podID="241e6f6a-6c36-4487-b3ba-92941c524787" containerID="c0a244784c6e81328d58cd41e4ec2cddda2d79e0758be3a4cee0363107e3d249" exitCode=0 Oct 30 14:26:44.204117 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:44.203951 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5a9d7b-9bb744df9-2qrjh" Oct 30 14:26:44.204117 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:44.204003 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5a9d7b-9bb744df9-2qrjh" event={"ID":"241e6f6a-6c36-4487-b3ba-92941c524787","Type":"ContainerDied","Data":"c0a244784c6e81328d58cd41e4ec2cddda2d79e0758be3a4cee0363107e3d249"} Oct 30 14:26:44.204117 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:44.204044 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5a9d7b-9bb744df9-2qrjh" event={"ID":"241e6f6a-6c36-4487-b3ba-92941c524787","Type":"ContainerDied","Data":"e7fced250406957ab624861619257af1bd4e64ae17058484a26f5a76ca632269"} Oct 30 14:26:44.204117 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:44.204070 4982 scope.go:122] "RemoveContainer" containerID="c0a244784c6e81328d58cd41e4ec2cddda2d79e0758be3a4cee0363107e3d249" Oct 30 14:26:44.214775 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:44.214695 4982 scope.go:122] "RemoveContainer" containerID="c0a244784c6e81328d58cd41e4ec2cddda2d79e0758be3a4cee0363107e3d249" Oct 30 14:26:44.215003 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:26:44.214980 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0a244784c6e81328d58cd41e4ec2cddda2d79e0758be3a4cee0363107e3d249\": container with ID starting with c0a244784c6e81328d58cd41e4ec2cddda2d79e0758be3a4cee0363107e3d249 not found: ID does not exist" containerID="c0a244784c6e81328d58cd41e4ec2cddda2d79e0758be3a4cee0363107e3d249" Oct 30 14:26:44.215085 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:44.215015 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0a244784c6e81328d58cd41e4ec2cddda2d79e0758be3a4cee0363107e3d249"} err="failed to get container status \"c0a244784c6e81328d58cd41e4ec2cddda2d79e0758be3a4cee0363107e3d249\": rpc error: code = NotFound desc = could not find container \"c0a244784c6e81328d58cd41e4ec2cddda2d79e0758be3a4cee0363107e3d249\": container with ID starting with c0a244784c6e81328d58cd41e4ec2cddda2d79e0758be3a4cee0363107e3d249 not found: ID does not exist" Oct 30 14:26:44.278543 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:44.278514 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/241e6f6a-6c36-4487-b3ba-92941c524787-kube-api-access-rw87g\" (UniqueName: \"kubernetes.io/projected/241e6f6a-6c36-4487-b3ba-92941c524787-kube-api-access-rw87g\") pod \"241e6f6a-6c36-4487-b3ba-92941c524787\" (UID: \"241e6f6a-6c36-4487-b3ba-92941c524787\") " Oct 30 14:26:44.278796 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:44.278781 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/241e6f6a-6c36-4487-b3ba-92941c524787-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/241e6f6a-6c36-4487-b3ba-92941c524787-plugins-config-volume\") pod \"241e6f6a-6c36-4487-b3ba-92941c524787\" (UID: \"241e6f6a-6c36-4487-b3ba-92941c524787\") " Oct 30 14:26:44.279221 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:44.279187 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/241e6f6a-6c36-4487-b3ba-92941c524787-plugins-config-volume" pod "241e6f6a-6c36-4487-b3ba-92941c524787" (UID: "241e6f6a-6c36-4487-b3ba-92941c524787"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:26:44.282898 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:44.281227 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/241e6f6a-6c36-4487-b3ba-92941c524787-kube-api-access-rw87g" pod "241e6f6a-6c36-4487-b3ba-92941c524787" (UID: "241e6f6a-6c36-4487-b3ba-92941c524787"). InnerVolumeSpecName "kube-api-access-rw87g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:26:44.380809 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:44.380757 4982 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/241e6f6a-6c36-4487-b3ba-92941c524787-plugins-config-volume\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:26:44.380809 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:44.380804 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rw87g\" (UniqueName: \"kubernetes.io/projected/241e6f6a-6c36-4487-b3ba-92941c524787-kube-api-access-rw87g\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:26:44.403008 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:44.402924 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ca3d212-3272-406a-9c00-d3968904a19c" path="/var/lib/kubelet/pods/0ca3d212-3272-406a-9c00-d3968904a19c/volumes" Oct 30 14:26:44.549668 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:44.549620 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5a9d7b-9bb744df9-2qrjh"] Oct 30 14:26:44.562654 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:44.562605 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5a9d7b-9bb744df9-2qrjh"] Oct 30 14:26:44.815527 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:44.815484 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:26:45.010035 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:45.009983 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:26:45.091717 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:45.090757 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-019e5b-75f68bd755-f22gh" Oct 30 14:26:46.056661 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:46.054517 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:26:46.405622 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:46.405524 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="241e6f6a-6c36-4487-b3ba-92941c524787" path="/var/lib/kubelet/pods/241e6f6a-6c36-4487-b3ba-92941c524787/volumes" Oct 30 14:26:47.078552 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:47.078513 4982 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 14:26:47.078951 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:47.078588 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml" podUID="a6687b0d-4a6b-4fe8-9c62-47057042db79" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 14:26:47.819335 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:47.819294 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:26:48.633573 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:48.633544 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml" Oct 30 14:26:48.656510 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:48.656480 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6687b0d-4a6b-4fe8-9c62-47057042db79-istio-token\" (UniqueName: \"kubernetes.io/projected/a6687b0d-4a6b-4fe8-9c62-47057042db79-istio-token\") pod \"a6687b0d-4a6b-4fe8-9c62-47057042db79\" (UID: \"a6687b0d-4a6b-4fe8-9c62-47057042db79\") " Oct 30 14:26:48.656664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:48.656590 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6687b0d-4a6b-4fe8-9c62-47057042db79-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6687b0d-4a6b-4fe8-9c62-47057042db79-istiod-ca-cert\") pod \"a6687b0d-4a6b-4fe8-9c62-47057042db79\" (UID: \"a6687b0d-4a6b-4fe8-9c62-47057042db79\") " Oct 30 14:26:48.656664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:48.656631 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6687b0d-4a6b-4fe8-9c62-47057042db79-kube-api-access-qjdkt\" (UniqueName: \"kubernetes.io/projected/a6687b0d-4a6b-4fe8-9c62-47057042db79-kube-api-access-qjdkt\") pod \"a6687b0d-4a6b-4fe8-9c62-47057042db79\" (UID: \"a6687b0d-4a6b-4fe8-9c62-47057042db79\") " Oct 30 14:26:48.656750 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:48.656682 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6687b0d-4a6b-4fe8-9c62-47057042db79-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6687b0d-4a6b-4fe8-9c62-47057042db79-credential-socket\") pod \"a6687b0d-4a6b-4fe8-9c62-47057042db79\" (UID: \"a6687b0d-4a6b-4fe8-9c62-47057042db79\") " Oct 30 14:26:48.656750 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:48.656711 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6687b0d-4a6b-4fe8-9c62-47057042db79-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6687b0d-4a6b-4fe8-9c62-47057042db79-istio-envoy\") pod \"a6687b0d-4a6b-4fe8-9c62-47057042db79\" (UID: \"a6687b0d-4a6b-4fe8-9c62-47057042db79\") " Oct 30 14:26:48.656848 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:48.656756 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6687b0d-4a6b-4fe8-9c62-47057042db79-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6687b0d-4a6b-4fe8-9c62-47057042db79-workload-socket\") pod \"a6687b0d-4a6b-4fe8-9c62-47057042db79\" (UID: \"a6687b0d-4a6b-4fe8-9c62-47057042db79\") " Oct 30 14:26:48.656848 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:48.656783 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6687b0d-4a6b-4fe8-9c62-47057042db79-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6687b0d-4a6b-4fe8-9c62-47057042db79-istio-data\") pod \"a6687b0d-4a6b-4fe8-9c62-47057042db79\" (UID: \"a6687b0d-4a6b-4fe8-9c62-47057042db79\") " Oct 30 14:26:48.656848 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:48.656829 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a6687b0d-4a6b-4fe8-9c62-47057042db79-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6687b0d-4a6b-4fe8-9c62-47057042db79-istio-podinfo\") pod \"a6687b0d-4a6b-4fe8-9c62-47057042db79\" (UID: \"a6687b0d-4a6b-4fe8-9c62-47057042db79\") " Oct 30 14:26:48.657000 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:48.656862 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6687b0d-4a6b-4fe8-9c62-47057042db79-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6687b0d-4a6b-4fe8-9c62-47057042db79-workload-certs\") pod \"a6687b0d-4a6b-4fe8-9c62-47057042db79\" (UID: \"a6687b0d-4a6b-4fe8-9c62-47057042db79\") " Oct 30 14:26:48.657064 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:48.657011 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6687b0d-4a6b-4fe8-9c62-47057042db79-credential-socket" pod "a6687b0d-4a6b-4fe8-9c62-47057042db79" (UID: "a6687b0d-4a6b-4fe8-9c62-47057042db79"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:48.657064 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:48.657018 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6687b0d-4a6b-4fe8-9c62-47057042db79-istiod-ca-cert" pod "a6687b0d-4a6b-4fe8-9c62-47057042db79" (UID: "a6687b0d-4a6b-4fe8-9c62-47057042db79"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:26:48.657185 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:48.657161 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6687b0d-4a6b-4fe8-9c62-47057042db79-istio-data" pod "a6687b0d-4a6b-4fe8-9c62-47057042db79" (UID: "a6687b0d-4a6b-4fe8-9c62-47057042db79"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:48.657327 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:48.657300 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6687b0d-4a6b-4fe8-9c62-47057042db79-workload-certs" pod "a6687b0d-4a6b-4fe8-9c62-47057042db79" (UID: "a6687b0d-4a6b-4fe8-9c62-47057042db79"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:48.657327 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:48.657315 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6687b0d-4a6b-4fe8-9c62-47057042db79-workload-socket" pod "a6687b0d-4a6b-4fe8-9c62-47057042db79" (UID: "a6687b0d-4a6b-4fe8-9c62-47057042db79"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:48.657529 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:48.657407 4982 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6687b0d-4a6b-4fe8-9c62-47057042db79-istiod-ca-cert\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:26:48.657529 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:48.657425 4982 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6687b0d-4a6b-4fe8-9c62-47057042db79-credential-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:26:48.657529 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:48.657438 4982 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6687b0d-4a6b-4fe8-9c62-47057042db79-workload-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:26:48.657529 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:48.657446 4982 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6687b0d-4a6b-4fe8-9c62-47057042db79-istio-data\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:26:48.657529 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:48.657454 4982 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6687b0d-4a6b-4fe8-9c62-47057042db79-workload-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:26:48.659052 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:48.659028 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a6687b0d-4a6b-4fe8-9c62-47057042db79-istio-podinfo" pod "a6687b0d-4a6b-4fe8-9c62-47057042db79" (UID: "a6687b0d-4a6b-4fe8-9c62-47057042db79"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:26:48.659143 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:48.659087 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6687b0d-4a6b-4fe8-9c62-47057042db79-istio-envoy" pod "a6687b0d-4a6b-4fe8-9c62-47057042db79" (UID: "a6687b0d-4a6b-4fe8-9c62-47057042db79"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:48.659290 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:48.659268 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6687b0d-4a6b-4fe8-9c62-47057042db79-kube-api-access-qjdkt" pod "a6687b0d-4a6b-4fe8-9c62-47057042db79" (UID: "a6687b0d-4a6b-4fe8-9c62-47057042db79"). InnerVolumeSpecName "kube-api-access-qjdkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:26:48.659328 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:48.659305 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6687b0d-4a6b-4fe8-9c62-47057042db79-istio-token" pod "a6687b0d-4a6b-4fe8-9c62-47057042db79" (UID: "a6687b0d-4a6b-4fe8-9c62-47057042db79"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:26:48.759047 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:48.759017 4982 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6687b0d-4a6b-4fe8-9c62-47057042db79-istio-podinfo\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:26:48.759047 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:48.759045 4982 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6687b0d-4a6b-4fe8-9c62-47057042db79-istio-token\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:26:48.759233 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:48.759056 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjdkt\" (UniqueName: \"kubernetes.io/projected/a6687b0d-4a6b-4fe8-9c62-47057042db79-kube-api-access-qjdkt\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:26:48.759233 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:48.759066 4982 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6687b0d-4a6b-4fe8-9c62-47057042db79-istio-envoy\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:26:49.357715 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:49.343946 4982 generic.go:356] "Generic (PLEG): container finished" podID="a6687b0d-4a6b-4fe8-9c62-47057042db79" containerID="f28c75d166598ecf32a358d61a8bcaf00ed385e2d78c966ace93a5c61b896556" exitCode=0 Oct 30 14:26:49.357715 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:49.344111 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml" event={"ID":"a6687b0d-4a6b-4fe8-9c62-47057042db79","Type":"ContainerDied","Data":"f28c75d166598ecf32a358d61a8bcaf00ed385e2d78c966ace93a5c61b896556"} Oct 30 14:26:49.357715 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:49.344167 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml" event={"ID":"a6687b0d-4a6b-4fe8-9c62-47057042db79","Type":"ContainerDied","Data":"9aeeaf69c10a8aa26ef60c3924725c9c2cf671800963ded9a5a9f9cadd272aa1"} Oct 30 14:26:49.357715 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:49.344199 4982 scope.go:122] "RemoveContainer" containerID="f28c75d166598ecf32a358d61a8bcaf00ed385e2d78c966ace93a5c61b896556" Oct 30 14:26:49.357715 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:49.344424 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml" Oct 30 14:26:49.404096 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:49.403804 4982 scope.go:122] "RemoveContainer" containerID="f28c75d166598ecf32a358d61a8bcaf00ed385e2d78c966ace93a5c61b896556" Oct 30 14:26:49.404882 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:26:49.404292 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f28c75d166598ecf32a358d61a8bcaf00ed385e2d78c966ace93a5c61b896556\": container with ID starting with f28c75d166598ecf32a358d61a8bcaf00ed385e2d78c966ace93a5c61b896556 not found: ID does not exist" containerID="f28c75d166598ecf32a358d61a8bcaf00ed385e2d78c966ace93a5c61b896556" Oct 30 14:26:49.404882 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:49.404338 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f28c75d166598ecf32a358d61a8bcaf00ed385e2d78c966ace93a5c61b896556"} err="failed to get container status \"f28c75d166598ecf32a358d61a8bcaf00ed385e2d78c966ace93a5c61b896556\": rpc error: code = NotFound desc = could not find container \"f28c75d166598ecf32a358d61a8bcaf00ed385e2d78c966ace93a5c61b896556\": container with ID starting with f28c75d166598ecf32a358d61a8bcaf00ed385e2d78c966ace93a5c61b896556 not found: ID does not exist" Oct 30 14:26:49.407269 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:49.407245 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ca02c9-7fc799d5c8-pkcxp"] Oct 30 14:26:49.431215 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:49.431184 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml"] Oct 30 14:26:49.439260 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:49.439232 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-k5gml"] Oct 30 14:26:49.585694 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:49.585651 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:26:49.744441 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:49.744353 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:26:49.891829 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:49.891795 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:26:50.481564 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:50.481530 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6687b0d-4a6b-4fe8-9c62-47057042db79" path="/var/lib/kubelet/pods/a6687b0d-4a6b-4fe8-9c62-47057042db79/volumes" Oct 30 14:26:51.027296 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:51.027259 4982 scope.go:122] "RemoveContainer" containerID="f65febe65e870197362445c4cbbe79159f4fb58d539c6c3efb7e4d1e5d8893c5" Oct 30 14:26:51.039270 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:51.039253 4982 scope.go:122] "RemoveContainer" containerID="f74cdb1385614e779f8d24ec813c24d43d481a53b994a46a5e5a94344c3909ba" Oct 30 14:26:51.048402 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:51.048381 4982 scope.go:122] "RemoveContainer" containerID="62db31b6e442c83835af08227ef58d9af15774b093c8b5730cdd98172dd4d50b" Oct 30 14:26:51.936223 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:51.936180 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:26:53.825120 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:53.824773 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:26:54.376313 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:54.376280 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:26:54.407974 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:54.407935 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-ca02c9-7fc799d5c8-pkcxp" podUID="8c680f8b-331a-4286-a587-300ca1429de7" containerName="payload-processing" containerID="cri-o://65e61c20034e94e091691c72a2c81c5e22644ea4f70d9bc2f8bb390249ae8deb" gracePeriod=25 Oct 30 14:26:54.760813 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:54.760779 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ca02c9-7fc799d5c8-pkcxp" Oct 30 14:26:54.821026 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:54.820982 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c680f8b-331a-4286-a587-300ca1429de7-kube-api-access-7thm4\" (UniqueName: \"kubernetes.io/projected/8c680f8b-331a-4286-a587-300ca1429de7-kube-api-access-7thm4\") pod \"8c680f8b-331a-4286-a587-300ca1429de7\" (UID: \"8c680f8b-331a-4286-a587-300ca1429de7\") " Oct 30 14:26:54.821236 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:54.821104 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c680f8b-331a-4286-a587-300ca1429de7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c680f8b-331a-4286-a587-300ca1429de7-plugins-config-volume\") pod \"8c680f8b-331a-4286-a587-300ca1429de7\" (UID: \"8c680f8b-331a-4286-a587-300ca1429de7\") " Oct 30 14:26:54.823109 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:54.822914 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c680f8b-331a-4286-a587-300ca1429de7-plugins-config-volume" pod "8c680f8b-331a-4286-a587-300ca1429de7" (UID: "8c680f8b-331a-4286-a587-300ca1429de7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:26:54.825180 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:54.825103 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c680f8b-331a-4286-a587-300ca1429de7-kube-api-access-7thm4" pod "8c680f8b-331a-4286-a587-300ca1429de7" (UID: "8c680f8b-331a-4286-a587-300ca1429de7"). InnerVolumeSpecName "kube-api-access-7thm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:26:54.924303 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:54.924205 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7thm4\" (UniqueName: \"kubernetes.io/projected/8c680f8b-331a-4286-a587-300ca1429de7-kube-api-access-7thm4\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:26:54.924303 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:54.924236 4982 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c680f8b-331a-4286-a587-300ca1429de7-plugins-config-volume\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:26:55.444501 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:55.444467 4982 generic.go:356] "Generic (PLEG): container finished" podID="8c680f8b-331a-4286-a587-300ca1429de7" containerID="65e61c20034e94e091691c72a2c81c5e22644ea4f70d9bc2f8bb390249ae8deb" exitCode=0 Oct 30 14:26:55.444710 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:55.444513 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ca02c9-7fc799d5c8-pkcxp" Oct 30 14:26:55.444710 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:55.444511 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ca02c9-7fc799d5c8-pkcxp" event={"ID":"8c680f8b-331a-4286-a587-300ca1429de7","Type":"ContainerDied","Data":"65e61c20034e94e091691c72a2c81c5e22644ea4f70d9bc2f8bb390249ae8deb"} Oct 30 14:26:55.444710 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:55.444678 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ca02c9-7fc799d5c8-pkcxp" event={"ID":"8c680f8b-331a-4286-a587-300ca1429de7","Type":"ContainerDied","Data":"31d31bb49f0ebcd85b2c2a1d9a344ea91e8b34d57f4cf0f65deee7abe1ff3469"} Oct 30 14:26:55.444886 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:55.444713 4982 scope.go:122] "RemoveContainer" containerID="65e61c20034e94e091691c72a2c81c5e22644ea4f70d9bc2f8bb390249ae8deb" Oct 30 14:26:55.455587 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:55.455572 4982 scope.go:122] "RemoveContainer" containerID="65e61c20034e94e091691c72a2c81c5e22644ea4f70d9bc2f8bb390249ae8deb" Oct 30 14:26:55.455855 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:26:55.455838 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65e61c20034e94e091691c72a2c81c5e22644ea4f70d9bc2f8bb390249ae8deb\": container with ID starting with 65e61c20034e94e091691c72a2c81c5e22644ea4f70d9bc2f8bb390249ae8deb not found: ID does not exist" containerID="65e61c20034e94e091691c72a2c81c5e22644ea4f70d9bc2f8bb390249ae8deb" Oct 30 14:26:55.455901 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:55.455865 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65e61c20034e94e091691c72a2c81c5e22644ea4f70d9bc2f8bb390249ae8deb"} err="failed to get container status \"65e61c20034e94e091691c72a2c81c5e22644ea4f70d9bc2f8bb390249ae8deb\": rpc error: code = NotFound desc = could not find container \"65e61c20034e94e091691c72a2c81c5e22644ea4f70d9bc2f8bb390249ae8deb\": container with ID starting with 65e61c20034e94e091691c72a2c81c5e22644ea4f70d9bc2f8bb390249ae8deb not found: ID does not exist" Oct 30 14:26:55.476417 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:55.476375 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:26:55.514719 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:55.513474 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ca02c9-7fc799d5c8-pkcxp"] Oct 30 14:26:55.520089 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:55.520034 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ca02c9-7fc799d5c8-pkcxp"] Oct 30 14:26:55.627256 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:55.627197 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:26:55.826867 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:55.826827 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:26:56.404527 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:56.404434 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c680f8b-331a-4286-a587-300ca1429de7" path="/var/lib/kubelet/pods/8c680f8b-331a-4286-a587-300ca1429de7/volumes" Oct 30 14:26:57.562663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:57.561268 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:26:58.015541 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:26:58.015449 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:26:59.354370 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:26:59.354335 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:27:00.499668 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:00.499601 4982 generic.go:356] "Generic (PLEG): container finished" podID="d57125b3-c6b9-4d3b-b04c-7f6b30094325" containerID="d2f5f4d9ec23ca09019f62b706839660f2af42d297cc465750ba89ed7c052cd8" exitCode=0 Oct 30 14:27:00.500220 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:00.499814 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh" event={"ID":"d57125b3-c6b9-4d3b-b04c-7f6b30094325","Type":"ContainerDied","Data":"d2f5f4d9ec23ca09019f62b706839660f2af42d297cc465750ba89ed7c052cd8"} Oct 30 14:27:00.500290 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:00.500263 4982 scope.go:122] "RemoveContainer" containerID="d2f5f4d9ec23ca09019f62b706839660f2af42d297cc465750ba89ed7c052cd8" Oct 30 14:27:00.502412 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:00.502374 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:27:00.502504 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:00.502463 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:27:00.502561 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:00.502511 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:27:01.249953 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:01.249894 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:27:01.520899 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:01.520810 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh" event={"ID":"d57125b3-c6b9-4d3b-b04c-7f6b30094325","Type":"ContainerStarted","Data":"fadf1c5e2f38abe61af6385309baa37457b191d90d279076e79beb34a1c5b9d1"} Oct 30 14:27:01.812762 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:01.812720 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh" Oct 30 14:27:01.812954 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:01.812779 4982 patch_prober.go:29] interesting pod/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 30 14:27:01.812954 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:01.812842 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh" podUID="d57125b3-c6b9-4d3b-b04c-7f6b30094325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 30 14:27:02.813572 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:02.813529 4982 patch_prober.go:29] interesting pod/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 30 14:27:02.814007 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:02.813608 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh" podUID="d57125b3-c6b9-4d3b-b04c-7f6b30094325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 30 14:27:03.223157 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:03.223040 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:27:03.813157 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:03.813121 4982 patch_prober.go:29] interesting pod/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 30 14:27:03.813479 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:03.813436 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh" podUID="d57125b3-c6b9-4d3b-b04c-7f6b30094325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 30 14:27:04.814092 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:04.814052 4982 patch_prober.go:29] interesting pod/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 30 14:27:04.814715 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:04.814680 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh" podUID="d57125b3-c6b9-4d3b-b04c-7f6b30094325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 30 14:27:05.071949 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:05.071835 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:27:05.812824 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:05.812770 4982 patch_prober.go:29] interesting pod/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 30 14:27:05.813052 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:05.812849 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh" podUID="d57125b3-c6b9-4d3b-b04c-7f6b30094325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 30 14:27:07.166011 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:07.165978 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh" Oct 30 14:27:07.167052 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:07.167004 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh" Oct 30 14:27:07.167969 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:07.167948 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh" Oct 30 14:27:07.219385 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:07.219350 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:27:08.191993 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:08.191960 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:27:09.446231 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:09.446191 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:27:11.325543 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:11.325506 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:27:12.402622 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:12.402590 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:27:13.184706 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:13.184622 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:27:15.246990 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:15.246931 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:27:17.021257 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:17.021224 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:27:17.617353 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:17.617310 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:27:18.526767 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:18.526710 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:27:18.681193 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:18.681143 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:27:18.933200 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:18.933162 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:27:19.119980 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:19.119943 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:27:19.274979 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:19.274885 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:27:21.163480 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:21.163436 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:27:23.057627 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:23.057590 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:27:25.008414 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:25.008369 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:27:26.788910 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:26.788875 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:27:27.884942 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:27.884900 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:27:28.011918 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:27:28.011876 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:27:28.740744 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:28.740702 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:27:30.314972 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:30.314934 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:27:30.887945 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:30.887913 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:27:31.007513 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:31.007473 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ca02c9-7rvdk"] Oct 30 14:27:31.022401 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:31.022363 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ca02c9-7rvdk"] Oct 30 14:27:31.034360 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:31.034331 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:27:31.181115 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:31.181018 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:27:31.549267 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:31.549189 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:27:32.401874 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:32.401837 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90a4f855-40da-44f6-b482-647247fbee35" path="/var/lib/kubelet/pods/90a4f855-40da-44f6-b482-647247fbee35/volumes" Oct 30 14:27:33.041943 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:33.041892 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:27:34.872811 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:34.872760 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:27:36.916143 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:36.916075 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:27:38.902027 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:38.901988 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:27:39.689983 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:39.689948 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:27:39.845848 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:39.845803 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:27:40.028709 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:40.028010 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:27:40.312551 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:40.312515 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:27:40.844147 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:40.844104 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:27:40.989726 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:40.989692 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:27:41.339317 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:41.339278 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:27:41.501660 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:41.501140 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:27:43.056048 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:43.056008 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:27:44.928952 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:44.928905 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:27:45.071783 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:45.071752 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:27:45.493318 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:45.493282 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:27:45.644656 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:45.644586 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:27:46.035986 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:46.035942 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-34bcc5/test-model-34bcc5-kserve-788cf96895-psz95"] Oct 30 14:27:46.036785 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:46.036760 4982 cpu_manager.go:414] "containerMap: removing container" podUID="90a4f855-40da-44f6-b482-647247fbee35" containerName="revoke-keys" Oct 30 14:27:46.036905 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:46.036785 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90a4f855-40da-44f6-b482-647247fbee35" containerName="revoke-keys" Oct 30 14:27:46.036905 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:46.036809 4982 cpu_manager.go:414] "containerMap: removing container" podUID="241e6f6a-6c36-4487-b3ba-92941c524787" containerName="payload-pre-processing" Oct 30 14:27:46.036905 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:46.036818 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="241e6f6a-6c36-4487-b3ba-92941c524787" containerName="payload-pre-processing" Oct 30 14:27:46.036905 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:46.036835 4982 cpu_manager.go:414] "containerMap: removing container" podUID="8c680f8b-331a-4286-a587-300ca1429de7" containerName="payload-processing" Oct 30 14:27:46.036905 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:46.036840 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c680f8b-331a-4286-a587-300ca1429de7" containerName="payload-processing" Oct 30 14:27:46.036905 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:46.036875 4982 cpu_manager.go:414] "containerMap: removing container" podUID="a6687b0d-4a6b-4fe8-9c62-47057042db79" containerName="istio-proxy" Oct 30 14:27:46.036905 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:46.036882 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6687b0d-4a6b-4fe8-9c62-47057042db79" containerName="istio-proxy" Oct 30 14:27:46.037179 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:46.037029 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6687b0d-4a6b-4fe8-9c62-47057042db79" containerName="istio-proxy" Oct 30 14:27:46.037179 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:46.037049 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="90a4f855-40da-44f6-b482-647247fbee35" containerName="revoke-keys" Oct 30 14:27:46.037179 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:46.037059 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="241e6f6a-6c36-4487-b3ba-92941c524787" containerName="payload-pre-processing" Oct 30 14:27:46.037179 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:46.037069 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c680f8b-331a-4286-a587-300ca1429de7" containerName="payload-processing" Oct 30 14:27:46.043113 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:46.043085 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-34bcc5/test-model-34bcc5-kserve-788cf96895-psz95" Oct 30 14:27:46.066222 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:46.066196 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-34bcc5/test-model-34bcc5-kserve-788cf96895-psz95"] Oct 30 14:27:46.075680 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:46.075653 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-34bcc5\"/\"default-dockercfg-2rlsz\"" Oct 30 14:27:46.075881 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:46.075862 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-34bcc5\"/\"test-model-34bcc5-kserve-self-signed-certs\"" Oct 30 14:27:46.077471 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:46.077427 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-34bcc5\"/\"openshift-service-ca.crt\"" Oct 30 14:27:46.077578 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:46.077536 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-34bcc5\"/\"kube-root-ca.crt\"" Oct 30 14:27:46.157328 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:46.157281 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r85n\" (UniqueName: \"kubernetes.io/projected/42c0836c-fae4-4eff-a0f9-298aafae924b-kube-api-access-7r85n\") pod \"test-model-34bcc5-kserve-788cf96895-psz95\" (UID: \"42c0836c-fae4-4eff-a0f9-298aafae924b\") " pod="ai-tenant-e2e-inf-a-34bcc5/test-model-34bcc5-kserve-788cf96895-psz95" Oct 30 14:27:46.157328 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:46.157324 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42c0836c-fae4-4eff-a0f9-298aafae924b-tmp-dir\") pod \"test-model-34bcc5-kserve-788cf96895-psz95\" (UID: \"42c0836c-fae4-4eff-a0f9-298aafae924b\") " pod="ai-tenant-e2e-inf-a-34bcc5/test-model-34bcc5-kserve-788cf96895-psz95" Oct 30 14:27:46.157594 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:46.157485 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42c0836c-fae4-4eff-a0f9-298aafae924b-home\") pod \"test-model-34bcc5-kserve-788cf96895-psz95\" (UID: \"42c0836c-fae4-4eff-a0f9-298aafae924b\") " pod="ai-tenant-e2e-inf-a-34bcc5/test-model-34bcc5-kserve-788cf96895-psz95" Oct 30 14:27:46.157594 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:46.157533 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42c0836c-fae4-4eff-a0f9-298aafae924b-model-cache\") pod \"test-model-34bcc5-kserve-788cf96895-psz95\" (UID: \"42c0836c-fae4-4eff-a0f9-298aafae924b\") " pod="ai-tenant-e2e-inf-a-34bcc5/test-model-34bcc5-kserve-788cf96895-psz95" Oct 30 14:27:46.157759 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:46.157734 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42c0836c-fae4-4eff-a0f9-298aafae924b-tls-certs\") pod \"test-model-34bcc5-kserve-788cf96895-psz95\" (UID: \"42c0836c-fae4-4eff-a0f9-298aafae924b\") " pod="ai-tenant-e2e-inf-a-34bcc5/test-model-34bcc5-kserve-788cf96895-psz95" Oct 30 14:27:46.157891 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:46.157855 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42c0836c-fae4-4eff-a0f9-298aafae924b-dshm\") pod \"test-model-34bcc5-kserve-788cf96895-psz95\" (UID: \"42c0836c-fae4-4eff-a0f9-298aafae924b\") " pod="ai-tenant-e2e-inf-a-34bcc5/test-model-34bcc5-kserve-788cf96895-psz95" Oct 30 14:27:46.241198 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:46.241165 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:27:46.259773 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:46.259734 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42c0836c-fae4-4eff-a0f9-298aafae924b-home\") pod \"test-model-34bcc5-kserve-788cf96895-psz95\" (UID: \"42c0836c-fae4-4eff-a0f9-298aafae924b\") " pod="ai-tenant-e2e-inf-a-34bcc5/test-model-34bcc5-kserve-788cf96895-psz95" Oct 30 14:27:46.259948 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:46.259782 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42c0836c-fae4-4eff-a0f9-298aafae924b-model-cache\") pod \"test-model-34bcc5-kserve-788cf96895-psz95\" (UID: \"42c0836c-fae4-4eff-a0f9-298aafae924b\") " pod="ai-tenant-e2e-inf-a-34bcc5/test-model-34bcc5-kserve-788cf96895-psz95" Oct 30 14:27:46.259948 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:46.259862 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42c0836c-fae4-4eff-a0f9-298aafae924b-tls-certs\") pod \"test-model-34bcc5-kserve-788cf96895-psz95\" (UID: \"42c0836c-fae4-4eff-a0f9-298aafae924b\") " pod="ai-tenant-e2e-inf-a-34bcc5/test-model-34bcc5-kserve-788cf96895-psz95" Oct 30 14:27:46.259948 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:46.259920 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42c0836c-fae4-4eff-a0f9-298aafae924b-dshm\") pod \"test-model-34bcc5-kserve-788cf96895-psz95\" (UID: \"42c0836c-fae4-4eff-a0f9-298aafae924b\") " pod="ai-tenant-e2e-inf-a-34bcc5/test-model-34bcc5-kserve-788cf96895-psz95" Oct 30 14:27:46.260123 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:46.259987 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r85n\" (UniqueName: \"kubernetes.io/projected/42c0836c-fae4-4eff-a0f9-298aafae924b-kube-api-access-7r85n\") pod \"test-model-34bcc5-kserve-788cf96895-psz95\" (UID: \"42c0836c-fae4-4eff-a0f9-298aafae924b\") " pod="ai-tenant-e2e-inf-a-34bcc5/test-model-34bcc5-kserve-788cf96895-psz95" Oct 30 14:27:46.260123 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:46.260017 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42c0836c-fae4-4eff-a0f9-298aafae924b-tmp-dir\") pod \"test-model-34bcc5-kserve-788cf96895-psz95\" (UID: \"42c0836c-fae4-4eff-a0f9-298aafae924b\") " pod="ai-tenant-e2e-inf-a-34bcc5/test-model-34bcc5-kserve-788cf96895-psz95" Oct 30 14:27:46.260235 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:46.260193 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42c0836c-fae4-4eff-a0f9-298aafae924b-home\") pod \"test-model-34bcc5-kserve-788cf96895-psz95\" (UID: \"42c0836c-fae4-4eff-a0f9-298aafae924b\") " pod="ai-tenant-e2e-inf-a-34bcc5/test-model-34bcc5-kserve-788cf96895-psz95" Oct 30 14:27:46.260289 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:46.260245 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42c0836c-fae4-4eff-a0f9-298aafae924b-model-cache\") pod \"test-model-34bcc5-kserve-788cf96895-psz95\" (UID: \"42c0836c-fae4-4eff-a0f9-298aafae924b\") " pod="ai-tenant-e2e-inf-a-34bcc5/test-model-34bcc5-kserve-788cf96895-psz95" Oct 30 14:27:46.260533 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:46.260505 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42c0836c-fae4-4eff-a0f9-298aafae924b-tmp-dir\") pod \"test-model-34bcc5-kserve-788cf96895-psz95\" (UID: \"42c0836c-fae4-4eff-a0f9-298aafae924b\") " pod="ai-tenant-e2e-inf-a-34bcc5/test-model-34bcc5-kserve-788cf96895-psz95" Oct 30 14:27:46.262343 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:46.262315 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42c0836c-fae4-4eff-a0f9-298aafae924b-dshm\") pod \"test-model-34bcc5-kserve-788cf96895-psz95\" (UID: \"42c0836c-fae4-4eff-a0f9-298aafae924b\") " pod="ai-tenant-e2e-inf-a-34bcc5/test-model-34bcc5-kserve-788cf96895-psz95" Oct 30 14:27:46.262668 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:46.262627 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42c0836c-fae4-4eff-a0f9-298aafae924b-tls-certs\") pod \"test-model-34bcc5-kserve-788cf96895-psz95\" (UID: \"42c0836c-fae4-4eff-a0f9-298aafae924b\") " pod="ai-tenant-e2e-inf-a-34bcc5/test-model-34bcc5-kserve-788cf96895-psz95" Oct 30 14:27:46.287612 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:46.287553 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r85n\" (UniqueName: \"kubernetes.io/projected/42c0836c-fae4-4eff-a0f9-298aafae924b-kube-api-access-7r85n\") pod \"test-model-34bcc5-kserve-788cf96895-psz95\" (UID: \"42c0836c-fae4-4eff-a0f9-298aafae924b\") " pod="ai-tenant-e2e-inf-a-34bcc5/test-model-34bcc5-kserve-788cf96895-psz95" Oct 30 14:27:46.361379 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:46.359875 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-34bcc5/test-model-34bcc5-kserve-788cf96895-psz95" Oct 30 14:27:46.784071 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:46.784021 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-34bcc5/test-model-34bcc5-kserve-788cf96895-psz95"] Oct 30 14:27:46.785097 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:27:46.785072 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod42c0836c_fae4_4eff_a0f9_298aafae924b.slice/crio-5fda6af4cf8ed3dbbd3f80b840ebb534456aa4d83a8e38399c88b5c84001f4fe WatchSource:0}: Error finding container 5fda6af4cf8ed3dbbd3f80b840ebb534456aa4d83a8e38399c88b5c84001f4fe: Status 404 returned error can't find the container with id 5fda6af4cf8ed3dbbd3f80b840ebb534456aa4d83a8e38399c88b5c84001f4fe Oct 30 14:27:46.987397 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:46.987355 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-34bcc5/test-model-34bcc5-kserve-788cf96895-psz95" event={"ID":"42c0836c-fae4-4eff-a0f9-298aafae924b","Type":"ContainerStarted","Data":"2fced0d76febf14fcaf0666501ab137d9d3e8f13dde4770f92d249ceffbb2f4b"} Oct 30 14:27:46.987571 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:46.987410 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-34bcc5/test-model-34bcc5-kserve-788cf96895-psz95" event={"ID":"42c0836c-fae4-4eff-a0f9-298aafae924b","Type":"ContainerStarted","Data":"5fda6af4cf8ed3dbbd3f80b840ebb534456aa4d83a8e38399c88b5c84001f4fe"} Oct 30 14:27:47.035323 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:47.035266 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-34bcc5/test-model-34bcc5-kserve-788cf96895-psz95" podStartSLOduration=1.035253951 podStartE2EDuration="1.035253951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:27:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:27:47.031944959 +0000 UTC m=+2336.776079000" watchObservedRunningTime="2026-10-30 14:27:47.035253951 +0000 UTC m=+2336.779387991" Oct 30 14:27:47.071491 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:47.071462 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:27:47.360754 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:47.360669 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-34bcc5/test-model-34bcc5-kserve-788cf96895-psz95" Oct 30 14:27:48.762344 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:48.762308 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:27:48.904059 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:48.904025 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:27:49.095752 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:49.095716 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:27:50.880935 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:50.880893 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:27:51.149474 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:51.149387 4982 scope.go:122] "RemoveContainer" containerID="c697967614cff5094a283125ee9ab25b56553bdb6e99e3af9e3b280703297328" Oct 30 14:27:51.159240 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:51.159219 4982 scope.go:122] "RemoveContainer" containerID="c690de79cf7a93b408455d6e843de01c3d86ad471ae9b28ebc331684b87fd281" Oct 30 14:27:51.169196 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:51.169173 4982 scope.go:122] "RemoveContainer" containerID="2d9714211b91d266cf176e5f58b77f7331efdf614ef9cd9e1e127eaffc973e1a" Oct 30 14:27:51.179129 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:51.179108 4982 scope.go:122] "RemoveContainer" containerID="3a250ab6a9dd0472ffc9fc0969548993323975db47ccb7d10c1c2135f21c92cf" Oct 30 14:27:51.188810 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:51.188786 4982 scope.go:122] "RemoveContainer" containerID="3dfde50d83a37bb9cac2b7f9ca55ea6070830d3d29abe559464966f1394c554a" Oct 30 14:27:53.015930 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:53.015895 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:27:53.284871 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:53.284773 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:27:55.033705 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:55.033635 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:27:55.548868 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:55.548820 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-409364/test-model-409364-kserve-588bbd6c75-k96rk"] Oct 30 14:27:55.557629 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:55.557604 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-409364/test-model-409364-kserve-588bbd6c75-k96rk" Oct 30 14:27:55.571753 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:55.571722 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-409364/test-model-409364-kserve-588bbd6c75-k96rk"] Oct 30 14:27:55.589922 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:55.589890 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-409364\"/\"default-dockercfg-ghjkk\"" Oct 30 14:27:55.590077 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:55.589984 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-409364\"/\"test-model-409364-kserve-self-signed-certs\"" Oct 30 14:27:55.590077 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:55.589998 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-409364\"/\"kube-root-ca.crt\"" Oct 30 14:27:55.590077 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:55.590003 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-409364\"/\"openshift-service-ca.crt\"" Oct 30 14:27:55.603363 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:55.603325 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b106e9f4-a3ff-46b4-8880-5ac6b7c16372-dshm\") pod \"test-model-409364-kserve-588bbd6c75-k96rk\" (UID: \"b106e9f4-a3ff-46b4-8880-5ac6b7c16372\") " pod="ai-tenant-e2e-inf-a-409364/test-model-409364-kserve-588bbd6c75-k96rk" Oct 30 14:27:55.603506 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:55.603377 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b106e9f4-a3ff-46b4-8880-5ac6b7c16372-home\") pod \"test-model-409364-kserve-588bbd6c75-k96rk\" (UID: \"b106e9f4-a3ff-46b4-8880-5ac6b7c16372\") " pod="ai-tenant-e2e-inf-a-409364/test-model-409364-kserve-588bbd6c75-k96rk" Oct 30 14:27:55.603584 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:55.603503 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd9hq\" (UniqueName: \"kubernetes.io/projected/b106e9f4-a3ff-46b4-8880-5ac6b7c16372-kube-api-access-gd9hq\") pod \"test-model-409364-kserve-588bbd6c75-k96rk\" (UID: \"b106e9f4-a3ff-46b4-8880-5ac6b7c16372\") " pod="ai-tenant-e2e-inf-a-409364/test-model-409364-kserve-588bbd6c75-k96rk" Oct 30 14:27:55.603684 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:55.603625 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b106e9f4-a3ff-46b4-8880-5ac6b7c16372-model-cache\") pod \"test-model-409364-kserve-588bbd6c75-k96rk\" (UID: \"b106e9f4-a3ff-46b4-8880-5ac6b7c16372\") " pod="ai-tenant-e2e-inf-a-409364/test-model-409364-kserve-588bbd6c75-k96rk" Oct 30 14:27:55.603779 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:55.603707 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b106e9f4-a3ff-46b4-8880-5ac6b7c16372-tls-certs\") pod \"test-model-409364-kserve-588bbd6c75-k96rk\" (UID: \"b106e9f4-a3ff-46b4-8880-5ac6b7c16372\") " pod="ai-tenant-e2e-inf-a-409364/test-model-409364-kserve-588bbd6c75-k96rk" Oct 30 14:27:55.603859 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:55.603842 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b106e9f4-a3ff-46b4-8880-5ac6b7c16372-tmp-dir\") pod \"test-model-409364-kserve-588bbd6c75-k96rk\" (UID: \"b106e9f4-a3ff-46b4-8880-5ac6b7c16372\") " pod="ai-tenant-e2e-inf-a-409364/test-model-409364-kserve-588bbd6c75-k96rk" Oct 30 14:27:55.629440 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:55.629405 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:27:55.705813 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:55.705781 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b106e9f4-a3ff-46b4-8880-5ac6b7c16372-model-cache\") pod \"test-model-409364-kserve-588bbd6c75-k96rk\" (UID: \"b106e9f4-a3ff-46b4-8880-5ac6b7c16372\") " pod="ai-tenant-e2e-inf-a-409364/test-model-409364-kserve-588bbd6c75-k96rk" Oct 30 14:27:55.705996 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:55.705823 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b106e9f4-a3ff-46b4-8880-5ac6b7c16372-tls-certs\") pod \"test-model-409364-kserve-588bbd6c75-k96rk\" (UID: \"b106e9f4-a3ff-46b4-8880-5ac6b7c16372\") " pod="ai-tenant-e2e-inf-a-409364/test-model-409364-kserve-588bbd6c75-k96rk" Oct 30 14:27:55.705996 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:55.705919 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b106e9f4-a3ff-46b4-8880-5ac6b7c16372-tmp-dir\") pod \"test-model-409364-kserve-588bbd6c75-k96rk\" (UID: \"b106e9f4-a3ff-46b4-8880-5ac6b7c16372\") " pod="ai-tenant-e2e-inf-a-409364/test-model-409364-kserve-588bbd6c75-k96rk" Oct 30 14:27:55.706123 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:55.706051 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b106e9f4-a3ff-46b4-8880-5ac6b7c16372-dshm\") pod \"test-model-409364-kserve-588bbd6c75-k96rk\" (UID: \"b106e9f4-a3ff-46b4-8880-5ac6b7c16372\") " pod="ai-tenant-e2e-inf-a-409364/test-model-409364-kserve-588bbd6c75-k96rk" Oct 30 14:27:55.706123 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:55.706098 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b106e9f4-a3ff-46b4-8880-5ac6b7c16372-home\") pod \"test-model-409364-kserve-588bbd6c75-k96rk\" (UID: \"b106e9f4-a3ff-46b4-8880-5ac6b7c16372\") " pod="ai-tenant-e2e-inf-a-409364/test-model-409364-kserve-588bbd6c75-k96rk" Oct 30 14:27:55.706231 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:55.706133 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd9hq\" (UniqueName: \"kubernetes.io/projected/b106e9f4-a3ff-46b4-8880-5ac6b7c16372-kube-api-access-gd9hq\") pod \"test-model-409364-kserve-588bbd6c75-k96rk\" (UID: \"b106e9f4-a3ff-46b4-8880-5ac6b7c16372\") " pod="ai-tenant-e2e-inf-a-409364/test-model-409364-kserve-588bbd6c75-k96rk" Oct 30 14:27:55.706231 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:55.706220 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b106e9f4-a3ff-46b4-8880-5ac6b7c16372-model-cache\") pod \"test-model-409364-kserve-588bbd6c75-k96rk\" (UID: \"b106e9f4-a3ff-46b4-8880-5ac6b7c16372\") " pod="ai-tenant-e2e-inf-a-409364/test-model-409364-kserve-588bbd6c75-k96rk" Oct 30 14:27:55.706666 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:55.706608 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b106e9f4-a3ff-46b4-8880-5ac6b7c16372-home\") pod \"test-model-409364-kserve-588bbd6c75-k96rk\" (UID: \"b106e9f4-a3ff-46b4-8880-5ac6b7c16372\") " pod="ai-tenant-e2e-inf-a-409364/test-model-409364-kserve-588bbd6c75-k96rk" Oct 30 14:27:55.706847 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:55.706759 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b106e9f4-a3ff-46b4-8880-5ac6b7c16372-tmp-dir\") pod \"test-model-409364-kserve-588bbd6c75-k96rk\" (UID: \"b106e9f4-a3ff-46b4-8880-5ac6b7c16372\") " pod="ai-tenant-e2e-inf-a-409364/test-model-409364-kserve-588bbd6c75-k96rk" Oct 30 14:27:55.708458 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:55.708435 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b106e9f4-a3ff-46b4-8880-5ac6b7c16372-dshm\") pod \"test-model-409364-kserve-588bbd6c75-k96rk\" (UID: \"b106e9f4-a3ff-46b4-8880-5ac6b7c16372\") " pod="ai-tenant-e2e-inf-a-409364/test-model-409364-kserve-588bbd6c75-k96rk" Oct 30 14:27:55.708651 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:55.708621 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b106e9f4-a3ff-46b4-8880-5ac6b7c16372-tls-certs\") pod \"test-model-409364-kserve-588bbd6c75-k96rk\" (UID: \"b106e9f4-a3ff-46b4-8880-5ac6b7c16372\") " pod="ai-tenant-e2e-inf-a-409364/test-model-409364-kserve-588bbd6c75-k96rk" Oct 30 14:27:55.731982 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:55.731952 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd9hq\" (UniqueName: \"kubernetes.io/projected/b106e9f4-a3ff-46b4-8880-5ac6b7c16372-kube-api-access-gd9hq\") pod \"test-model-409364-kserve-588bbd6c75-k96rk\" (UID: \"b106e9f4-a3ff-46b4-8880-5ac6b7c16372\") " pod="ai-tenant-e2e-inf-a-409364/test-model-409364-kserve-588bbd6c75-k96rk" Oct 30 14:27:55.871742 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:55.871702 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-409364/test-model-409364-kserve-588bbd6c75-k96rk" Oct 30 14:27:56.212288 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:56.212246 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-409364/test-model-409364-kserve-588bbd6c75-k96rk"] Oct 30 14:27:56.212812 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:27:56.212788 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb106e9f4_a3ff_46b4_8880_5ac6b7c16372.slice/crio-994710c3c4bcf635670964f71b435c3b34f11d3d3c86550e525de224233dc146 WatchSource:0}: Error finding container 994710c3c4bcf635670964f71b435c3b34f11d3d3c86550e525de224233dc146: Status 404 returned error can't find the container with id 994710c3c4bcf635670964f71b435c3b34f11d3d3c86550e525de224233dc146 Oct 30 14:27:56.361452 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:56.361419 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-34bcc5/test-model-34bcc5-kserve-788cf96895-psz95" Oct 30 14:27:56.374178 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:56.374150 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-34bcc5/test-model-34bcc5-kserve-788cf96895-psz95" Oct 30 14:27:56.920631 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:56.920582 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:27:57.123351 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:57.123316 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-409364/test-model-409364-kserve-588bbd6c75-k96rk" event={"ID":"b106e9f4-a3ff-46b4-8880-5ac6b7c16372","Type":"ContainerStarted","Data":"45da45ebe86de633560c645409aff8dec28d47a488b6eb059bfbdde3fbe66034"} Oct 30 14:27:57.123544 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:57.123366 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-409364/test-model-409364-kserve-588bbd6c75-k96rk" event={"ID":"b106e9f4-a3ff-46b4-8880-5ac6b7c16372","Type":"ContainerStarted","Data":"994710c3c4bcf635670964f71b435c3b34f11d3d3c86550e525de224233dc146"} Oct 30 14:27:57.135558 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:57.135530 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-34bcc5/test-model-34bcc5-kserve-788cf96895-psz95" Oct 30 14:27:57.169263 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:57.169213 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-409364/test-model-409364-kserve-588bbd6c75-k96rk" podStartSLOduration=2.169197221 podStartE2EDuration="2.169197221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:27:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:27:57.168097356 +0000 UTC m=+2346.912231397" watchObservedRunningTime="2026-10-30 14:27:57.169197221 +0000 UTC m=+2346.913331263" Oct 30 14:27:57.871938 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:57.871905 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-409364/test-model-409364-kserve-588bbd6c75-k96rk" Oct 30 14:27:58.013670 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:27:58.013623 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:27:58.816150 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:27:58.816115 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:28:00.631532 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:00.630237 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:28:02.486929 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:02.486893 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:28:04.676184 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:04.676098 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:28:05.872445 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:05.872412 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-409364/test-model-409364-kserve-588bbd6c75-k96rk" Oct 30 14:28:05.887798 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:05.887767 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-409364/test-model-409364-kserve-588bbd6c75-k96rk" Oct 30 14:28:06.208527 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:06.208435 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-409364/test-model-409364-kserve-588bbd6c75-k96rk" Oct 30 14:28:06.227690 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:06.227659 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-b9c4b4/test-model-b9c4b4-kserve-f588646c5-52gfq"] Oct 30 14:28:06.231908 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:06.231886 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b9c4b4/test-model-b9c4b4-kserve-f588646c5-52gfq" Oct 30 14:28:06.251389 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:06.251342 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-b9c4b4\"/\"kube-root-ca.crt\"" Oct 30 14:28:06.251494 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:06.251442 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-b9c4b4\"/\"openshift-service-ca.crt\"" Oct 30 14:28:06.252051 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:06.252030 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-b9c4b4\"/\"default-dockercfg-x4q2d\"" Oct 30 14:28:06.252151 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:06.252083 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-b9c4b4\"/\"test-model-b9c4b4-kserve-self-signed-certs\"" Oct 30 14:28:06.254909 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:06.254881 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-b9c4b4/test-model-b9c4b4-kserve-f588646c5-52gfq"] Oct 30 14:28:06.343154 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:06.343116 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:28:06.361943 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:06.361909 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttwkz\" (UniqueName: \"kubernetes.io/projected/34670e92-5762-42a0-9899-863d0a92a6bf-kube-api-access-ttwkz\") pod \"test-model-b9c4b4-kserve-f588646c5-52gfq\" (UID: \"34670e92-5762-42a0-9899-863d0a92a6bf\") " pod="ai-tenant-e2e-inf-b-b9c4b4/test-model-b9c4b4-kserve-f588646c5-52gfq" Oct 30 14:28:06.362117 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:06.361968 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34670e92-5762-42a0-9899-863d0a92a6bf-dshm\") pod \"test-model-b9c4b4-kserve-f588646c5-52gfq\" (UID: \"34670e92-5762-42a0-9899-863d0a92a6bf\") " pod="ai-tenant-e2e-inf-b-b9c4b4/test-model-b9c4b4-kserve-f588646c5-52gfq" Oct 30 14:28:06.362117 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:06.362030 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34670e92-5762-42a0-9899-863d0a92a6bf-tmp-dir\") pod \"test-model-b9c4b4-kserve-f588646c5-52gfq\" (UID: \"34670e92-5762-42a0-9899-863d0a92a6bf\") " pod="ai-tenant-e2e-inf-b-b9c4b4/test-model-b9c4b4-kserve-f588646c5-52gfq" Oct 30 14:28:06.362235 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:06.362205 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34670e92-5762-42a0-9899-863d0a92a6bf-home\") pod \"test-model-b9c4b4-kserve-f588646c5-52gfq\" (UID: \"34670e92-5762-42a0-9899-863d0a92a6bf\") " pod="ai-tenant-e2e-inf-b-b9c4b4/test-model-b9c4b4-kserve-f588646c5-52gfq" Oct 30 14:28:06.362290 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:06.362263 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34670e92-5762-42a0-9899-863d0a92a6bf-tls-certs\") pod \"test-model-b9c4b4-kserve-f588646c5-52gfq\" (UID: \"34670e92-5762-42a0-9899-863d0a92a6bf\") " pod="ai-tenant-e2e-inf-b-b9c4b4/test-model-b9c4b4-kserve-f588646c5-52gfq" Oct 30 14:28:06.362344 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:06.362308 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34670e92-5762-42a0-9899-863d0a92a6bf-model-cache\") pod \"test-model-b9c4b4-kserve-f588646c5-52gfq\" (UID: \"34670e92-5762-42a0-9899-863d0a92a6bf\") " pod="ai-tenant-e2e-inf-b-b9c4b4/test-model-b9c4b4-kserve-f588646c5-52gfq" Oct 30 14:28:06.465786 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:06.465687 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34670e92-5762-42a0-9899-863d0a92a6bf-home\") pod \"test-model-b9c4b4-kserve-f588646c5-52gfq\" (UID: \"34670e92-5762-42a0-9899-863d0a92a6bf\") " pod="ai-tenant-e2e-inf-b-b9c4b4/test-model-b9c4b4-kserve-f588646c5-52gfq" Oct 30 14:28:06.466098 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:06.466072 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34670e92-5762-42a0-9899-863d0a92a6bf-tls-certs\") pod \"test-model-b9c4b4-kserve-f588646c5-52gfq\" (UID: \"34670e92-5762-42a0-9899-863d0a92a6bf\") " pod="ai-tenant-e2e-inf-b-b9c4b4/test-model-b9c4b4-kserve-f588646c5-52gfq" Oct 30 14:28:06.466261 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:06.466245 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34670e92-5762-42a0-9899-863d0a92a6bf-model-cache\") pod \"test-model-b9c4b4-kserve-f588646c5-52gfq\" (UID: \"34670e92-5762-42a0-9899-863d0a92a6bf\") " pod="ai-tenant-e2e-inf-b-b9c4b4/test-model-b9c4b4-kserve-f588646c5-52gfq" Oct 30 14:28:06.466497 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:06.466480 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttwkz\" (UniqueName: \"kubernetes.io/projected/34670e92-5762-42a0-9899-863d0a92a6bf-kube-api-access-ttwkz\") pod \"test-model-b9c4b4-kserve-f588646c5-52gfq\" (UID: \"34670e92-5762-42a0-9899-863d0a92a6bf\") " pod="ai-tenant-e2e-inf-b-b9c4b4/test-model-b9c4b4-kserve-f588646c5-52gfq" Oct 30 14:28:06.466608 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:06.466595 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34670e92-5762-42a0-9899-863d0a92a6bf-dshm\") pod \"test-model-b9c4b4-kserve-f588646c5-52gfq\" (UID: \"34670e92-5762-42a0-9899-863d0a92a6bf\") " pod="ai-tenant-e2e-inf-b-b9c4b4/test-model-b9c4b4-kserve-f588646c5-52gfq" Oct 30 14:28:06.466784 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:06.466766 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34670e92-5762-42a0-9899-863d0a92a6bf-tmp-dir\") pod \"test-model-b9c4b4-kserve-f588646c5-52gfq\" (UID: \"34670e92-5762-42a0-9899-863d0a92a6bf\") " pod="ai-tenant-e2e-inf-b-b9c4b4/test-model-b9c4b4-kserve-f588646c5-52gfq" Oct 30 14:28:06.467331 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:06.467304 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34670e92-5762-42a0-9899-863d0a92a6bf-tmp-dir\") pod \"test-model-b9c4b4-kserve-f588646c5-52gfq\" (UID: \"34670e92-5762-42a0-9899-863d0a92a6bf\") " pod="ai-tenant-e2e-inf-b-b9c4b4/test-model-b9c4b4-kserve-f588646c5-52gfq" Oct 30 14:28:06.467548 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:06.467529 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34670e92-5762-42a0-9899-863d0a92a6bf-home\") pod \"test-model-b9c4b4-kserve-f588646c5-52gfq\" (UID: \"34670e92-5762-42a0-9899-863d0a92a6bf\") " pod="ai-tenant-e2e-inf-b-b9c4b4/test-model-b9c4b4-kserve-f588646c5-52gfq" Oct 30 14:28:06.469921 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:06.469897 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34670e92-5762-42a0-9899-863d0a92a6bf-model-cache\") pod \"test-model-b9c4b4-kserve-f588646c5-52gfq\" (UID: \"34670e92-5762-42a0-9899-863d0a92a6bf\") " pod="ai-tenant-e2e-inf-b-b9c4b4/test-model-b9c4b4-kserve-f588646c5-52gfq" Oct 30 14:28:06.476387 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:06.476355 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34670e92-5762-42a0-9899-863d0a92a6bf-dshm\") pod \"test-model-b9c4b4-kserve-f588646c5-52gfq\" (UID: \"34670e92-5762-42a0-9899-863d0a92a6bf\") " pod="ai-tenant-e2e-inf-b-b9c4b4/test-model-b9c4b4-kserve-f588646c5-52gfq" Oct 30 14:28:06.477665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:06.477585 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34670e92-5762-42a0-9899-863d0a92a6bf-tls-certs\") pod \"test-model-b9c4b4-kserve-f588646c5-52gfq\" (UID: \"34670e92-5762-42a0-9899-863d0a92a6bf\") " pod="ai-tenant-e2e-inf-b-b9c4b4/test-model-b9c4b4-kserve-f588646c5-52gfq" Oct 30 14:28:06.502241 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:06.502185 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttwkz\" (UniqueName: \"kubernetes.io/projected/34670e92-5762-42a0-9899-863d0a92a6bf-kube-api-access-ttwkz\") pod \"test-model-b9c4b4-kserve-f588646c5-52gfq\" (UID: \"34670e92-5762-42a0-9899-863d0a92a6bf\") " pod="ai-tenant-e2e-inf-b-b9c4b4/test-model-b9c4b4-kserve-f588646c5-52gfq" Oct 30 14:28:06.552203 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:06.551933 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b9c4b4/test-model-b9c4b4-kserve-f588646c5-52gfq" Oct 30 14:28:06.936861 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:06.936824 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-b9c4b4/test-model-b9c4b4-kserve-f588646c5-52gfq"] Oct 30 14:28:06.937204 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:28:06.936835 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod34670e92_5762_42a0_9899_863d0a92a6bf.slice/crio-79b3145ef2b374a5cfefa6fe1390c0ca6c0400f029b71e07f5626734bd09e199 WatchSource:0}: Error finding container 79b3145ef2b374a5cfefa6fe1390c0ca6c0400f029b71e07f5626734bd09e199: Status 404 returned error can't find the container with id 79b3145ef2b374a5cfefa6fe1390c0ca6c0400f029b71e07f5626734bd09e199 Oct 30 14:28:07.079866 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:07.079808 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:28:07.211744 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:07.206440 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b9c4b4/test-model-b9c4b4-kserve-f588646c5-52gfq" event={"ID":"34670e92-5762-42a0-9899-863d0a92a6bf","Type":"ContainerStarted","Data":"a67aa1ecab5f2e5b7ccb7e2a5b94a730817f8873f69aa1b68334409ae5447b59"} Oct 30 14:28:07.211744 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:07.206507 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b9c4b4/test-model-b9c4b4-kserve-f588646c5-52gfq" event={"ID":"34670e92-5762-42a0-9899-863d0a92a6bf","Type":"ContainerStarted","Data":"79b3145ef2b374a5cfefa6fe1390c0ca6c0400f029b71e07f5626734bd09e199"} Oct 30 14:28:07.254066 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:07.254014 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-b9c4b4/test-model-b9c4b4-kserve-f588646c5-52gfq" podStartSLOduration=1.253997254 podStartE2EDuration="1.253997254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:28:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:28:07.250751108 +0000 UTC m=+2356.994885148" watchObservedRunningTime="2026-10-30 14:28:07.253997254 +0000 UTC m=+2356.998131337" Oct 30 14:28:07.552772 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:07.552692 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-b9c4b4/test-model-b9c4b4-kserve-f588646c5-52gfq" Oct 30 14:28:09.170219 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:09.170180 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:28:11.448968 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:11.448897 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:28:12.699510 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:12.699462 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:28:13.169867 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:13.169825 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:28:13.694356 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:13.694319 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:28:13.861973 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:13.861932 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:28:14.234249 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:14.234201 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c5e8070-5qdn2"] Oct 30 14:28:14.239174 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:14.239149 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c5e8070-5qdn2" Oct 30 14:28:14.259679 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:14.259629 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:28:14.259846 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:14.259754 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:28:14.264405 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:14.264375 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c5e8070-5qdn2"] Oct 30 14:28:14.310324 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:14.310288 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cdc8048-a4db-4e5f-82d1-b0229c255311-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c5e8070-5qdn2\" (UID: \"8cdc8048-a4db-4e5f-82d1-b0229c255311\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c5e8070-5qdn2" Oct 30 14:28:14.310691 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:14.310666 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cdc8048-a4db-4e5f-82d1-b0229c255311-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c5e8070-5qdn2\" (UID: \"8cdc8048-a4db-4e5f-82d1-b0229c255311\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c5e8070-5qdn2" Oct 30 14:28:14.311006 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:14.310987 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7n2b\" (UniqueName: \"kubernetes.io/projected/8cdc8048-a4db-4e5f-82d1-b0229c255311-kube-api-access-h7n2b\") pod \"maas-api-revoke-sub-8c5e8070-5qdn2\" (UID: \"8cdc8048-a4db-4e5f-82d1-b0229c255311\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c5e8070-5qdn2" Oct 30 14:28:14.351679 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:14.346403 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:28:14.412361 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:14.412328 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cdc8048-a4db-4e5f-82d1-b0229c255311-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c5e8070-5qdn2\" (UID: \"8cdc8048-a4db-4e5f-82d1-b0229c255311\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c5e8070-5qdn2" Oct 30 14:28:14.412635 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:14.412392 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cdc8048-a4db-4e5f-82d1-b0229c255311-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c5e8070-5qdn2\" (UID: \"8cdc8048-a4db-4e5f-82d1-b0229c255311\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c5e8070-5qdn2" Oct 30 14:28:14.412635 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:14.412502 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7n2b\" (UniqueName: \"kubernetes.io/projected/8cdc8048-a4db-4e5f-82d1-b0229c255311-kube-api-access-h7n2b\") pod \"maas-api-revoke-sub-8c5e8070-5qdn2\" (UID: \"8cdc8048-a4db-4e5f-82d1-b0229c255311\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c5e8070-5qdn2" Oct 30 14:28:14.413777 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:14.413751 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cdc8048-a4db-4e5f-82d1-b0229c255311-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c5e8070-5qdn2\" (UID: \"8cdc8048-a4db-4e5f-82d1-b0229c255311\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c5e8070-5qdn2" Oct 30 14:28:14.424162 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:14.424128 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cdc8048-a4db-4e5f-82d1-b0229c255311-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c5e8070-5qdn2\" (UID: \"8cdc8048-a4db-4e5f-82d1-b0229c255311\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c5e8070-5qdn2" Oct 30 14:28:14.439151 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:14.439119 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7n2b\" (UniqueName: \"kubernetes.io/projected/8cdc8048-a4db-4e5f-82d1-b0229c255311-kube-api-access-h7n2b\") pod \"maas-api-revoke-sub-8c5e8070-5qdn2\" (UID: \"8cdc8048-a4db-4e5f-82d1-b0229c255311\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c5e8070-5qdn2" Oct 30 14:28:14.557008 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:14.556983 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c5e8070-5qdn2" Oct 30 14:28:14.924748 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:14.924716 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c5e8070-5qdn2"] Oct 30 14:28:15.277850 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:15.277769 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c5e8070-5qdn2" event={"ID":"8cdc8048-a4db-4e5f-82d1-b0229c255311","Type":"ContainerStarted","Data":"cc13540c68cba254523755405d316d04b734171e0335c79908abf38d2979542e"} Oct 30 14:28:15.769580 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:15.769513 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:28:16.289059 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:16.289024 4982 generic.go:356] "Generic (PLEG): container finished" podID="8cdc8048-a4db-4e5f-82d1-b0229c255311" containerID="4c8f72d19b911eabce0835956f6e51b7eed25ebdb91132ffef97f228c4efb453" exitCode=0 Oct 30 14:28:16.289454 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:16.289113 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c5e8070-5qdn2" event={"ID":"8cdc8048-a4db-4e5f-82d1-b0229c255311","Type":"ContainerDied","Data":"4c8f72d19b911eabce0835956f6e51b7eed25ebdb91132ffef97f228c4efb453"} Oct 30 14:28:16.552880 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:16.552798 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-b9c4b4/test-model-b9c4b4-kserve-f588646c5-52gfq" Oct 30 14:28:16.565108 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:16.565080 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-b9c4b4/test-model-b9c4b4-kserve-f588646c5-52gfq" Oct 30 14:28:17.312432 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:17.312399 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-b9c4b4/test-model-b9c4b4-kserve-f588646c5-52gfq" Oct 30 14:28:17.486715 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:17.486684 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c5e8070-5qdn2" Oct 30 14:28:17.558690 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:17.558620 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cdc8048-a4db-4e5f-82d1-b0229c255311-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cdc8048-a4db-4e5f-82d1-b0229c255311-maas-api-openshift-ca\") pod \"8cdc8048-a4db-4e5f-82d1-b0229c255311\" (UID: \"8cdc8048-a4db-4e5f-82d1-b0229c255311\") " Oct 30 14:28:17.558945 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:17.558921 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cdc8048-a4db-4e5f-82d1-b0229c255311-maas-api-openshift-ca" pod "8cdc8048-a4db-4e5f-82d1-b0229c255311" (UID: "8cdc8048-a4db-4e5f-82d1-b0229c255311"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:28:17.559036 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:17.558957 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cdc8048-a4db-4e5f-82d1-b0229c255311-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cdc8048-a4db-4e5f-82d1-b0229c255311-maas-api-xks-ca\") pod \"8cdc8048-a4db-4e5f-82d1-b0229c255311\" (UID: \"8cdc8048-a4db-4e5f-82d1-b0229c255311\") " Oct 30 14:28:17.559036 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:17.559014 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cdc8048-a4db-4e5f-82d1-b0229c255311-kube-api-access-h7n2b\" (UniqueName: \"kubernetes.io/projected/8cdc8048-a4db-4e5f-82d1-b0229c255311-kube-api-access-h7n2b\") pod \"8cdc8048-a4db-4e5f-82d1-b0229c255311\" (UID: \"8cdc8048-a4db-4e5f-82d1-b0229c255311\") " Oct 30 14:28:17.561137 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:17.559593 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cdc8048-a4db-4e5f-82d1-b0229c255311-maas-api-openshift-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:28:17.561295 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:17.561060 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cdc8048-a4db-4e5f-82d1-b0229c255311-maas-api-xks-ca" pod "8cdc8048-a4db-4e5f-82d1-b0229c255311" (UID: "8cdc8048-a4db-4e5f-82d1-b0229c255311"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:28:17.561527 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:17.561502 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cdc8048-a4db-4e5f-82d1-b0229c255311-kube-api-access-h7n2b" pod "8cdc8048-a4db-4e5f-82d1-b0229c255311" (UID: "8cdc8048-a4db-4e5f-82d1-b0229c255311"). InnerVolumeSpecName "kube-api-access-h7n2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:28:17.662817 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:17.662786 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cdc8048-a4db-4e5f-82d1-b0229c255311-maas-api-xks-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:28:17.662817 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:17.662816 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7n2b\" (UniqueName: \"kubernetes.io/projected/8cdc8048-a4db-4e5f-82d1-b0229c255311-kube-api-access-h7n2b\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:28:17.854667 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:17.854567 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:28:18.003139 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:18.003098 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:28:18.306628 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:18.306545 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c5e8070-5qdn2" Oct 30 14:28:18.306628 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:18.306574 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c5e8070-5qdn2" event={"ID":"8cdc8048-a4db-4e5f-82d1-b0229c255311","Type":"ContainerDied","Data":"cc13540c68cba254523755405d316d04b734171e0335c79908abf38d2979542e"} Oct 30 14:28:18.306870 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:18.306634 4982 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc13540c68cba254523755405d316d04b734171e0335c79908abf38d2979542e" Oct 30 14:28:19.692373 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:19.692336 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:28:19.819306 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:19.819273 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1d3696-69b88b47c5-kpq6k"] Oct 30 14:28:19.819970 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:19.819953 4982 cpu_manager.go:414] "containerMap: removing container" podUID="8cdc8048-a4db-4e5f-82d1-b0229c255311" containerName="revoke-keys" Oct 30 14:28:19.820032 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:19.819971 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cdc8048-a4db-4e5f-82d1-b0229c255311" containerName="revoke-keys" Oct 30 14:28:19.820092 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:19.820084 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cdc8048-a4db-4e5f-82d1-b0229c255311" containerName="revoke-keys" Oct 30 14:28:19.823982 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:19.823962 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1d3696-69b88b47c5-kpq6k" Oct 30 14:28:19.844557 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:19.844536 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-1d3696-dockercfg-crvv4\"" Oct 30 14:28:19.844690 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:19.844596 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-1d3696\"" Oct 30 14:28:19.844690 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:19.844598 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1d3696-69b88b47c5-kpq6k"] Oct 30 14:28:19.855400 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:19.855377 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:28:20.098713 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:20.091133 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9484k\" (UniqueName: \"kubernetes.io/projected/1d7bb15a-a136-49f4-b8cd-6739f38bfb99-kube-api-access-9484k\") pod \"payload-pre-processing-e2e-inf-b-1d3696-69b88b47c5-kpq6k\" (UID: \"1d7bb15a-a136-49f4-b8cd-6739f38bfb99\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1d3696-69b88b47c5-kpq6k" Oct 30 14:28:20.099125 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:20.099096 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d7bb15a-a136-49f4-b8cd-6739f38bfb99-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-1d3696-69b88b47c5-kpq6k\" (UID: \"1d7bb15a-a136-49f4-b8cd-6739f38bfb99\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1d3696-69b88b47c5-kpq6k" Oct 30 14:28:20.118968 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:20.116373 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:28:20.200328 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:20.200286 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d7bb15a-a136-49f4-b8cd-6739f38bfb99-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-1d3696-69b88b47c5-kpq6k\" (UID: \"1d7bb15a-a136-49f4-b8cd-6739f38bfb99\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1d3696-69b88b47c5-kpq6k" Oct 30 14:28:20.200531 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:20.200509 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9484k\" (UniqueName: \"kubernetes.io/projected/1d7bb15a-a136-49f4-b8cd-6739f38bfb99-kube-api-access-9484k\") pod \"payload-pre-processing-e2e-inf-b-1d3696-69b88b47c5-kpq6k\" (UID: \"1d7bb15a-a136-49f4-b8cd-6739f38bfb99\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1d3696-69b88b47c5-kpq6k" Oct 30 14:28:20.201124 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:20.201102 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d7bb15a-a136-49f4-b8cd-6739f38bfb99-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-1d3696-69b88b47c5-kpq6k\" (UID: \"1d7bb15a-a136-49f4-b8cd-6739f38bfb99\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1d3696-69b88b47c5-kpq6k" Oct 30 14:28:20.226611 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:20.226578 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9484k\" (UniqueName: \"kubernetes.io/projected/1d7bb15a-a136-49f4-b8cd-6739f38bfb99-kube-api-access-9484k\") pod \"payload-pre-processing-e2e-inf-b-1d3696-69b88b47c5-kpq6k\" (UID: \"1d7bb15a-a136-49f4-b8cd-6739f38bfb99\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1d3696-69b88b47c5-kpq6k" Oct 30 14:28:20.438154 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:20.438076 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1d3696-69b88b47c5-kpq6k" Oct 30 14:28:20.764499 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:28:20.764465 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d7bb15a_a136_49f4_b8cd_6739f38bfb99.slice/crio-80c404cc4d5894bf92d880e52c6dc08135d74baf785a970f89affb433f90f6bf WatchSource:0}: Error finding container 80c404cc4d5894bf92d880e52c6dc08135d74baf785a970f89affb433f90f6bf: Status 404 returned error can't find the container with id 80c404cc4d5894bf92d880e52c6dc08135d74baf785a970f89affb433f90f6bf Oct 30 14:28:20.764832 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:20.764556 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1d3696-69b88b47c5-kpq6k"] Oct 30 14:28:21.340562 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:21.340519 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1d3696-69b88b47c5-kpq6k" event={"ID":"1d7bb15a-a136-49f4-b8cd-6739f38bfb99","Type":"ContainerStarted","Data":"8f87ccdb4e872c8e560c2a6e8700a93d3438ca9f836e1428149057601b16bab2"} Oct 30 14:28:21.340853 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:21.340591 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1d3696-69b88b47c5-kpq6k" event={"ID":"1d7bb15a-a136-49f4-b8cd-6739f38bfb99","Type":"ContainerStarted","Data":"80c404cc4d5894bf92d880e52c6dc08135d74baf785a970f89affb433f90f6bf"} Oct 30 14:28:21.340853 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:21.340699 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1d3696-69b88b47c5-kpq6k" Oct 30 14:28:21.387574 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:21.387452 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1d3696-69b88b47c5-kpq6k" podStartSLOduration=2.387433234 podStartE2EDuration="2.387433234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:28:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:28:21.385950861 +0000 UTC m=+2371.130084902" watchObservedRunningTime="2026-10-30 14:28:21.387433234 +0000 UTC m=+2371.131567277" Oct 30 14:28:21.887317 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:21.887281 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:28:24.028812 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:24.028769 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:28:25.941077 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:25.941041 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:28:27.172421 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:27.172381 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:28:27.812413 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:27.812377 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4edd955-47wlv"] Oct 30 14:28:27.817075 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:27.817051 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4edd955-47wlv" Oct 30 14:28:27.837901 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:27.837876 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4edd955-47wlv"] Oct 30 14:28:27.852918 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:27.846406 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:28:27.852918 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:27.846456 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:28:27.933392 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:27.933342 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8rg7\" (UniqueName: \"kubernetes.io/projected/b2b1bddb-21f5-4576-b5cb-ae696fc925f4-kube-api-access-n8rg7\") pod \"maas-api-revoke-sub-d4edd955-47wlv\" (UID: \"b2b1bddb-21f5-4576-b5cb-ae696fc925f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4edd955-47wlv" Oct 30 14:28:27.933573 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:27.933462 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2b1bddb-21f5-4576-b5cb-ae696fc925f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4edd955-47wlv\" (UID: \"b2b1bddb-21f5-4576-b5cb-ae696fc925f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4edd955-47wlv" Oct 30 14:28:27.933740 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:27.933717 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2b1bddb-21f5-4576-b5cb-ae696fc925f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4edd955-47wlv\" (UID: \"b2b1bddb-21f5-4576-b5cb-ae696fc925f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4edd955-47wlv" Oct 30 14:28:28.013136 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:28:28.013095 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:28:28.028548 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:28.028507 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:28:28.040634 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:28.037044 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8rg7\" (UniqueName: \"kubernetes.io/projected/b2b1bddb-21f5-4576-b5cb-ae696fc925f4-kube-api-access-n8rg7\") pod \"maas-api-revoke-sub-d4edd955-47wlv\" (UID: \"b2b1bddb-21f5-4576-b5cb-ae696fc925f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4edd955-47wlv" Oct 30 14:28:28.040634 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:28.039091 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2b1bddb-21f5-4576-b5cb-ae696fc925f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4edd955-47wlv\" (UID: \"b2b1bddb-21f5-4576-b5cb-ae696fc925f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4edd955-47wlv" Oct 30 14:28:28.040634 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:28.039295 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2b1bddb-21f5-4576-b5cb-ae696fc925f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4edd955-47wlv\" (UID: \"b2b1bddb-21f5-4576-b5cb-ae696fc925f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4edd955-47wlv" Oct 30 14:28:28.044285 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:28.044250 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2b1bddb-21f5-4576-b5cb-ae696fc925f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4edd955-47wlv\" (UID: \"b2b1bddb-21f5-4576-b5cb-ae696fc925f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4edd955-47wlv" Oct 30 14:28:28.044455 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:28.044338 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2b1bddb-21f5-4576-b5cb-ae696fc925f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4edd955-47wlv\" (UID: \"b2b1bddb-21f5-4576-b5cb-ae696fc925f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4edd955-47wlv" Oct 30 14:28:28.062994 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:28.062923 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8rg7\" (UniqueName: \"kubernetes.io/projected/b2b1bddb-21f5-4576-b5cb-ae696fc925f4-kube-api-access-n8rg7\") pod \"maas-api-revoke-sub-d4edd955-47wlv\" (UID: \"b2b1bddb-21f5-4576-b5cb-ae696fc925f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4edd955-47wlv" Oct 30 14:28:28.131280 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:28.131243 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4edd955-47wlv" Oct 30 14:28:28.411600 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:28.411517 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:28:28.449480 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:28:28.449451 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2b1bddb_21f5_4576_b5cb_ae696fc925f4.slice/crio-80ee3d61f4e02cf9abda06d513a74a93b16f677e92117d807ca6bddef1db145a WatchSource:0}: Error finding container 80ee3d61f4e02cf9abda06d513a74a93b16f677e92117d807ca6bddef1db145a: Status 404 returned error can't find the container with id 80ee3d61f4e02cf9abda06d513a74a93b16f677e92117d807ca6bddef1db145a Oct 30 14:28:28.450192 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:28.450167 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4edd955-47wlv"] Oct 30 14:28:28.516862 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:28.516826 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp"] Oct 30 14:28:28.517037 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:28.517014 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp" podUID="2ae22b83-f2b9-47c9-b51b-f70a43f699b5" containerName="istio-proxy" containerID="cri-o://8ff6d043b111d16c1d416b1b59c2487fd3cf6462c7087b77548acf73b286ed95" gracePeriod=30 Oct 30 14:28:28.575825 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:28.575785 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:28:29.419383 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:29.419340 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4edd955-47wlv" event={"ID":"b2b1bddb-21f5-4576-b5cb-ae696fc925f4","Type":"ContainerStarted","Data":"07bb29a424bed8849c24f080f5f4b7c04a0a8d0bfd37d4da75cdc4b81b122d95"} Oct 30 14:28:29.419883 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:29.419446 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4edd955-47wlv" event={"ID":"b2b1bddb-21f5-4576-b5cb-ae696fc925f4","Type":"ContainerStarted","Data":"80ee3d61f4e02cf9abda06d513a74a93b16f677e92117d807ca6bddef1db145a"} Oct 30 14:28:29.465628 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:29.465574 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4edd955-47wlv" podStartSLOduration=2.465559095 podStartE2EDuration="2.465559095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:28:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:28:29.462070643 +0000 UTC m=+2379.206204684" watchObservedRunningTime="2026-10-30 14:28:29.465559095 +0000 UTC m=+2379.209693134" Oct 30 14:28:29.877671 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:29.876966 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:28:30.429092 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:30.429057 4982 generic.go:356] "Generic (PLEG): container finished" podID="b2b1bddb-21f5-4576-b5cb-ae696fc925f4" containerID="07bb29a424bed8849c24f080f5f4b7c04a0a8d0bfd37d4da75cdc4b81b122d95" exitCode=0 Oct 30 14:28:30.429594 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:30.429130 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4edd955-47wlv" event={"ID":"b2b1bddb-21f5-4576-b5cb-ae696fc925f4","Type":"ContainerDied","Data":"07bb29a424bed8849c24f080f5f4b7c04a0a8d0bfd37d4da75cdc4b81b122d95"} Oct 30 14:28:31.616439 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:31.616412 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4edd955-47wlv" Oct 30 14:28:31.669428 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:31.669398 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:28:31.836880 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:31.836838 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2b1bddb-21f5-4576-b5cb-ae696fc925f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2b1bddb-21f5-4576-b5cb-ae696fc925f4-maas-api-xks-ca\") pod \"b2b1bddb-21f5-4576-b5cb-ae696fc925f4\" (UID: \"b2b1bddb-21f5-4576-b5cb-ae696fc925f4\") " Oct 30 14:28:31.837071 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:31.836990 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2b1bddb-21f5-4576-b5cb-ae696fc925f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2b1bddb-21f5-4576-b5cb-ae696fc925f4-maas-api-openshift-ca\") pod \"b2b1bddb-21f5-4576-b5cb-ae696fc925f4\" (UID: \"b2b1bddb-21f5-4576-b5cb-ae696fc925f4\") " Oct 30 14:28:31.837142 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:31.837099 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2b1bddb-21f5-4576-b5cb-ae696fc925f4-kube-api-access-n8rg7\" (UniqueName: \"kubernetes.io/projected/b2b1bddb-21f5-4576-b5cb-ae696fc925f4-kube-api-access-n8rg7\") pod \"b2b1bddb-21f5-4576-b5cb-ae696fc925f4\" (UID: \"b2b1bddb-21f5-4576-b5cb-ae696fc925f4\") " Oct 30 14:28:31.837415 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:31.837388 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2b1bddb-21f5-4576-b5cb-ae696fc925f4-maas-api-openshift-ca" pod "b2b1bddb-21f5-4576-b5cb-ae696fc925f4" (UID: "b2b1bddb-21f5-4576-b5cb-ae696fc925f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:28:31.837628 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:31.837612 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2b1bddb-21f5-4576-b5cb-ae696fc925f4-maas-api-openshift-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:28:31.839054 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:31.839027 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2b1bddb-21f5-4576-b5cb-ae696fc925f4-maas-api-xks-ca" pod "b2b1bddb-21f5-4576-b5cb-ae696fc925f4" (UID: "b2b1bddb-21f5-4576-b5cb-ae696fc925f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:28:31.839181 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:31.839160 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2b1bddb-21f5-4576-b5cb-ae696fc925f4-kube-api-access-n8rg7" pod "b2b1bddb-21f5-4576-b5cb-ae696fc925f4" (UID: "b2b1bddb-21f5-4576-b5cb-ae696fc925f4"). InnerVolumeSpecName "kube-api-access-n8rg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:28:31.938951 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:31.938916 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8rg7\" (UniqueName: \"kubernetes.io/projected/b2b1bddb-21f5-4576-b5cb-ae696fc925f4-kube-api-access-n8rg7\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:28:31.938951 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:31.938944 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2b1bddb-21f5-4576-b5cb-ae696fc925f4-maas-api-xks-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:28:32.352769 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:32.352737 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1d3696-69b88b47c5-kpq6k" Oct 30 14:28:32.472404 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:32.470265 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4edd955-47wlv" Oct 30 14:28:32.472404 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:32.470279 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4edd955-47wlv" event={"ID":"b2b1bddb-21f5-4576-b5cb-ae696fc925f4","Type":"ContainerDied","Data":"80ee3d61f4e02cf9abda06d513a74a93b16f677e92117d807ca6bddef1db145a"} Oct 30 14:28:32.472404 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:32.470328 4982 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80ee3d61f4e02cf9abda06d513a74a93b16f677e92117d807ca6bddef1db145a" Oct 30 14:28:32.649531 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:32.649444 4982 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 14:28:32.649531 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:32.649513 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp" podUID="2ae22b83-f2b9-47c9-b51b-f70a43f699b5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 14:28:33.745220 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:33.745179 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:28:33.851172 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:33.851138 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp" Oct 30 14:28:33.879980 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:33.879947 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ae22b83-f2b9-47c9-b51b-f70a43f699b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae22b83-f2b9-47c9-b51b-f70a43f699b5-credential-socket\") pod \"2ae22b83-f2b9-47c9-b51b-f70a43f699b5\" (UID: \"2ae22b83-f2b9-47c9-b51b-f70a43f699b5\") " Oct 30 14:28:33.880165 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:33.880020 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ae22b83-f2b9-47c9-b51b-f70a43f699b5-kube-api-access-sjmrk\" (UniqueName: \"kubernetes.io/projected/2ae22b83-f2b9-47c9-b51b-f70a43f699b5-kube-api-access-sjmrk\") pod \"2ae22b83-f2b9-47c9-b51b-f70a43f699b5\" (UID: \"2ae22b83-f2b9-47c9-b51b-f70a43f699b5\") " Oct 30 14:28:33.880165 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:33.880090 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ae22b83-f2b9-47c9-b51b-f70a43f699b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae22b83-f2b9-47c9-b51b-f70a43f699b5-workload-socket\") pod \"2ae22b83-f2b9-47c9-b51b-f70a43f699b5\" (UID: \"2ae22b83-f2b9-47c9-b51b-f70a43f699b5\") " Oct 30 14:28:33.880165 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:33.880123 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ae22b83-f2b9-47c9-b51b-f70a43f699b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ae22b83-f2b9-47c9-b51b-f70a43f699b5-workload-certs\") pod \"2ae22b83-f2b9-47c9-b51b-f70a43f699b5\" (UID: \"2ae22b83-f2b9-47c9-b51b-f70a43f699b5\") " Oct 30 14:28:33.880348 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:33.880192 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ae22b83-f2b9-47c9-b51b-f70a43f699b5-istio-token\" (UniqueName: \"kubernetes.io/projected/2ae22b83-f2b9-47c9-b51b-f70a43f699b5-istio-token\") pod \"2ae22b83-f2b9-47c9-b51b-f70a43f699b5\" (UID: \"2ae22b83-f2b9-47c9-b51b-f70a43f699b5\") " Oct 30 14:28:33.880348 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:33.880220 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2ae22b83-f2b9-47c9-b51b-f70a43f699b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ae22b83-f2b9-47c9-b51b-f70a43f699b5-istio-podinfo\") pod \"2ae22b83-f2b9-47c9-b51b-f70a43f699b5\" (UID: \"2ae22b83-f2b9-47c9-b51b-f70a43f699b5\") " Oct 30 14:28:33.880348 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:33.880254 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ae22b83-f2b9-47c9-b51b-f70a43f699b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ae22b83-f2b9-47c9-b51b-f70a43f699b5-istio-envoy\") pod \"2ae22b83-f2b9-47c9-b51b-f70a43f699b5\" (UID: \"2ae22b83-f2b9-47c9-b51b-f70a43f699b5\") " Oct 30 14:28:33.880348 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:33.880272 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae22b83-f2b9-47c9-b51b-f70a43f699b5-credential-socket" pod "2ae22b83-f2b9-47c9-b51b-f70a43f699b5" (UID: "2ae22b83-f2b9-47c9-b51b-f70a43f699b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:28:33.880348 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:33.880325 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ae22b83-f2b9-47c9-b51b-f70a43f699b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ae22b83-f2b9-47c9-b51b-f70a43f699b5-istiod-ca-cert\") pod \"2ae22b83-f2b9-47c9-b51b-f70a43f699b5\" (UID: \"2ae22b83-f2b9-47c9-b51b-f70a43f699b5\") " Oct 30 14:28:33.880592 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:33.880444 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ae22b83-f2b9-47c9-b51b-f70a43f699b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ae22b83-f2b9-47c9-b51b-f70a43f699b5-istio-data\") pod \"2ae22b83-f2b9-47c9-b51b-f70a43f699b5\" (UID: \"2ae22b83-f2b9-47c9-b51b-f70a43f699b5\") " Oct 30 14:28:33.880886 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:33.880857 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae22b83-f2b9-47c9-b51b-f70a43f699b5-workload-certs" pod "2ae22b83-f2b9-47c9-b51b-f70a43f699b5" (UID: "2ae22b83-f2b9-47c9-b51b-f70a43f699b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:28:33.881095 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:33.880859 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ae22b83-f2b9-47c9-b51b-f70a43f699b5-istiod-ca-cert" pod "2ae22b83-f2b9-47c9-b51b-f70a43f699b5" (UID: "2ae22b83-f2b9-47c9-b51b-f70a43f699b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:28:33.881240 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:33.881078 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae22b83-f2b9-47c9-b51b-f70a43f699b5-istio-data" pod "2ae22b83-f2b9-47c9-b51b-f70a43f699b5" (UID: "2ae22b83-f2b9-47c9-b51b-f70a43f699b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:28:33.881489 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:33.881473 4982 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ae22b83-f2b9-47c9-b51b-f70a43f699b5-istio-data\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:28:33.881604 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:33.881592 4982 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae22b83-f2b9-47c9-b51b-f70a43f699b5-credential-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:28:33.881736 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:33.881724 4982 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ae22b83-f2b9-47c9-b51b-f70a43f699b5-workload-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:28:33.881831 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:33.881821 4982 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ae22b83-f2b9-47c9-b51b-f70a43f699b5-istiod-ca-cert\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:28:33.881990 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:33.881959 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae22b83-f2b9-47c9-b51b-f70a43f699b5-workload-socket" pod "2ae22b83-f2b9-47c9-b51b-f70a43f699b5" (UID: "2ae22b83-f2b9-47c9-b51b-f70a43f699b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:28:33.882855 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:33.882827 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ae22b83-f2b9-47c9-b51b-f70a43f699b5-istio-token" pod "2ae22b83-f2b9-47c9-b51b-f70a43f699b5" (UID: "2ae22b83-f2b9-47c9-b51b-f70a43f699b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:28:33.882957 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:33.882893 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae22b83-f2b9-47c9-b51b-f70a43f699b5-istio-envoy" pod "2ae22b83-f2b9-47c9-b51b-f70a43f699b5" (UID: "2ae22b83-f2b9-47c9-b51b-f70a43f699b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:28:33.883008 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:33.882977 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ae22b83-f2b9-47c9-b51b-f70a43f699b5-istio-podinfo" pod "2ae22b83-f2b9-47c9-b51b-f70a43f699b5" (UID: "2ae22b83-f2b9-47c9-b51b-f70a43f699b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:28:33.883358 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:33.883332 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ae22b83-f2b9-47c9-b51b-f70a43f699b5-kube-api-access-sjmrk" pod "2ae22b83-f2b9-47c9-b51b-f70a43f699b5" (UID: "2ae22b83-f2b9-47c9-b51b-f70a43f699b5"). InnerVolumeSpecName "kube-api-access-sjmrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:28:33.984005 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:33.983962 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjmrk\" (UniqueName: \"kubernetes.io/projected/2ae22b83-f2b9-47c9-b51b-f70a43f699b5-kube-api-access-sjmrk\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:28:33.984005 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:33.984000 4982 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae22b83-f2b9-47c9-b51b-f70a43f699b5-workload-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:28:33.984276 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:33.984016 4982 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ae22b83-f2b9-47c9-b51b-f70a43f699b5-istio-token\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:28:33.984276 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:33.984029 4982 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ae22b83-f2b9-47c9-b51b-f70a43f699b5-istio-podinfo\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:28:33.984276 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:33.984043 4982 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ae22b83-f2b9-47c9-b51b-f70a43f699b5-istio-envoy\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:28:34.488984 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:34.488949 4982 generic.go:356] "Generic (PLEG): container finished" podID="2ae22b83-f2b9-47c9-b51b-f70a43f699b5" containerID="8ff6d043b111d16c1d416b1b59c2487fd3cf6462c7087b77548acf73b286ed95" exitCode=0 Oct 30 14:28:34.489138 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:34.489009 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp" Oct 30 14:28:34.489138 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:34.489028 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp" event={"ID":"2ae22b83-f2b9-47c9-b51b-f70a43f699b5","Type":"ContainerDied","Data":"8ff6d043b111d16c1d416b1b59c2487fd3cf6462c7087b77548acf73b286ed95"} Oct 30 14:28:34.489138 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:34.489084 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp" event={"ID":"2ae22b83-f2b9-47c9-b51b-f70a43f699b5","Type":"ContainerDied","Data":"6ea1d8ed918c17e3f334e5ec877477ec529e2294c8a51cb3c51f4703fc6455b1"} Oct 30 14:28:34.489138 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:34.489109 4982 scope.go:122] "RemoveContainer" containerID="8ff6d043b111d16c1d416b1b59c2487fd3cf6462c7087b77548acf73b286ed95" Oct 30 14:28:34.500837 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:34.500702 4982 scope.go:122] "RemoveContainer" containerID="8ff6d043b111d16c1d416b1b59c2487fd3cf6462c7087b77548acf73b286ed95" Oct 30 14:28:34.501104 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:28:34.501082 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ff6d043b111d16c1d416b1b59c2487fd3cf6462c7087b77548acf73b286ed95\": container with ID starting with 8ff6d043b111d16c1d416b1b59c2487fd3cf6462c7087b77548acf73b286ed95 not found: ID does not exist" containerID="8ff6d043b111d16c1d416b1b59c2487fd3cf6462c7087b77548acf73b286ed95" Oct 30 14:28:34.501170 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:34.501112 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ff6d043b111d16c1d416b1b59c2487fd3cf6462c7087b77548acf73b286ed95"} err="failed to get container status \"8ff6d043b111d16c1d416b1b59c2487fd3cf6462c7087b77548acf73b286ed95\": rpc error: code = NotFound desc = could not find container \"8ff6d043b111d16c1d416b1b59c2487fd3cf6462c7087b77548acf73b286ed95\": container with ID starting with 8ff6d043b111d16c1d416b1b59c2487fd3cf6462c7087b77548acf73b286ed95 not found: ID does not exist" Oct 30 14:28:34.561372 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:34.561338 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp"] Oct 30 14:28:34.574569 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:34.574528 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-frblp"] Oct 30 14:28:35.690019 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:35.689987 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:28:35.833405 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:35.833329 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:28:35.983667 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:35.983556 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:28:36.402032 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:36.401995 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ae22b83-f2b9-47c9-b51b-f70a43f699b5" path="/var/lib/kubelet/pods/2ae22b83-f2b9-47c9-b51b-f70a43f699b5/volumes" Oct 30 14:28:37.768082 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:37.768040 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:28:39.757170 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:39.757131 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:28:41.812272 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:41.812229 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:28:43.670393 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:43.670356 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:28:45.052140 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:45.052104 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:28:45.527820 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:45.527763 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:28:47.459205 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:47.459165 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:28:49.324018 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:49.323982 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:28:50.756185 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:50.756148 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ntlqq_dc967875-1546-40bb-8376-fde09416c204/console-operator/2.log" Oct 30 14:28:50.761477 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:50.761441 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhxjf_aca47fe8-7068-4743-b6ab-283ee0874f66/ovn-acl-logging/0.log" Oct 30 14:28:50.761676 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:50.761624 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ntlqq_dc967875-1546-40bb-8376-fde09416c204/console-operator/2.log" Oct 30 14:28:50.766724 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:50.766701 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhxjf_aca47fe8-7068-4743-b6ab-283ee0874f66/ovn-acl-logging/0.log" Oct 30 14:28:51.238228 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:51.238197 4982 scope.go:122] "RemoveContainer" containerID="dfe3e682b67b312c010fed9debc7182ee3ef806744ae3d6313b8aa9f3fb888b0" Oct 30 14:28:51.261671 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:51.261618 4982 scope.go:122] "RemoveContainer" containerID="0517059342f6695688d9c422acc67f45aa1a913db0540c180b94530f916d2c98" Oct 30 14:28:51.341532 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:51.341478 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:28:53.170613 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:53.170574 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:28:55.019628 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:55.019591 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:28:56.532670 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:56.532620 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:28:56.806692 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:56.806569 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-648d5f-lrpgp"] Oct 30 14:28:56.807308 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:56.807288 4982 cpu_manager.go:414] "containerMap: removing container" podUID="2ae22b83-f2b9-47c9-b51b-f70a43f699b5" containerName="istio-proxy" Oct 30 14:28:56.807308 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:56.807306 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ae22b83-f2b9-47c9-b51b-f70a43f699b5" containerName="istio-proxy" Oct 30 14:28:56.807488 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:56.807328 4982 cpu_manager.go:414] "containerMap: removing container" podUID="b2b1bddb-21f5-4576-b5cb-ae696fc925f4" containerName="revoke-keys" Oct 30 14:28:56.807488 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:56.807334 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2b1bddb-21f5-4576-b5cb-ae696fc925f4" containerName="revoke-keys" Oct 30 14:28:56.807488 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:56.807432 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2b1bddb-21f5-4576-b5cb-ae696fc925f4" containerName="revoke-keys" Oct 30 14:28:56.807488 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:56.807449 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ae22b83-f2b9-47c9-b51b-f70a43f699b5" containerName="istio-proxy" Oct 30 14:28:56.812717 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:56.812696 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-648d5f-lrpgp" Oct 30 14:28:56.831035 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:56.831010 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-648d5f-lrpgp"] Oct 30 14:28:56.840283 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:56.840258 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:28:56.840507 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:56.840490 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:28:56.926436 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:56.926402 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r22gz\" (UniqueName: \"kubernetes.io/projected/a0a42b17-8c26-4e05-af89-7c2aaede3d9e-kube-api-access-r22gz\") pod \"maas-api-revoke-keys-e2e-inf-a-648d5f-lrpgp\" (UID: \"a0a42b17-8c26-4e05-af89-7c2aaede3d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-648d5f-lrpgp" Oct 30 14:28:56.926606 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:56.926569 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0a42b17-8c26-4e05-af89-7c2aaede3d9e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-648d5f-lrpgp\" (UID: \"a0a42b17-8c26-4e05-af89-7c2aaede3d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-648d5f-lrpgp" Oct 30 14:28:56.926606 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:56.926597 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0a42b17-8c26-4e05-af89-7c2aaede3d9e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-648d5f-lrpgp\" (UID: \"a0a42b17-8c26-4e05-af89-7c2aaede3d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-648d5f-lrpgp" Oct 30 14:28:57.028442 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:57.028400 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r22gz\" (UniqueName: \"kubernetes.io/projected/a0a42b17-8c26-4e05-af89-7c2aaede3d9e-kube-api-access-r22gz\") pod \"maas-api-revoke-keys-e2e-inf-a-648d5f-lrpgp\" (UID: \"a0a42b17-8c26-4e05-af89-7c2aaede3d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-648d5f-lrpgp" Oct 30 14:28:57.028622 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:57.028536 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0a42b17-8c26-4e05-af89-7c2aaede3d9e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-648d5f-lrpgp\" (UID: \"a0a42b17-8c26-4e05-af89-7c2aaede3d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-648d5f-lrpgp" Oct 30 14:28:57.028622 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:57.028566 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0a42b17-8c26-4e05-af89-7c2aaede3d9e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-648d5f-lrpgp\" (UID: \"a0a42b17-8c26-4e05-af89-7c2aaede3d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-648d5f-lrpgp" Oct 30 14:28:57.029288 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:57.029260 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0a42b17-8c26-4e05-af89-7c2aaede3d9e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-648d5f-lrpgp\" (UID: \"a0a42b17-8c26-4e05-af89-7c2aaede3d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-648d5f-lrpgp" Oct 30 14:28:57.030820 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:57.030799 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0a42b17-8c26-4e05-af89-7c2aaede3d9e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-648d5f-lrpgp\" (UID: \"a0a42b17-8c26-4e05-af89-7c2aaede3d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-648d5f-lrpgp" Oct 30 14:28:57.032908 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:57.032883 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:28:57.058345 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:57.058271 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r22gz\" (UniqueName: \"kubernetes.io/projected/a0a42b17-8c26-4e05-af89-7c2aaede3d9e-kube-api-access-r22gz\") pod \"maas-api-revoke-keys-e2e-inf-a-648d5f-lrpgp\" (UID: \"a0a42b17-8c26-4e05-af89-7c2aaede3d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-648d5f-lrpgp" Oct 30 14:28:57.129598 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:57.129565 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-648d5f-lrpgp" Oct 30 14:28:57.437807 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:28:57.437779 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0a42b17_8c26_4e05_af89_7c2aaede3d9e.slice/crio-6cf000f3db5b6d93ef6573130d51e275a76d1a95bd7ec8fe232f259e522f95d7 WatchSource:0}: Error finding container 6cf000f3db5b6d93ef6573130d51e275a76d1a95bd7ec8fe232f259e522f95d7: Status 404 returned error can't find the container with id 6cf000f3db5b6d93ef6573130d51e275a76d1a95bd7ec8fe232f259e522f95d7 Oct 30 14:28:57.439669 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:57.439208 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-648d5f-lrpgp"] Oct 30 14:28:57.690241 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:57.690167 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-648d5f-lrpgp" event={"ID":"a0a42b17-8c26-4e05-af89-7c2aaede3d9e","Type":"ContainerStarted","Data":"6cf000f3db5b6d93ef6573130d51e275a76d1a95bd7ec8fe232f259e522f95d7"} Oct 30 14:28:58.012899 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:28:58.012814 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:28:58.466389 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:28:58.466356 4982 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0a42b17_8c26_4e05_af89_7c2aaede3d9e.slice/crio-conmon-15b3d5c0581f12e08adf710a308653770fe2fb7144946fc36b4b667899d96850.scope\": RecentStats: unable to find data in memory cache]" Oct 30 14:28:58.699125 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:58.699046 4982 generic.go:356] "Generic (PLEG): container finished" podID="a0a42b17-8c26-4e05-af89-7c2aaede3d9e" containerID="15b3d5c0581f12e08adf710a308653770fe2fb7144946fc36b4b667899d96850" exitCode=0 Oct 30 14:28:58.699537 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:58.699129 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-648d5f-lrpgp" event={"ID":"a0a42b17-8c26-4e05-af89-7c2aaede3d9e","Type":"ContainerDied","Data":"15b3d5c0581f12e08adf710a308653770fe2fb7144946fc36b4b667899d96850"} Oct 30 14:28:59.179297 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:59.179265 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:28:59.877019 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:59.876993 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-648d5f-lrpgp" Oct 30 14:28:59.964344 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:59.964229 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0a42b17-8c26-4e05-af89-7c2aaede3d9e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0a42b17-8c26-4e05-af89-7c2aaede3d9e-maas-api-xks-ca\") pod \"a0a42b17-8c26-4e05-af89-7c2aaede3d9e\" (UID: \"a0a42b17-8c26-4e05-af89-7c2aaede3d9e\") " Oct 30 14:28:59.964344 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:59.964271 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0a42b17-8c26-4e05-af89-7c2aaede3d9e-kube-api-access-r22gz\" (UniqueName: \"kubernetes.io/projected/a0a42b17-8c26-4e05-af89-7c2aaede3d9e-kube-api-access-r22gz\") pod \"a0a42b17-8c26-4e05-af89-7c2aaede3d9e\" (UID: \"a0a42b17-8c26-4e05-af89-7c2aaede3d9e\") " Oct 30 14:28:59.964659 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:59.964626 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0a42b17-8c26-4e05-af89-7c2aaede3d9e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0a42b17-8c26-4e05-af89-7c2aaede3d9e-maas-api-openshift-ca\") pod \"a0a42b17-8c26-4e05-af89-7c2aaede3d9e\" (UID: \"a0a42b17-8c26-4e05-af89-7c2aaede3d9e\") " Oct 30 14:28:59.965022 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:59.964993 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0a42b17-8c26-4e05-af89-7c2aaede3d9e-maas-api-openshift-ca" pod "a0a42b17-8c26-4e05-af89-7c2aaede3d9e" (UID: "a0a42b17-8c26-4e05-af89-7c2aaede3d9e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:28:59.965285 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:59.965259 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0a42b17-8c26-4e05-af89-7c2aaede3d9e-maas-api-openshift-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:28:59.966858 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:59.966836 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0a42b17-8c26-4e05-af89-7c2aaede3d9e-maas-api-xks-ca" pod "a0a42b17-8c26-4e05-af89-7c2aaede3d9e" (UID: "a0a42b17-8c26-4e05-af89-7c2aaede3d9e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:28:59.966961 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:28:59.966878 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0a42b17-8c26-4e05-af89-7c2aaede3d9e-kube-api-access-r22gz" pod "a0a42b17-8c26-4e05-af89-7c2aaede3d9e" (UID: "a0a42b17-8c26-4e05-af89-7c2aaede3d9e"). InnerVolumeSpecName "kube-api-access-r22gz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:29:00.066547 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:00.066503 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0a42b17-8c26-4e05-af89-7c2aaede3d9e-maas-api-xks-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:29:00.066547 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:00.066532 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r22gz\" (UniqueName: \"kubernetes.io/projected/a0a42b17-8c26-4e05-af89-7c2aaede3d9e-kube-api-access-r22gz\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:29:00.714253 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:00.714179 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-648d5f-lrpgp" Oct 30 14:29:00.714395 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:00.714184 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-648d5f-lrpgp" event={"ID":"a0a42b17-8c26-4e05-af89-7c2aaede3d9e","Type":"ContainerDied","Data":"6cf000f3db5b6d93ef6573130d51e275a76d1a95bd7ec8fe232f259e522f95d7"} Oct 30 14:29:00.714395 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:00.714322 4982 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6cf000f3db5b6d93ef6573130d51e275a76d1a95bd7ec8fe232f259e522f95d7" Oct 30 14:29:01.333334 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:01.333286 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:29:03.344184 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:03.344145 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:29:05.211322 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:05.211287 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:29:07.057318 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:07.057277 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:29:07.151392 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:07.151358 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-56mz2"] Oct 30 14:29:07.151583 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:07.151561 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-56mz2" podUID="c389875f-cfd2-4fe3-b659-d4f96c6b549f" containerName="istio-proxy" containerID="cri-o://b7722336c6102861fa699a5dc63bdc1df944baf7c9b5948ede06a4b61c3e1c1a" gracePeriod=30 Oct 30 14:29:07.274281 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:07.274248 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-648d5f-54b7c56458-79pb4"] Oct 30 14:29:07.465083 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:07.465004 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:29:07.611670 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:07.611622 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:29:08.968812 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:08.968776 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1d3696-9b5954949-dfgdl"] Oct 30 14:29:08.969425 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:08.969411 4982 cpu_manager.go:414] "containerMap: removing container" podUID="a0a42b17-8c26-4e05-af89-7c2aaede3d9e" containerName="revoke-keys" Oct 30 14:29:08.969473 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:08.969426 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0a42b17-8c26-4e05-af89-7c2aaede3d9e" containerName="revoke-keys" Oct 30 14:29:08.969550 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:08.969541 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0a42b17-8c26-4e05-af89-7c2aaede3d9e" containerName="revoke-keys" Oct 30 14:29:08.973396 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:08.973377 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d3696-9b5954949-dfgdl" Oct 30 14:29:08.995941 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:08.995910 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1d3696-9b5954949-dfgdl"] Oct 30 14:29:09.003686 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:09.003631 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-1d3696-serving-cert\"" Oct 30 14:29:09.092706 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:09.092676 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:29:09.097091 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:09.097063 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z59n\" (UniqueName: \"kubernetes.io/projected/180b2f0c-daa7-4a34-9a5a-bb4fa79e8bcd-kube-api-access-2z59n\") pod \"maas-api-e2e-inf-b-1d3696-9b5954949-dfgdl\" (UID: \"180b2f0c-daa7-4a34-9a5a-bb4fa79e8bcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d3696-9b5954949-dfgdl" Oct 30 14:29:09.097202 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:09.097184 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/180b2f0c-daa7-4a34-9a5a-bb4fa79e8bcd-maas-api-tls\") pod \"maas-api-e2e-inf-b-1d3696-9b5954949-dfgdl\" (UID: \"180b2f0c-daa7-4a34-9a5a-bb4fa79e8bcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d3696-9b5954949-dfgdl" Oct 30 14:29:09.097256 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:09.097244 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/180b2f0c-daa7-4a34-9a5a-bb4fa79e8bcd-metrics-certs\") pod \"maas-api-e2e-inf-b-1d3696-9b5954949-dfgdl\" (UID: \"180b2f0c-daa7-4a34-9a5a-bb4fa79e8bcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d3696-9b5954949-dfgdl" Oct 30 14:29:09.097411 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:09.097386 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/180b2f0c-daa7-4a34-9a5a-bb4fa79e8bcd-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1d3696-9b5954949-dfgdl\" (UID: \"180b2f0c-daa7-4a34-9a5a-bb4fa79e8bcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d3696-9b5954949-dfgdl" Oct 30 14:29:09.203447 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:09.203404 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z59n\" (UniqueName: \"kubernetes.io/projected/180b2f0c-daa7-4a34-9a5a-bb4fa79e8bcd-kube-api-access-2z59n\") pod \"maas-api-e2e-inf-b-1d3696-9b5954949-dfgdl\" (UID: \"180b2f0c-daa7-4a34-9a5a-bb4fa79e8bcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d3696-9b5954949-dfgdl" Oct 30 14:29:09.203821 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:09.203795 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/180b2f0c-daa7-4a34-9a5a-bb4fa79e8bcd-maas-api-tls\") pod \"maas-api-e2e-inf-b-1d3696-9b5954949-dfgdl\" (UID: \"180b2f0c-daa7-4a34-9a5a-bb4fa79e8bcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d3696-9b5954949-dfgdl" Oct 30 14:29:09.204027 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:09.204007 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/180b2f0c-daa7-4a34-9a5a-bb4fa79e8bcd-metrics-certs\") pod \"maas-api-e2e-inf-b-1d3696-9b5954949-dfgdl\" (UID: \"180b2f0c-daa7-4a34-9a5a-bb4fa79e8bcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d3696-9b5954949-dfgdl" Oct 30 14:29:09.204189 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:09.204173 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/180b2f0c-daa7-4a34-9a5a-bb4fa79e8bcd-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1d3696-9b5954949-dfgdl\" (UID: \"180b2f0c-daa7-4a34-9a5a-bb4fa79e8bcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d3696-9b5954949-dfgdl" Oct 30 14:29:09.210630 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:09.210587 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/180b2f0c-daa7-4a34-9a5a-bb4fa79e8bcd-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1d3696-9b5954949-dfgdl\" (UID: \"180b2f0c-daa7-4a34-9a5a-bb4fa79e8bcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d3696-9b5954949-dfgdl" Oct 30 14:29:09.233318 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:09.233246 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/180b2f0c-daa7-4a34-9a5a-bb4fa79e8bcd-metrics-certs\") pod \"maas-api-e2e-inf-b-1d3696-9b5954949-dfgdl\" (UID: \"180b2f0c-daa7-4a34-9a5a-bb4fa79e8bcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d3696-9b5954949-dfgdl" Oct 30 14:29:09.234190 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:09.234160 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/180b2f0c-daa7-4a34-9a5a-bb4fa79e8bcd-maas-api-tls\") pod \"maas-api-e2e-inf-b-1d3696-9b5954949-dfgdl\" (UID: \"180b2f0c-daa7-4a34-9a5a-bb4fa79e8bcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d3696-9b5954949-dfgdl" Oct 30 14:29:09.236061 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:09.236038 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z59n\" (UniqueName: \"kubernetes.io/projected/180b2f0c-daa7-4a34-9a5a-bb4fa79e8bcd-kube-api-access-2z59n\") pod \"maas-api-e2e-inf-b-1d3696-9b5954949-dfgdl\" (UID: \"180b2f0c-daa7-4a34-9a5a-bb4fa79e8bcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d3696-9b5954949-dfgdl" Oct 30 14:29:09.243948 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:09.243904 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:29:09.289302 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:09.289277 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d3696-9b5954949-dfgdl" Oct 30 14:29:09.610596 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:09.610562 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1d3696-9b5954949-dfgdl"] Oct 30 14:29:09.611328 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:29:09.611300 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod180b2f0c_daa7_4a34_9a5a_bb4fa79e8bcd.slice/crio-ba4cdcf53f19679c9fedde196be1fbad84d6198794958b5722faaa3657c4c328 WatchSource:0}: Error finding container ba4cdcf53f19679c9fedde196be1fbad84d6198794958b5722faaa3657c4c328: Status 404 returned error can't find the container with id ba4cdcf53f19679c9fedde196be1fbad84d6198794958b5722faaa3657c4c328 Oct 30 14:29:09.784963 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:09.784916 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d3696-9b5954949-dfgdl" event={"ID":"180b2f0c-daa7-4a34-9a5a-bb4fa79e8bcd","Type":"ContainerStarted","Data":"ba4cdcf53f19679c9fedde196be1fbad84d6198794958b5722faaa3657c4c328"} Oct 30 14:29:10.794880 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:10.794833 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d3696-9b5954949-dfgdl" event={"ID":"180b2f0c-daa7-4a34-9a5a-bb4fa79e8bcd","Type":"ContainerStarted","Data":"fdb867d87f41bc8f76c047aa14ec48658aab16986209ffdb3f4789112a218cc6"} Oct 30 14:29:10.795282 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:10.794898 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d3696-9b5954949-dfgdl" Oct 30 14:29:10.844268 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:10.844219 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d3696-9b5954949-dfgdl" podStartSLOduration=2.505829341 podStartE2EDuration="2.844202667s" totalImagesPullingTime="338.373326ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:29:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:29:10.84156666 +0000 UTC m=+2420.585700700" watchObservedRunningTime="2026-10-30 14:29:10.844202667 +0000 UTC m=+2420.588336707" Oct 30 14:29:11.028270 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:11.028226 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:29:12.275120 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:12.275065 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-648d5f-54b7c56458-79pb4" podUID="60f102c4-46ef-4dbb-8eba-cec0e78a3b3e" containerName="payload-processing" containerID="cri-o://5758d33bc72a8921fc1b8291c874b2650322cc4a087e611e4740b0e75ea71a9e" gracePeriod=25 Oct 30 14:29:12.449929 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:12.449896 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-56mz2" Oct 30 14:29:12.463833 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:12.463801 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c389875f-cfd2-4fe3-b659-d4f96c6b549f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c389875f-cfd2-4fe3-b659-d4f96c6b549f-istiod-ca-cert\") pod \"c389875f-cfd2-4fe3-b659-d4f96c6b549f\" (UID: \"c389875f-cfd2-4fe3-b659-d4f96c6b549f\") " Oct 30 14:29:12.463983 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:12.463868 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c389875f-cfd2-4fe3-b659-d4f96c6b549f-kube-api-access-87cmz\" (UniqueName: \"kubernetes.io/projected/c389875f-cfd2-4fe3-b659-d4f96c6b549f-kube-api-access-87cmz\") pod \"c389875f-cfd2-4fe3-b659-d4f96c6b549f\" (UID: \"c389875f-cfd2-4fe3-b659-d4f96c6b549f\") " Oct 30 14:29:12.463983 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:12.463921 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c389875f-cfd2-4fe3-b659-d4f96c6b549f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c389875f-cfd2-4fe3-b659-d4f96c6b549f-istio-data\") pod \"c389875f-cfd2-4fe3-b659-d4f96c6b549f\" (UID: \"c389875f-cfd2-4fe3-b659-d4f96c6b549f\") " Oct 30 14:29:12.464103 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:12.464045 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c389875f-cfd2-4fe3-b659-d4f96c6b549f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c389875f-cfd2-4fe3-b659-d4f96c6b549f-istio-envoy\") pod \"c389875f-cfd2-4fe3-b659-d4f96c6b549f\" (UID: \"c389875f-cfd2-4fe3-b659-d4f96c6b549f\") " Oct 30 14:29:12.464165 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:12.464096 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c389875f-cfd2-4fe3-b659-d4f96c6b549f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c389875f-cfd2-4fe3-b659-d4f96c6b549f-istio-podinfo\") pod \"c389875f-cfd2-4fe3-b659-d4f96c6b549f\" (UID: \"c389875f-cfd2-4fe3-b659-d4f96c6b549f\") " Oct 30 14:29:12.464237 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:12.464161 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c389875f-cfd2-4fe3-b659-d4f96c6b549f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c389875f-cfd2-4fe3-b659-d4f96c6b549f-credential-socket\") pod \"c389875f-cfd2-4fe3-b659-d4f96c6b549f\" (UID: \"c389875f-cfd2-4fe3-b659-d4f96c6b549f\") " Oct 30 14:29:12.464237 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:12.464209 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c389875f-cfd2-4fe3-b659-d4f96c6b549f-istio-data" pod "c389875f-cfd2-4fe3-b659-d4f96c6b549f" (UID: "c389875f-cfd2-4fe3-b659-d4f96c6b549f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:29:12.464349 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:12.464246 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c389875f-cfd2-4fe3-b659-d4f96c6b549f-istiod-ca-cert" pod "c389875f-cfd2-4fe3-b659-d4f96c6b549f" (UID: "c389875f-cfd2-4fe3-b659-d4f96c6b549f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:29:12.464349 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:12.464285 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c389875f-cfd2-4fe3-b659-d4f96c6b549f-istio-token\" (UniqueName: \"kubernetes.io/projected/c389875f-cfd2-4fe3-b659-d4f96c6b549f-istio-token\") pod \"c389875f-cfd2-4fe3-b659-d4f96c6b549f\" (UID: \"c389875f-cfd2-4fe3-b659-d4f96c6b549f\") " Oct 30 14:29:12.464478 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:12.464386 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c389875f-cfd2-4fe3-b659-d4f96c6b549f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c389875f-cfd2-4fe3-b659-d4f96c6b549f-workload-socket\") pod \"c389875f-cfd2-4fe3-b659-d4f96c6b549f\" (UID: \"c389875f-cfd2-4fe3-b659-d4f96c6b549f\") " Oct 30 14:29:12.464478 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:12.464468 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c389875f-cfd2-4fe3-b659-d4f96c6b549f-credential-socket" pod "c389875f-cfd2-4fe3-b659-d4f96c6b549f" (UID: "c389875f-cfd2-4fe3-b659-d4f96c6b549f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:29:12.464610 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:12.464473 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c389875f-cfd2-4fe3-b659-d4f96c6b549f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c389875f-cfd2-4fe3-b659-d4f96c6b549f-workload-certs\") pod \"c389875f-cfd2-4fe3-b659-d4f96c6b549f\" (UID: \"c389875f-cfd2-4fe3-b659-d4f96c6b549f\") " Oct 30 14:29:12.464724 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:12.464703 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c389875f-cfd2-4fe3-b659-d4f96c6b549f-workload-certs" pod "c389875f-cfd2-4fe3-b659-d4f96c6b549f" (UID: "c389875f-cfd2-4fe3-b659-d4f96c6b549f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:29:12.464814 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:12.464732 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c389875f-cfd2-4fe3-b659-d4f96c6b549f-workload-socket" pod "c389875f-cfd2-4fe3-b659-d4f96c6b549f" (UID: "c389875f-cfd2-4fe3-b659-d4f96c6b549f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:29:12.465227 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:12.465202 4982 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c389875f-cfd2-4fe3-b659-d4f96c6b549f-workload-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:29:12.465330 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:12.465233 4982 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c389875f-cfd2-4fe3-b659-d4f96c6b549f-workload-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:29:12.465330 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:12.465251 4982 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c389875f-cfd2-4fe3-b659-d4f96c6b549f-istiod-ca-cert\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:29:12.465330 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:12.465264 4982 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c389875f-cfd2-4fe3-b659-d4f96c6b549f-istio-data\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:29:12.465330 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:12.465277 4982 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c389875f-cfd2-4fe3-b659-d4f96c6b549f-credential-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:29:12.466546 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:12.466521 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c389875f-cfd2-4fe3-b659-d4f96c6b549f-kube-api-access-87cmz" pod "c389875f-cfd2-4fe3-b659-d4f96c6b549f" (UID: "c389875f-cfd2-4fe3-b659-d4f96c6b549f"). InnerVolumeSpecName "kube-api-access-87cmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:29:12.466546 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:12.466533 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c389875f-cfd2-4fe3-b659-d4f96c6b549f-istio-envoy" pod "c389875f-cfd2-4fe3-b659-d4f96c6b549f" (UID: "c389875f-cfd2-4fe3-b659-d4f96c6b549f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:29:12.466687 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:12.466616 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c389875f-cfd2-4fe3-b659-d4f96c6b549f-istio-podinfo" pod "c389875f-cfd2-4fe3-b659-d4f96c6b549f" (UID: "c389875f-cfd2-4fe3-b659-d4f96c6b549f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:29:12.466869 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:12.466850 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c389875f-cfd2-4fe3-b659-d4f96c6b549f-istio-token" pod "c389875f-cfd2-4fe3-b659-d4f96c6b549f" (UID: "c389875f-cfd2-4fe3-b659-d4f96c6b549f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:29:12.566344 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:12.566289 4982 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c389875f-cfd2-4fe3-b659-d4f96c6b549f-istio-token\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:29:12.566344 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:12.566337 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87cmz\" (UniqueName: \"kubernetes.io/projected/c389875f-cfd2-4fe3-b659-d4f96c6b549f-kube-api-access-87cmz\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:29:12.566344 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:12.566352 4982 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c389875f-cfd2-4fe3-b659-d4f96c6b549f-istio-envoy\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:29:12.566655 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:12.566364 4982 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c389875f-cfd2-4fe3-b659-d4f96c6b549f-istio-podinfo\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:29:12.813084 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:12.813047 4982 generic.go:356] "Generic (PLEG): container finished" podID="c389875f-cfd2-4fe3-b659-d4f96c6b549f" containerID="b7722336c6102861fa699a5dc63bdc1df944baf7c9b5948ede06a4b61c3e1c1a" exitCode=0 Oct 30 14:29:12.813284 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:12.813109 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-56mz2" Oct 30 14:29:12.813284 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:12.813139 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-56mz2" event={"ID":"c389875f-cfd2-4fe3-b659-d4f96c6b549f","Type":"ContainerDied","Data":"b7722336c6102861fa699a5dc63bdc1df944baf7c9b5948ede06a4b61c3e1c1a"} Oct 30 14:29:12.813284 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:12.813205 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-56mz2" event={"ID":"c389875f-cfd2-4fe3-b659-d4f96c6b549f","Type":"ContainerDied","Data":"6e2cd4ae0615c729c0a254e5d87e9ba172faf345ab5ea9538289ca4e1f1263f5"} Oct 30 14:29:12.813284 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:12.813235 4982 scope.go:122] "RemoveContainer" containerID="b7722336c6102861fa699a5dc63bdc1df944baf7c9b5948ede06a4b61c3e1c1a" Oct 30 14:29:12.825605 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:12.825585 4982 scope.go:122] "RemoveContainer" containerID="b7722336c6102861fa699a5dc63bdc1df944baf7c9b5948ede06a4b61c3e1c1a" Oct 30 14:29:12.825953 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:29:12.825935 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7722336c6102861fa699a5dc63bdc1df944baf7c9b5948ede06a4b61c3e1c1a\": container with ID starting with b7722336c6102861fa699a5dc63bdc1df944baf7c9b5948ede06a4b61c3e1c1a not found: ID does not exist" containerID="b7722336c6102861fa699a5dc63bdc1df944baf7c9b5948ede06a4b61c3e1c1a" Oct 30 14:29:12.826031 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:12.825962 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7722336c6102861fa699a5dc63bdc1df944baf7c9b5948ede06a4b61c3e1c1a"} err="failed to get container status \"b7722336c6102861fa699a5dc63bdc1df944baf7c9b5948ede06a4b61c3e1c1a\": rpc error: code = NotFound desc = could not find container \"b7722336c6102861fa699a5dc63bdc1df944baf7c9b5948ede06a4b61c3e1c1a\": container with ID starting with b7722336c6102861fa699a5dc63bdc1df944baf7c9b5948ede06a4b61c3e1c1a not found: ID does not exist" Oct 30 14:29:12.874866 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:12.874841 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-56mz2"] Oct 30 14:29:12.888258 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:12.888229 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-56mz2"] Oct 30 14:29:13.091602 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:13.091515 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:29:14.401550 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:14.401516 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c389875f-cfd2-4fe3-b659-d4f96c6b549f" path="/var/lib/kubelet/pods/c389875f-cfd2-4fe3-b659-d4f96c6b549f/volumes" Oct 30 14:29:15.034245 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:15.034201 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:29:16.802337 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:16.802294 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:29:16.834943 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:16.833359 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d3696-9b5954949-dfgdl" Oct 30 14:29:18.439332 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:18.439291 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-648d5f-54b7c56458-79pb4" podUID="60f102c4-46ef-4dbb-8eba-cec0e78a3b3e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:29:18.660558 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:18.660517 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:29:20.656105 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:20.656066 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:29:22.619620 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:22.619563 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:29:24.443197 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:24.443156 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:29:26.425322 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:26.425285 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:29:28.014417 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:29:28.014380 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:29:28.260219 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:28.260187 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:29:28.439485 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:28.439441 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-648d5f-54b7c56458-79pb4" podUID="60f102c4-46ef-4dbb-8eba-cec0e78a3b3e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:29:30.117898 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:30.117852 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:29:32.153442 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:32.153391 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:29:32.596184 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:32.596155 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-648d5f-54b7c56458-79pb4" Oct 30 14:29:32.668273 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:32.668240 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60f102c4-46ef-4dbb-8eba-cec0e78a3b3e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60f102c4-46ef-4dbb-8eba-cec0e78a3b3e-plugins-config-volume\") pod \"60f102c4-46ef-4dbb-8eba-cec0e78a3b3e\" (UID: \"60f102c4-46ef-4dbb-8eba-cec0e78a3b3e\") " Oct 30 14:29:32.668480 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:32.668349 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60f102c4-46ef-4dbb-8eba-cec0e78a3b3e-kube-api-access-52mbw\" (UniqueName: \"kubernetes.io/projected/60f102c4-46ef-4dbb-8eba-cec0e78a3b3e-kube-api-access-52mbw\") pod \"60f102c4-46ef-4dbb-8eba-cec0e78a3b3e\" (UID: \"60f102c4-46ef-4dbb-8eba-cec0e78a3b3e\") " Oct 30 14:29:32.668706 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:32.668679 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60f102c4-46ef-4dbb-8eba-cec0e78a3b3e-plugins-config-volume" pod "60f102c4-46ef-4dbb-8eba-cec0e78a3b3e" (UID: "60f102c4-46ef-4dbb-8eba-cec0e78a3b3e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:29:32.669048 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:32.669027 4982 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60f102c4-46ef-4dbb-8eba-cec0e78a3b3e-plugins-config-volume\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:29:32.671148 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:32.671116 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60f102c4-46ef-4dbb-8eba-cec0e78a3b3e-kube-api-access-52mbw" pod "60f102c4-46ef-4dbb-8eba-cec0e78a3b3e" (UID: "60f102c4-46ef-4dbb-8eba-cec0e78a3b3e"). InnerVolumeSpecName "kube-api-access-52mbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:29:32.770478 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:32.770399 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52mbw\" (UniqueName: \"kubernetes.io/projected/60f102c4-46ef-4dbb-8eba-cec0e78a3b3e-kube-api-access-52mbw\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:29:32.792912 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:32.792531 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:29:32.992777 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:32.992743 4982 generic.go:356] "Generic (PLEG): container finished" podID="60f102c4-46ef-4dbb-8eba-cec0e78a3b3e" containerID="5758d33bc72a8921fc1b8291c874b2650322cc4a087e611e4740b0e75ea71a9e" exitCode=0 Oct 30 14:29:32.992976 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:32.992787 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-648d5f-54b7c56458-79pb4" Oct 30 14:29:32.992976 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:32.992834 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-648d5f-54b7c56458-79pb4" event={"ID":"60f102c4-46ef-4dbb-8eba-cec0e78a3b3e","Type":"ContainerDied","Data":"5758d33bc72a8921fc1b8291c874b2650322cc4a087e611e4740b0e75ea71a9e"} Oct 30 14:29:32.992976 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:32.992891 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-648d5f-54b7c56458-79pb4" event={"ID":"60f102c4-46ef-4dbb-8eba-cec0e78a3b3e","Type":"ContainerDied","Data":"932e5d6e60455e30cfe4a8b71f19cec11b4ecbb581fe9cde339f829a5ca3d446"} Oct 30 14:29:32.992976 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:32.992921 4982 scope.go:122] "RemoveContainer" containerID="5758d33bc72a8921fc1b8291c874b2650322cc4a087e611e4740b0e75ea71a9e" Oct 30 14:29:33.005876 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:33.005857 4982 scope.go:122] "RemoveContainer" containerID="5758d33bc72a8921fc1b8291c874b2650322cc4a087e611e4740b0e75ea71a9e" Oct 30 14:29:33.006178 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:29:33.006159 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5758d33bc72a8921fc1b8291c874b2650322cc4a087e611e4740b0e75ea71a9e\": container with ID starting with 5758d33bc72a8921fc1b8291c874b2650322cc4a087e611e4740b0e75ea71a9e not found: ID does not exist" containerID="5758d33bc72a8921fc1b8291c874b2650322cc4a087e611e4740b0e75ea71a9e" Oct 30 14:29:33.006254 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:33.006189 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5758d33bc72a8921fc1b8291c874b2650322cc4a087e611e4740b0e75ea71a9e"} err="failed to get container status \"5758d33bc72a8921fc1b8291c874b2650322cc4a087e611e4740b0e75ea71a9e\": rpc error: code = NotFound desc = could not find container \"5758d33bc72a8921fc1b8291c874b2650322cc4a087e611e4740b0e75ea71a9e\": container with ID starting with 5758d33bc72a8921fc1b8291c874b2650322cc4a087e611e4740b0e75ea71a9e not found: ID does not exist" Oct 30 14:29:33.054925 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:33.054860 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-648d5f-54b7c56458-79pb4"] Oct 30 14:29:33.069887 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:33.069857 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-648d5f-54b7c56458-79pb4"] Oct 30 14:29:33.234023 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:33.233975 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:29:33.627918 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:33.627878 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2"] Oct 30 14:29:33.628885 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:33.628861 4982 cpu_manager.go:414] "containerMap: removing container" podUID="c389875f-cfd2-4fe3-b659-d4f96c6b549f" containerName="istio-proxy" Oct 30 14:29:33.628995 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:33.628886 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c389875f-cfd2-4fe3-b659-d4f96c6b549f" containerName="istio-proxy" Oct 30 14:29:33.628995 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:33.628952 4982 cpu_manager.go:414] "containerMap: removing container" podUID="60f102c4-46ef-4dbb-8eba-cec0e78a3b3e" containerName="payload-processing" Oct 30 14:29:33.628995 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:33.628961 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60f102c4-46ef-4dbb-8eba-cec0e78a3b3e" containerName="payload-processing" Oct 30 14:29:33.629156 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:33.629119 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="c389875f-cfd2-4fe3-b659-d4f96c6b549f" containerName="istio-proxy" Oct 30 14:29:33.629156 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:33.629142 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="60f102c4-46ef-4dbb-8eba-cec0e78a3b3e" containerName="payload-processing" Oct 30 14:29:33.634733 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:33.634708 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2" Oct 30 14:29:33.658826 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:33.658797 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2"] Oct 30 14:29:33.672686 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:33.666024 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-1d6347-openshift-default-dockercfg-6jjln\"" Oct 30 14:29:33.684665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:33.684579 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30019687-440e-44fd-8fd8-0d2902927ca0-istio-data\") pod \"e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2\" (UID: \"30019687-440e-44fd-8fd8-0d2902927ca0\") " pod="openshift-ingress/e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2" Oct 30 14:29:33.684665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:33.684631 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt2nm\" (UniqueName: \"kubernetes.io/projected/30019687-440e-44fd-8fd8-0d2902927ca0-kube-api-access-wt2nm\") pod \"e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2\" (UID: \"30019687-440e-44fd-8fd8-0d2902927ca0\") " pod="openshift-ingress/e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2" Oct 30 14:29:33.684893 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:33.684678 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30019687-440e-44fd-8fd8-0d2902927ca0-workload-socket\") pod \"e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2\" (UID: \"30019687-440e-44fd-8fd8-0d2902927ca0\") " pod="openshift-ingress/e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2" Oct 30 14:29:33.684893 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:33.684754 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30019687-440e-44fd-8fd8-0d2902927ca0-istio-podinfo\") pod \"e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2\" (UID: \"30019687-440e-44fd-8fd8-0d2902927ca0\") " pod="openshift-ingress/e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2" Oct 30 14:29:33.684893 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:33.684783 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30019687-440e-44fd-8fd8-0d2902927ca0-credential-socket\") pod \"e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2\" (UID: \"30019687-440e-44fd-8fd8-0d2902927ca0\") " pod="openshift-ingress/e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2" Oct 30 14:29:33.684893 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:33.684851 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30019687-440e-44fd-8fd8-0d2902927ca0-istiod-ca-cert\") pod \"e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2\" (UID: \"30019687-440e-44fd-8fd8-0d2902927ca0\") " pod="openshift-ingress/e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2" Oct 30 14:29:33.685097 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:33.684925 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30019687-440e-44fd-8fd8-0d2902927ca0-workload-certs\") pod \"e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2\" (UID: \"30019687-440e-44fd-8fd8-0d2902927ca0\") " pod="openshift-ingress/e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2" Oct 30 14:29:33.685097 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:33.685063 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30019687-440e-44fd-8fd8-0d2902927ca0-istio-envoy\") pod \"e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2\" (UID: \"30019687-440e-44fd-8fd8-0d2902927ca0\") " pod="openshift-ingress/e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2" Oct 30 14:29:33.685199 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:33.685148 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30019687-440e-44fd-8fd8-0d2902927ca0-istio-token\") pod \"e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2\" (UID: \"30019687-440e-44fd-8fd8-0d2902927ca0\") " pod="openshift-ingress/e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2" Oct 30 14:29:33.717398 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:33.717368 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:29:33.797673 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:33.797619 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30019687-440e-44fd-8fd8-0d2902927ca0-istio-podinfo\") pod \"e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2\" (UID: \"30019687-440e-44fd-8fd8-0d2902927ca0\") " pod="openshift-ingress/e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2" Oct 30 14:29:33.804845 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:33.804812 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30019687-440e-44fd-8fd8-0d2902927ca0-credential-socket\") pod \"e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2\" (UID: \"30019687-440e-44fd-8fd8-0d2902927ca0\") " pod="openshift-ingress/e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2" Oct 30 14:29:33.805494 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:33.805472 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30019687-440e-44fd-8fd8-0d2902927ca0-istiod-ca-cert\") pod \"e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2\" (UID: \"30019687-440e-44fd-8fd8-0d2902927ca0\") " pod="openshift-ingress/e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2" Oct 30 14:29:33.806480 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:33.806454 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30019687-440e-44fd-8fd8-0d2902927ca0-workload-certs\") pod \"e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2\" (UID: \"30019687-440e-44fd-8fd8-0d2902927ca0\") " pod="openshift-ingress/e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2" Oct 30 14:29:33.816665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:33.807041 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30019687-440e-44fd-8fd8-0d2902927ca0-istio-envoy\") pod \"e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2\" (UID: \"30019687-440e-44fd-8fd8-0d2902927ca0\") " pod="openshift-ingress/e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2" Oct 30 14:29:33.816943 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:33.816913 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30019687-440e-44fd-8fd8-0d2902927ca0-istio-token\") pod \"e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2\" (UID: \"30019687-440e-44fd-8fd8-0d2902927ca0\") " pod="openshift-ingress/e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2" Oct 30 14:29:33.817525 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:33.817503 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30019687-440e-44fd-8fd8-0d2902927ca0-istio-data\") pod \"e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2\" (UID: \"30019687-440e-44fd-8fd8-0d2902927ca0\") " pod="openshift-ingress/e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2" Oct 30 14:29:33.817988 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:33.817969 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt2nm\" (UniqueName: \"kubernetes.io/projected/30019687-440e-44fd-8fd8-0d2902927ca0-kube-api-access-wt2nm\") pod \"e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2\" (UID: \"30019687-440e-44fd-8fd8-0d2902927ca0\") " pod="openshift-ingress/e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2" Oct 30 14:29:33.818452 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:33.818432 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30019687-440e-44fd-8fd8-0d2902927ca0-workload-socket\") pod \"e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2\" (UID: \"30019687-440e-44fd-8fd8-0d2902927ca0\") " pod="openshift-ingress/e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2" Oct 30 14:29:33.819025 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:33.817907 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30019687-440e-44fd-8fd8-0d2902927ca0-istio-data\") pod \"e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2\" (UID: \"30019687-440e-44fd-8fd8-0d2902927ca0\") " pod="openshift-ingress/e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2" Oct 30 14:29:33.819170 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:33.806873 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30019687-440e-44fd-8fd8-0d2902927ca0-workload-certs\") pod \"e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2\" (UID: \"30019687-440e-44fd-8fd8-0d2902927ca0\") " pod="openshift-ingress/e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2" Oct 30 14:29:33.829806 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:33.805382 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30019687-440e-44fd-8fd8-0d2902927ca0-credential-socket\") pod \"e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2\" (UID: \"30019687-440e-44fd-8fd8-0d2902927ca0\") " pod="openshift-ingress/e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2" Oct 30 14:29:33.848543 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:33.818834 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30019687-440e-44fd-8fd8-0d2902927ca0-workload-socket\") pod \"e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2\" (UID: \"30019687-440e-44fd-8fd8-0d2902927ca0\") " pod="openshift-ingress/e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2" Oct 30 14:29:33.848810 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:33.806334 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30019687-440e-44fd-8fd8-0d2902927ca0-istiod-ca-cert\") pod \"e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2\" (UID: \"30019687-440e-44fd-8fd8-0d2902927ca0\") " pod="openshift-ingress/e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2" Oct 30 14:29:33.848939 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:33.831276 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30019687-440e-44fd-8fd8-0d2902927ca0-istio-podinfo\") pod \"e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2\" (UID: \"30019687-440e-44fd-8fd8-0d2902927ca0\") " pod="openshift-ingress/e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2" Oct 30 14:29:33.849180 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:33.849057 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30019687-440e-44fd-8fd8-0d2902927ca0-istio-envoy\") pod \"e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2\" (UID: \"30019687-440e-44fd-8fd8-0d2902927ca0\") " pod="openshift-ingress/e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2" Oct 30 14:29:33.856304 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:33.856265 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt2nm\" (UniqueName: \"kubernetes.io/projected/30019687-440e-44fd-8fd8-0d2902927ca0-kube-api-access-wt2nm\") pod \"e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2\" (UID: \"30019687-440e-44fd-8fd8-0d2902927ca0\") " pod="openshift-ingress/e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2" Oct 30 14:29:33.867709 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:33.863525 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30019687-440e-44fd-8fd8-0d2902927ca0-istio-token\") pod \"e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2\" (UID: \"30019687-440e-44fd-8fd8-0d2902927ca0\") " pod="openshift-ingress/e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2" Oct 30 14:29:33.951322 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:33.951239 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2" Oct 30 14:29:34.227496 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:34.227272 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:29:34.335110 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:29:34.335079 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30019687_440e_44fd_8fd8_0d2902927ca0.slice/crio-213953ad184cc54276d522de79b3ab600a266b45e281138ce27ced41ecf31f96 WatchSource:0}: Error finding container 213953ad184cc54276d522de79b3ab600a266b45e281138ce27ced41ecf31f96: Status 404 returned error can't find the container with id 213953ad184cc54276d522de79b3ab600a266b45e281138ce27ced41ecf31f96 Oct 30 14:29:34.336468 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:34.336436 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2"] Oct 30 14:29:34.338252 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:34.338217 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:29:34.338374 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:34.338281 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:29:34.338374 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:34.338331 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:29:34.387635 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:34.387577 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:29:34.439009 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:34.438355 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60f102c4-46ef-4dbb-8eba-cec0e78a3b3e" path="/var/lib/kubelet/pods/60f102c4-46ef-4dbb-8eba-cec0e78a3b3e/volumes" Oct 30 14:29:34.717843 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:34.717809 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:29:34.862051 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:34.862008 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:29:35.043534 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:35.043446 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2" event={"ID":"30019687-440e-44fd-8fd8-0d2902927ca0","Type":"ContainerStarted","Data":"883719cef92123e240e8eaca0d0210e852634519c879e191779b6e08cd6a87f7"} Oct 30 14:29:35.043534 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:35.043496 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2" event={"ID":"30019687-440e-44fd-8fd8-0d2902927ca0","Type":"ContainerStarted","Data":"213953ad184cc54276d522de79b3ab600a266b45e281138ce27ced41ecf31f96"} Oct 30 14:29:35.118515 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:35.118462 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2" podStartSLOduration=2.118447223 podStartE2EDuration="2.118447223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:29:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:29:35.116336333 +0000 UTC m=+2444.860470372" watchObservedRunningTime="2026-10-30 14:29:35.118447223 +0000 UTC m=+2444.862581263" Oct 30 14:29:35.952015 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:35.951983 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2" Oct 30 14:29:35.958464 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:35.958440 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2" Oct 30 14:29:36.050611 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:36.050585 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2" Oct 30 14:29:36.051543 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:36.051526 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2" Oct 30 14:29:36.224053 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:36.223974 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:29:37.519113 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:37.519062 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:29:37.675298 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:37.675259 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:29:38.245337 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:38.245302 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:29:39.290027 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:39.289992 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:29:39.433076 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:39.433040 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-1d6347-7778f9d655-545gn"] Oct 30 14:29:39.437464 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:39.437437 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-1d6347-7778f9d655-545gn" Oct 30 14:29:39.443545 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:39.443512 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:29:39.457927 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:39.457886 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-1d6347-7778f9d655-545gn"] Oct 30 14:29:39.466568 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:39.466539 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-1d6347\"" Oct 30 14:29:39.466787 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:39.466625 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-1d6347-dockercfg-dfgwf\"" Oct 30 14:29:39.609605 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:39.609567 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4dtf\" (UniqueName: \"kubernetes.io/projected/d15be33a-a221-4357-8b19-609ba138d557-kube-api-access-l4dtf\") pod \"payload-pre-processing-e2e-shared-b-w5-1d6347-7778f9d655-545gn\" (UID: \"d15be33a-a221-4357-8b19-609ba138d557\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-1d6347-7778f9d655-545gn" Oct 30 14:29:39.614863 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:39.614824 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d15be33a-a221-4357-8b19-609ba138d557-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-1d6347-7778f9d655-545gn\" (UID: \"d15be33a-a221-4357-8b19-609ba138d557\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-1d6347-7778f9d655-545gn" Oct 30 14:29:39.721482 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:39.721447 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4dtf\" (UniqueName: \"kubernetes.io/projected/d15be33a-a221-4357-8b19-609ba138d557-kube-api-access-l4dtf\") pod \"payload-pre-processing-e2e-shared-b-w5-1d6347-7778f9d655-545gn\" (UID: \"d15be33a-a221-4357-8b19-609ba138d557\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-1d6347-7778f9d655-545gn" Oct 30 14:29:39.721709 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:39.721512 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d15be33a-a221-4357-8b19-609ba138d557-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-1d6347-7778f9d655-545gn\" (UID: \"d15be33a-a221-4357-8b19-609ba138d557\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-1d6347-7778f9d655-545gn" Oct 30 14:29:39.722285 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:39.722257 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d15be33a-a221-4357-8b19-609ba138d557-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-1d6347-7778f9d655-545gn\" (UID: \"d15be33a-a221-4357-8b19-609ba138d557\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-1d6347-7778f9d655-545gn" Oct 30 14:29:39.747147 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:39.747121 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4dtf\" (UniqueName: \"kubernetes.io/projected/d15be33a-a221-4357-8b19-609ba138d557-kube-api-access-l4dtf\") pod \"payload-pre-processing-e2e-shared-b-w5-1d6347-7778f9d655-545gn\" (UID: \"d15be33a-a221-4357-8b19-609ba138d557\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-1d6347-7778f9d655-545gn" Oct 30 14:29:39.753552 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:39.753526 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-1d6347-7778f9d655-545gn" Oct 30 14:29:40.072124 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:40.072084 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-1d6347-7778f9d655-545gn"] Oct 30 14:29:40.072669 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:29:40.072622 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd15be33a_a221_4357_8b19_609ba138d557.slice/crio-70fb63ebf4e31d8baf60d24b8aa0a4f5617f7888d08236b308538184528210ef WatchSource:0}: Error finding container 70fb63ebf4e31d8baf60d24b8aa0a4f5617f7888d08236b308538184528210ef: Status 404 returned error can't find the container with id 70fb63ebf4e31d8baf60d24b8aa0a4f5617f7888d08236b308538184528210ef Oct 30 14:29:40.241632 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:40.241564 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:29:41.088940 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:41.088888 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-1d6347-7778f9d655-545gn" event={"ID":"d15be33a-a221-4357-8b19-609ba138d557","Type":"ContainerStarted","Data":"5a2145c7f3ea47c29161cb564a34d4be342b22ef2cc1abf97fa0c96662f1a6e1"} Oct 30 14:29:41.089329 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:41.089001 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-1d6347-7778f9d655-545gn" Oct 30 14:29:41.089329 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:41.089025 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-1d6347-7778f9d655-545gn" event={"ID":"d15be33a-a221-4357-8b19-609ba138d557","Type":"ContainerStarted","Data":"70fb63ebf4e31d8baf60d24b8aa0a4f5617f7888d08236b308538184528210ef"} Oct 30 14:29:41.133123 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:41.133068 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-1d6347-7778f9d655-545gn" podStartSLOduration=2.133055669 podStartE2EDuration="2.133055669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:29:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:29:41.132573847 +0000 UTC m=+2450.876707887" watchObservedRunningTime="2026-10-30 14:29:41.133055669 +0000 UTC m=+2450.877189709" Oct 30 14:29:42.058146 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:42.058110 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:29:44.039697 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:44.039659 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:29:44.102386 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:44.102349 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e83f182f-4d9d2"] Oct 30 14:29:44.106936 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:44.106916 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83f182f-4d9d2" Oct 30 14:29:44.128117 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:44.128080 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e83f182f-4d9d2"] Oct 30 14:29:44.136516 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:44.136492 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:29:44.136516 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:44.136511 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:29:44.195408 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:44.195375 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4771a9e0-445a-40ae-8f79-2e6de53a3660-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e83f182f-4d9d2\" (UID: \"4771a9e0-445a-40ae-8f79-2e6de53a3660\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83f182f-4d9d2" Oct 30 14:29:44.195591 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:44.195479 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4771a9e0-445a-40ae-8f79-2e6de53a3660-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e83f182f-4d9d2\" (UID: \"4771a9e0-445a-40ae-8f79-2e6de53a3660\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83f182f-4d9d2" Oct 30 14:29:44.195591 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:44.195534 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4q9m\" (UniqueName: \"kubernetes.io/projected/4771a9e0-445a-40ae-8f79-2e6de53a3660-kube-api-access-m4q9m\") pod \"maas-api-revoke-sub-e83f182f-4d9d2\" (UID: \"4771a9e0-445a-40ae-8f79-2e6de53a3660\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83f182f-4d9d2" Oct 30 14:29:44.297552 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:44.297470 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4771a9e0-445a-40ae-8f79-2e6de53a3660-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e83f182f-4d9d2\" (UID: \"4771a9e0-445a-40ae-8f79-2e6de53a3660\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83f182f-4d9d2" Oct 30 14:29:44.297735 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:44.297576 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4771a9e0-445a-40ae-8f79-2e6de53a3660-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e83f182f-4d9d2\" (UID: \"4771a9e0-445a-40ae-8f79-2e6de53a3660\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83f182f-4d9d2" Oct 30 14:29:44.297735 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:44.297671 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4q9m\" (UniqueName: \"kubernetes.io/projected/4771a9e0-445a-40ae-8f79-2e6de53a3660-kube-api-access-m4q9m\") pod \"maas-api-revoke-sub-e83f182f-4d9d2\" (UID: \"4771a9e0-445a-40ae-8f79-2e6de53a3660\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83f182f-4d9d2" Oct 30 14:29:44.298246 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:44.298221 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4771a9e0-445a-40ae-8f79-2e6de53a3660-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e83f182f-4d9d2\" (UID: \"4771a9e0-445a-40ae-8f79-2e6de53a3660\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83f182f-4d9d2" Oct 30 14:29:44.299969 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:44.299947 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4771a9e0-445a-40ae-8f79-2e6de53a3660-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e83f182f-4d9d2\" (UID: \"4771a9e0-445a-40ae-8f79-2e6de53a3660\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83f182f-4d9d2" Oct 30 14:29:44.323939 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:44.323918 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4q9m\" (UniqueName: \"kubernetes.io/projected/4771a9e0-445a-40ae-8f79-2e6de53a3660-kube-api-access-m4q9m\") pod \"maas-api-revoke-sub-e83f182f-4d9d2\" (UID: \"4771a9e0-445a-40ae-8f79-2e6de53a3660\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83f182f-4d9d2" Oct 30 14:29:44.421169 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:44.421140 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83f182f-4d9d2" Oct 30 14:29:44.729279 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:29:44.729247 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4771a9e0_445a_40ae_8f79_2e6de53a3660.slice/crio-c41220e73007a4ff6f543d7a1df1ed39cc759167a5ddc433fb08edf5d7da76fb WatchSource:0}: Error finding container c41220e73007a4ff6f543d7a1df1ed39cc759167a5ddc433fb08edf5d7da76fb: Status 404 returned error can't find the container with id c41220e73007a4ff6f543d7a1df1ed39cc759167a5ddc433fb08edf5d7da76fb Oct 30 14:29:44.729988 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:44.729962 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e83f182f-4d9d2"] Oct 30 14:29:45.129740 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:45.129701 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83f182f-4d9d2" event={"ID":"4771a9e0-445a-40ae-8f79-2e6de53a3660","Type":"ContainerStarted","Data":"7d98b5551028f271e97b3fbd2dff2488030dc7d2279c54e836e48c8d84f6b5dd"} Oct 30 14:29:45.130222 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:45.129762 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83f182f-4d9d2" event={"ID":"4771a9e0-445a-40ae-8f79-2e6de53a3660","Type":"ContainerStarted","Data":"c41220e73007a4ff6f543d7a1df1ed39cc759167a5ddc433fb08edf5d7da76fb"} Oct 30 14:29:45.176451 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:45.176417 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:29:45.176765 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:45.176726 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83f182f-4d9d2" podStartSLOduration=1.176712399 podStartE2EDuration="1.176712399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:29:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:29:45.17283587 +0000 UTC m=+2454.916969912" watchObservedRunningTime="2026-10-30 14:29:45.176712399 +0000 UTC m=+2454.920846439" Oct 30 14:29:45.821681 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:45.821633 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:29:46.139871 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:46.139833 4982 generic.go:356] "Generic (PLEG): container finished" podID="4771a9e0-445a-40ae-8f79-2e6de53a3660" containerID="7d98b5551028f271e97b3fbd2dff2488030dc7d2279c54e836e48c8d84f6b5dd" exitCode=0 Oct 30 14:29:46.140391 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:46.139931 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83f182f-4d9d2" event={"ID":"4771a9e0-445a-40ae-8f79-2e6de53a3660","Type":"ContainerDied","Data":"7d98b5551028f271e97b3fbd2dff2488030dc7d2279c54e836e48c8d84f6b5dd"} Oct 30 14:29:47.306619 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:47.306587 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83f182f-4d9d2" Oct 30 14:29:47.342266 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:47.342233 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4771a9e0-445a-40ae-8f79-2e6de53a3660-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4771a9e0-445a-40ae-8f79-2e6de53a3660-maas-api-openshift-ca\") pod \"4771a9e0-445a-40ae-8f79-2e6de53a3660\" (UID: \"4771a9e0-445a-40ae-8f79-2e6de53a3660\") " Oct 30 14:29:47.342430 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:47.342375 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4771a9e0-445a-40ae-8f79-2e6de53a3660-kube-api-access-m4q9m\" (UniqueName: \"kubernetes.io/projected/4771a9e0-445a-40ae-8f79-2e6de53a3660-kube-api-access-m4q9m\") pod \"4771a9e0-445a-40ae-8f79-2e6de53a3660\" (UID: \"4771a9e0-445a-40ae-8f79-2e6de53a3660\") " Oct 30 14:29:47.342602 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:47.342587 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4771a9e0-445a-40ae-8f79-2e6de53a3660-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4771a9e0-445a-40ae-8f79-2e6de53a3660-maas-api-xks-ca\") pod \"4771a9e0-445a-40ae-8f79-2e6de53a3660\" (UID: \"4771a9e0-445a-40ae-8f79-2e6de53a3660\") " Oct 30 14:29:47.342692 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:47.342664 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4771a9e0-445a-40ae-8f79-2e6de53a3660-maas-api-openshift-ca" pod "4771a9e0-445a-40ae-8f79-2e6de53a3660" (UID: "4771a9e0-445a-40ae-8f79-2e6de53a3660"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:29:47.343209 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:47.343185 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4771a9e0-445a-40ae-8f79-2e6de53a3660-maas-api-openshift-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:29:47.345292 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:47.345266 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4771a9e0-445a-40ae-8f79-2e6de53a3660-kube-api-access-m4q9m" pod "4771a9e0-445a-40ae-8f79-2e6de53a3660" (UID: "4771a9e0-445a-40ae-8f79-2e6de53a3660"). InnerVolumeSpecName "kube-api-access-m4q9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:29:47.386044 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:47.383150 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4771a9e0-445a-40ae-8f79-2e6de53a3660-maas-api-xks-ca" pod "4771a9e0-445a-40ae-8f79-2e6de53a3660" (UID: "4771a9e0-445a-40ae-8f79-2e6de53a3660"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:29:47.420716 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:47.417811 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:29:47.445918 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:47.445886 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4771a9e0-445a-40ae-8f79-2e6de53a3660-maas-api-xks-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:29:47.446134 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:47.446116 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4q9m\" (UniqueName: \"kubernetes.io/projected/4771a9e0-445a-40ae-8f79-2e6de53a3660-kube-api-access-m4q9m\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:29:48.044576 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:48.044540 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:29:48.156503 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:48.156472 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83f182f-4d9d2" Oct 30 14:29:48.156503 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:48.156487 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83f182f-4d9d2" event={"ID":"4771a9e0-445a-40ae-8f79-2e6de53a3660","Type":"ContainerDied","Data":"c41220e73007a4ff6f543d7a1df1ed39cc759167a5ddc433fb08edf5d7da76fb"} Oct 30 14:29:48.156766 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:48.156534 4982 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c41220e73007a4ff6f543d7a1df1ed39cc759167a5ddc433fb08edf5d7da76fb" Oct 30 14:29:48.167830 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:48.167804 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-648d5f-lrpgp"] Oct 30 14:29:48.193435 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:48.193396 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:29:48.200034 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:48.199992 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-648d5f-lrpgp"] Oct 30 14:29:48.402285 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:48.402254 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0a42b17-8c26-4e05-af89-7c2aaede3d9e" path="/var/lib/kubelet/pods/a0a42b17-8c26-4e05-af89-7c2aaede3d9e/volumes" Oct 30 14:29:48.658019 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:48.657928 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:29:50.182105 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:50.182067 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:29:51.300293 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:51.300263 4982 scope.go:122] "RemoveContainer" containerID="fadcccdc9cfc6ef91e7e7ca4cfa445a651cb77718e29f3454a37414540e3de3c" Oct 30 14:29:52.108171 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:52.108139 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-1d6347-7778f9d655-545gn" Oct 30 14:29:52.136758 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:52.136722 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:29:53.970212 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:53.970174 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:29:55.119850 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:55.119811 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:29:55.833994 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:55.833952 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:29:56.362189 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:56.362155 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:29:56.508497 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:56.508459 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:29:56.698839 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:56.698756 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:29:57.880765 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:57.880729 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:29:58.013904 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:29:58.013867 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:29:59.666221 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:29:59.666185 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:30:00.161404 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.161367 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fde75d-29889510-mppvj"] Oct 30 14:30:00.162195 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.162168 4982 cpu_manager.go:414] "containerMap: removing container" podUID="4771a9e0-445a-40ae-8f79-2e6de53a3660" containerName="revoke-keys" Oct 30 14:30:00.162195 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.162193 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4771a9e0-445a-40ae-8f79-2e6de53a3660" containerName="revoke-keys" Oct 30 14:30:00.162351 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.162319 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="4771a9e0-445a-40ae-8f79-2e6de53a3660" containerName="revoke-keys" Oct 30 14:30:00.167037 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.167018 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-93549e-29889510-w9dvh"] Oct 30 14:30:00.167148 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.167130 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fde75d-29889510-mppvj" Oct 30 14:30:00.171316 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.171288 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-acefbb-29889510-7tmmg"] Oct 30 14:30:00.171442 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.171420 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-93549e-29889510-w9dvh" Oct 30 14:30:00.175591 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.175572 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-acefbb-29889510-7tmmg" Oct 30 14:30:00.196880 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.196850 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:30:00.204724 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.204696 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fde75d-29889510-mppvj"] Oct 30 14:30:00.222470 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.222386 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-acefbb-29889510-7tmmg"] Oct 30 14:30:00.229989 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.229960 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-93549e-29889510-w9dvh"] Oct 30 14:30:00.235302 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.235275 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d15602-29889510-5jdpd"] Oct 30 14:30:00.242319 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.242025 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b9c4b4-29889510-dhhvd"] Oct 30 14:30:00.242319 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.242168 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d15602-29889510-5jdpd" Oct 30 14:30:00.250905 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.250870 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b9c4b4-29889510-dhhvd" Oct 30 14:30:00.254500 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.254458 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b9c4b4-29889510-dhhvd"] Oct 30 14:30:00.265281 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.265202 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d15602-29889510-5jdpd"] Oct 30 14:30:00.271716 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.271692 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a9542c-29889510-s2w5r"] Oct 30 14:30:00.277946 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.277904 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-34bcc5-29889510-hkg5s"] Oct 30 14:30:00.288764 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.281549 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a9542c-29889510-s2w5r" Oct 30 14:30:00.288764 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.286166 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-34bcc5-29889510-hkg5s" Oct 30 14:30:00.308858 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.308827 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889510-cqz8m"] Oct 30 14:30:00.320605 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.320578 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889510-jfwrx"] Oct 30 14:30:00.322949 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.321758 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889510-cqz8m" Oct 30 14:30:00.336804 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.336106 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889510-jfwrx" Oct 30 14:30:00.340833 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.340777 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-34bcc5-29889510-hkg5s"] Oct 30 14:30:00.342203 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.342118 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a9542c-29889510-s2w5r"] Oct 30 14:30:00.344042 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.343287 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd736b21-6929-413d-8eb4-a9f891ffe64a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-d15602-29889510-5jdpd\" (UID: \"bd736b21-6929-413d-8eb4-a9f891ffe64a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d15602-29889510-5jdpd" Oct 30 14:30:00.344042 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.343385 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc3717af-2029-422a-9230-b41ebc4dedc5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-acefbb-29889510-7tmmg\" (UID: \"fc3717af-2029-422a-9230-b41ebc4dedc5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-acefbb-29889510-7tmmg" Oct 30 14:30:00.344042 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.343479 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d6da5d4-7fd8-4b0e-9a82-40c44fc77816-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-93549e-29889510-w9dvh\" (UID: \"7d6da5d4-7fd8-4b0e-9a82-40c44fc77816\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-93549e-29889510-w9dvh" Oct 30 14:30:00.344042 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.343567 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ecbf4bb-822a-4bd6-891c-c1eb6d12ecaa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-fde75d-29889510-mppvj\" (UID: \"6ecbf4bb-822a-4bd6-891c-c1eb6d12ecaa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fde75d-29889510-mppvj" Oct 30 14:30:00.344042 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.343606 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d998f\" (UniqueName: \"kubernetes.io/projected/fc3717af-2029-422a-9230-b41ebc4dedc5-kube-api-access-d998f\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-acefbb-29889510-7tmmg\" (UID: \"fc3717af-2029-422a-9230-b41ebc4dedc5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-acefbb-29889510-7tmmg" Oct 30 14:30:00.344042 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.343666 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jzp4\" (UniqueName: \"kubernetes.io/projected/6ecbf4bb-822a-4bd6-891c-c1eb6d12ecaa-kube-api-access-4jzp4\") pod \"maas-api-key-cleanup-e2e-inf-b-fde75d-29889510-mppvj\" (UID: \"6ecbf4bb-822a-4bd6-891c-c1eb6d12ecaa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fde75d-29889510-mppvj" Oct 30 14:30:00.344042 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.343709 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lslhp\" (UniqueName: \"kubernetes.io/projected/bd736b21-6929-413d-8eb4-a9f891ffe64a-kube-api-access-lslhp\") pod \"maas-api-key-cleanup-e2e-inf-a-d15602-29889510-5jdpd\" (UID: \"bd736b21-6929-413d-8eb4-a9f891ffe64a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d15602-29889510-5jdpd" Oct 30 14:30:00.344042 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.343743 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6nc2\" (UniqueName: \"kubernetes.io/projected/7d6da5d4-7fd8-4b0e-9a82-40c44fc77816-kube-api-access-w6nc2\") pod \"maas-api-key-cleanup-e2e-inf-b-93549e-29889510-w9dvh\" (UID: \"7d6da5d4-7fd8-4b0e-9a82-40c44fc77816\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-93549e-29889510-w9dvh" Oct 30 14:30:00.346650 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.346618 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889510-ksc4p"] Oct 30 14:30:00.353534 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.353513 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889510-ksc4p" Oct 30 14:30:00.356930 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.356908 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889510-cqz8m"] Oct 30 14:30:00.364751 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.364732 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889510-jfwrx"] Oct 30 14:30:00.365111 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.365095 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889510-v8wkp"] Oct 30 14:30:00.370722 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.370704 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889510-ksc4p"] Oct 30 14:30:00.370969 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.370900 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889510-v8wkp" Oct 30 14:30:00.427869 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.427806 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889510-v8wkp"] Oct 30 14:30:00.446477 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.446441 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1dda7b0-fb53-4fec-96ad-b8e04eb8b24b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-46b13b-29889510-cqz8m\" (UID: \"b1dda7b0-fb53-4fec-96ad-b8e04eb8b24b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889510-cqz8m" Oct 30 14:30:00.446631 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.446493 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jlvf\" (UniqueName: \"kubernetes.io/projected/b1dda7b0-fb53-4fec-96ad-b8e04eb8b24b-kube-api-access-8jlvf\") pod \"maas-api-key-cleanup-e2e-worker-w4-46b13b-29889510-cqz8m\" (UID: \"b1dda7b0-fb53-4fec-96ad-b8e04eb8b24b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889510-cqz8m" Oct 30 14:30:00.446631 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.446536 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljzdg\" (UniqueName: \"kubernetes.io/projected/b60b5431-525a-4363-a3fb-3352fe6521e9-kube-api-access-ljzdg\") pod \"maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889510-v8wkp\" (UID: \"b60b5431-525a-4363-a3fb-3352fe6521e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889510-v8wkp" Oct 30 14:30:00.446631 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.446572 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d6da5d4-7fd8-4b0e-9a82-40c44fc77816-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-93549e-29889510-w9dvh\" (UID: \"7d6da5d4-7fd8-4b0e-9a82-40c44fc77816\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-93549e-29889510-w9dvh" Oct 30 14:30:00.446769 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.446661 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c9a6cee-328e-461d-8e42-69c7bdf0afd6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-b9c4b4-29889510-dhhvd\" (UID: \"5c9a6cee-328e-461d-8e42-69c7bdf0afd6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b9c4b4-29889510-dhhvd" Oct 30 14:30:00.446769 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.446714 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ecbf4bb-822a-4bd6-891c-c1eb6d12ecaa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-fde75d-29889510-mppvj\" (UID: \"6ecbf4bb-822a-4bd6-891c-c1eb6d12ecaa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fde75d-29889510-mppvj" Oct 30 14:30:00.446769 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.446744 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6d5d\" (UniqueName: \"kubernetes.io/projected/f300b507-503e-47e1-a09b-d907b975c2c2-kube-api-access-r6d5d\") pod \"maas-api-key-cleanup-e2e-inf-a-34bcc5-29889510-hkg5s\" (UID: \"f300b507-503e-47e1-a09b-d907b975c2c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-34bcc5-29889510-hkg5s" Oct 30 14:30:00.446864 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.446788 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d998f\" (UniqueName: \"kubernetes.io/projected/fc3717af-2029-422a-9230-b41ebc4dedc5-kube-api-access-d998f\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-acefbb-29889510-7tmmg\" (UID: \"fc3717af-2029-422a-9230-b41ebc4dedc5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-acefbb-29889510-7tmmg" Oct 30 14:30:00.446864 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.446835 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jzp4\" (UniqueName: \"kubernetes.io/projected/6ecbf4bb-822a-4bd6-891c-c1eb6d12ecaa-kube-api-access-4jzp4\") pod \"maas-api-key-cleanup-e2e-inf-b-fde75d-29889510-mppvj\" (UID: \"6ecbf4bb-822a-4bd6-891c-c1eb6d12ecaa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fde75d-29889510-mppvj" Oct 30 14:30:00.446923 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.446862 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9246e72d-bf25-49e9-a530-78843f462b48-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889510-jfwrx\" (UID: \"9246e72d-bf25-49e9-a530-78843f462b48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889510-jfwrx" Oct 30 14:30:00.446923 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.446886 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f0959c9-4c3d-4394-baa1-6fd5a31fbe78-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-a9542c-29889510-s2w5r\" (UID: \"6f0959c9-4c3d-4394-baa1-6fd5a31fbe78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a9542c-29889510-s2w5r" Oct 30 14:30:00.446923 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.446908 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lslhp\" (UniqueName: \"kubernetes.io/projected/bd736b21-6929-413d-8eb4-a9f891ffe64a-kube-api-access-lslhp\") pod \"maas-api-key-cleanup-e2e-inf-a-d15602-29889510-5jdpd\" (UID: \"bd736b21-6929-413d-8eb4-a9f891ffe64a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d15602-29889510-5jdpd" Oct 30 14:30:00.447018 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.446932 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6nc2\" (UniqueName: \"kubernetes.io/projected/7d6da5d4-7fd8-4b0e-9a82-40c44fc77816-kube-api-access-w6nc2\") pod \"maas-api-key-cleanup-e2e-inf-b-93549e-29889510-w9dvh\" (UID: \"7d6da5d4-7fd8-4b0e-9a82-40c44fc77816\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-93549e-29889510-w9dvh" Oct 30 14:30:00.447018 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.446957 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvfm6\" (UniqueName: \"kubernetes.io/projected/6f0959c9-4c3d-4394-baa1-6fd5a31fbe78-kube-api-access-xvfm6\") pod \"maas-api-key-cleanup-e2e-inf-b-a9542c-29889510-s2w5r\" (UID: \"6f0959c9-4c3d-4394-baa1-6fd5a31fbe78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a9542c-29889510-s2w5r" Oct 30 14:30:00.447018 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.446984 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b60b5431-525a-4363-a3fb-3352fe6521e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889510-v8wkp\" (UID: \"b60b5431-525a-4363-a3fb-3352fe6521e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889510-v8wkp" Oct 30 14:30:00.447108 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.447021 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqf49\" (UniqueName: \"kubernetes.io/projected/9246e72d-bf25-49e9-a530-78843f462b48-kube-api-access-xqf49\") pod \"maas-api-key-cleanup-29889510-jfwrx\" (UID: \"9246e72d-bf25-49e9-a530-78843f462b48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889510-jfwrx" Oct 30 14:30:00.447108 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.447068 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbb2z\" (UniqueName: \"kubernetes.io/projected/5c9a6cee-328e-461d-8e42-69c7bdf0afd6-kube-api-access-fbb2z\") pod \"maas-api-key-cleanup-e2e-inf-b-b9c4b4-29889510-dhhvd\" (UID: \"5c9a6cee-328e-461d-8e42-69c7bdf0afd6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b9c4b4-29889510-dhhvd" Oct 30 14:30:00.447108 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.447082 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7260a80-a857-497e-bb12-28d8b7fa8bab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-39529e-29889510-ksc4p\" (UID: \"e7260a80-a857-497e-bb12-28d8b7fa8bab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889510-ksc4p" Oct 30 14:30:00.447199 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.447117 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd736b21-6929-413d-8eb4-a9f891ffe64a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-d15602-29889510-5jdpd\" (UID: \"bd736b21-6929-413d-8eb4-a9f891ffe64a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d15602-29889510-5jdpd" Oct 30 14:30:00.447199 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.447149 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhhg9\" (UniqueName: \"kubernetes.io/projected/e7260a80-a857-497e-bb12-28d8b7fa8bab-kube-api-access-jhhg9\") pod \"maas-api-key-cleanup-e2e-worker-w6-39529e-29889510-ksc4p\" (UID: \"e7260a80-a857-497e-bb12-28d8b7fa8bab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889510-ksc4p" Oct 30 14:30:00.447199 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.447191 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc3717af-2029-422a-9230-b41ebc4dedc5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-acefbb-29889510-7tmmg\" (UID: \"fc3717af-2029-422a-9230-b41ebc4dedc5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-acefbb-29889510-7tmmg" Oct 30 14:30:00.447290 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.447212 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f300b507-503e-47e1-a09b-d907b975c2c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-34bcc5-29889510-hkg5s\" (UID: \"f300b507-503e-47e1-a09b-d907b975c2c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-34bcc5-29889510-hkg5s" Oct 30 14:30:00.447902 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.447877 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d6da5d4-7fd8-4b0e-9a82-40c44fc77816-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-93549e-29889510-w9dvh\" (UID: \"7d6da5d4-7fd8-4b0e-9a82-40c44fc77816\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-93549e-29889510-w9dvh" Oct 30 14:30:00.448544 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.448521 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd736b21-6929-413d-8eb4-a9f891ffe64a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-d15602-29889510-5jdpd\" (UID: \"bd736b21-6929-413d-8eb4-a9f891ffe64a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d15602-29889510-5jdpd" Oct 30 14:30:00.449088 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.449053 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc3717af-2029-422a-9230-b41ebc4dedc5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-acefbb-29889510-7tmmg\" (UID: \"fc3717af-2029-422a-9230-b41ebc4dedc5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-acefbb-29889510-7tmmg" Oct 30 14:30:00.449441 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.449424 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ecbf4bb-822a-4bd6-891c-c1eb6d12ecaa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-fde75d-29889510-mppvj\" (UID: \"6ecbf4bb-822a-4bd6-891c-c1eb6d12ecaa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fde75d-29889510-mppvj" Oct 30 14:30:00.473597 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.473575 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lslhp\" (UniqueName: \"kubernetes.io/projected/bd736b21-6929-413d-8eb4-a9f891ffe64a-kube-api-access-lslhp\") pod \"maas-api-key-cleanup-e2e-inf-a-d15602-29889510-5jdpd\" (UID: \"bd736b21-6929-413d-8eb4-a9f891ffe64a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d15602-29889510-5jdpd" Oct 30 14:30:00.473720 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.473613 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6nc2\" (UniqueName: \"kubernetes.io/projected/7d6da5d4-7fd8-4b0e-9a82-40c44fc77816-kube-api-access-w6nc2\") pod \"maas-api-key-cleanup-e2e-inf-b-93549e-29889510-w9dvh\" (UID: \"7d6da5d4-7fd8-4b0e-9a82-40c44fc77816\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-93549e-29889510-w9dvh" Oct 30 14:30:00.473769 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.473735 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jzp4\" (UniqueName: \"kubernetes.io/projected/6ecbf4bb-822a-4bd6-891c-c1eb6d12ecaa-kube-api-access-4jzp4\") pod \"maas-api-key-cleanup-e2e-inf-b-fde75d-29889510-mppvj\" (UID: \"6ecbf4bb-822a-4bd6-891c-c1eb6d12ecaa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fde75d-29889510-mppvj" Oct 30 14:30:00.473953 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.473934 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d998f\" (UniqueName: \"kubernetes.io/projected/fc3717af-2029-422a-9230-b41ebc4dedc5-kube-api-access-d998f\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-acefbb-29889510-7tmmg\" (UID: \"fc3717af-2029-422a-9230-b41ebc4dedc5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-acefbb-29889510-7tmmg" Oct 30 14:30:00.485542 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.485354 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fde75d-29889510-mppvj" Oct 30 14:30:00.529921 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.529890 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-93549e-29889510-w9dvh" Oct 30 14:30:00.536654 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.536478 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-acefbb-29889510-7tmmg" Oct 30 14:30:00.548254 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.548229 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f0959c9-4c3d-4394-baa1-6fd5a31fbe78-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-a9542c-29889510-s2w5r\" (UID: \"6f0959c9-4c3d-4394-baa1-6fd5a31fbe78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a9542c-29889510-s2w5r" Oct 30 14:30:00.548352 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.548281 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvfm6\" (UniqueName: \"kubernetes.io/projected/6f0959c9-4c3d-4394-baa1-6fd5a31fbe78-kube-api-access-xvfm6\") pod \"maas-api-key-cleanup-e2e-inf-b-a9542c-29889510-s2w5r\" (UID: \"6f0959c9-4c3d-4394-baa1-6fd5a31fbe78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a9542c-29889510-s2w5r" Oct 30 14:30:00.548352 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.548308 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b60b5431-525a-4363-a3fb-3352fe6521e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889510-v8wkp\" (UID: \"b60b5431-525a-4363-a3fb-3352fe6521e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889510-v8wkp" Oct 30 14:30:00.548352 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.548348 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqf49\" (UniqueName: \"kubernetes.io/projected/9246e72d-bf25-49e9-a530-78843f462b48-kube-api-access-xqf49\") pod \"maas-api-key-cleanup-29889510-jfwrx\" (UID: \"9246e72d-bf25-49e9-a530-78843f462b48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889510-jfwrx" Oct 30 14:30:00.548472 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.548379 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbb2z\" (UniqueName: \"kubernetes.io/projected/5c9a6cee-328e-461d-8e42-69c7bdf0afd6-kube-api-access-fbb2z\") pod \"maas-api-key-cleanup-e2e-inf-b-b9c4b4-29889510-dhhvd\" (UID: \"5c9a6cee-328e-461d-8e42-69c7bdf0afd6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b9c4b4-29889510-dhhvd" Oct 30 14:30:00.548472 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.548397 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7260a80-a857-497e-bb12-28d8b7fa8bab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-39529e-29889510-ksc4p\" (UID: \"e7260a80-a857-497e-bb12-28d8b7fa8bab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889510-ksc4p" Oct 30 14:30:00.548472 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.548448 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhhg9\" (UniqueName: \"kubernetes.io/projected/e7260a80-a857-497e-bb12-28d8b7fa8bab-kube-api-access-jhhg9\") pod \"maas-api-key-cleanup-e2e-worker-w6-39529e-29889510-ksc4p\" (UID: \"e7260a80-a857-497e-bb12-28d8b7fa8bab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889510-ksc4p" Oct 30 14:30:00.548567 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.548490 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f300b507-503e-47e1-a09b-d907b975c2c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-34bcc5-29889510-hkg5s\" (UID: \"f300b507-503e-47e1-a09b-d907b975c2c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-34bcc5-29889510-hkg5s" Oct 30 14:30:00.548567 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.548531 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1dda7b0-fb53-4fec-96ad-b8e04eb8b24b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-46b13b-29889510-cqz8m\" (UID: \"b1dda7b0-fb53-4fec-96ad-b8e04eb8b24b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889510-cqz8m" Oct 30 14:30:00.548567 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.548548 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jlvf\" (UniqueName: \"kubernetes.io/projected/b1dda7b0-fb53-4fec-96ad-b8e04eb8b24b-kube-api-access-8jlvf\") pod \"maas-api-key-cleanup-e2e-worker-w4-46b13b-29889510-cqz8m\" (UID: \"b1dda7b0-fb53-4fec-96ad-b8e04eb8b24b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889510-cqz8m" Oct 30 14:30:00.548680 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.548571 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljzdg\" (UniqueName: \"kubernetes.io/projected/b60b5431-525a-4363-a3fb-3352fe6521e9-kube-api-access-ljzdg\") pod \"maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889510-v8wkp\" (UID: \"b60b5431-525a-4363-a3fb-3352fe6521e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889510-v8wkp" Oct 30 14:30:00.548680 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.548618 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c9a6cee-328e-461d-8e42-69c7bdf0afd6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-b9c4b4-29889510-dhhvd\" (UID: \"5c9a6cee-328e-461d-8e42-69c7bdf0afd6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b9c4b4-29889510-dhhvd" Oct 30 14:30:00.548748 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.548702 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6d5d\" (UniqueName: \"kubernetes.io/projected/f300b507-503e-47e1-a09b-d907b975c2c2-kube-api-access-r6d5d\") pod \"maas-api-key-cleanup-e2e-inf-a-34bcc5-29889510-hkg5s\" (UID: \"f300b507-503e-47e1-a09b-d907b975c2c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-34bcc5-29889510-hkg5s" Oct 30 14:30:00.548779 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.548744 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9246e72d-bf25-49e9-a530-78843f462b48-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889510-jfwrx\" (UID: \"9246e72d-bf25-49e9-a530-78843f462b48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889510-jfwrx" Oct 30 14:30:00.550464 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.549304 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9246e72d-bf25-49e9-a530-78843f462b48-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889510-jfwrx\" (UID: \"9246e72d-bf25-49e9-a530-78843f462b48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889510-jfwrx" Oct 30 14:30:00.550464 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.549523 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f0959c9-4c3d-4394-baa1-6fd5a31fbe78-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-a9542c-29889510-s2w5r\" (UID: \"6f0959c9-4c3d-4394-baa1-6fd5a31fbe78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a9542c-29889510-s2w5r" Oct 30 14:30:00.550464 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.549756 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f300b507-503e-47e1-a09b-d907b975c2c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-34bcc5-29889510-hkg5s\" (UID: \"f300b507-503e-47e1-a09b-d907b975c2c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-34bcc5-29889510-hkg5s" Oct 30 14:30:00.550464 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.550407 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1dda7b0-fb53-4fec-96ad-b8e04eb8b24b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-46b13b-29889510-cqz8m\" (UID: \"b1dda7b0-fb53-4fec-96ad-b8e04eb8b24b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889510-cqz8m" Oct 30 14:30:00.550464 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.550433 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c9a6cee-328e-461d-8e42-69c7bdf0afd6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-b9c4b4-29889510-dhhvd\" (UID: \"5c9a6cee-328e-461d-8e42-69c7bdf0afd6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b9c4b4-29889510-dhhvd" Oct 30 14:30:00.550464 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.550442 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7260a80-a857-497e-bb12-28d8b7fa8bab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-39529e-29889510-ksc4p\" (UID: \"e7260a80-a857-497e-bb12-28d8b7fa8bab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889510-ksc4p" Oct 30 14:30:00.550798 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.550517 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b60b5431-525a-4363-a3fb-3352fe6521e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889510-v8wkp\" (UID: \"b60b5431-525a-4363-a3fb-3352fe6521e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889510-v8wkp" Oct 30 14:30:00.571880 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.566186 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d15602-29889510-5jdpd" Oct 30 14:30:00.632193 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.631942 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljzdg\" (UniqueName: \"kubernetes.io/projected/b60b5431-525a-4363-a3fb-3352fe6521e9-kube-api-access-ljzdg\") pod \"maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889510-v8wkp\" (UID: \"b60b5431-525a-4363-a3fb-3352fe6521e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889510-v8wkp" Oct 30 14:30:00.636981 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.636900 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvfm6\" (UniqueName: \"kubernetes.io/projected/6f0959c9-4c3d-4394-baa1-6fd5a31fbe78-kube-api-access-xvfm6\") pod \"maas-api-key-cleanup-e2e-inf-b-a9542c-29889510-s2w5r\" (UID: \"6f0959c9-4c3d-4394-baa1-6fd5a31fbe78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a9542c-29889510-s2w5r" Oct 30 14:30:00.636981 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.636902 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbb2z\" (UniqueName: \"kubernetes.io/projected/5c9a6cee-328e-461d-8e42-69c7bdf0afd6-kube-api-access-fbb2z\") pod \"maas-api-key-cleanup-e2e-inf-b-b9c4b4-29889510-dhhvd\" (UID: \"5c9a6cee-328e-461d-8e42-69c7bdf0afd6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b9c4b4-29889510-dhhvd" Oct 30 14:30:00.636981 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.636958 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhhg9\" (UniqueName: \"kubernetes.io/projected/e7260a80-a857-497e-bb12-28d8b7fa8bab-kube-api-access-jhhg9\") pod \"maas-api-key-cleanup-e2e-worker-w6-39529e-29889510-ksc4p\" (UID: \"e7260a80-a857-497e-bb12-28d8b7fa8bab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889510-ksc4p" Oct 30 14:30:00.638367 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.638336 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6d5d\" (UniqueName: \"kubernetes.io/projected/f300b507-503e-47e1-a09b-d907b975c2c2-kube-api-access-r6d5d\") pod \"maas-api-key-cleanup-e2e-inf-a-34bcc5-29889510-hkg5s\" (UID: \"f300b507-503e-47e1-a09b-d907b975c2c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-34bcc5-29889510-hkg5s" Oct 30 14:30:00.639554 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.639511 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jlvf\" (UniqueName: \"kubernetes.io/projected/b1dda7b0-fb53-4fec-96ad-b8e04eb8b24b-kube-api-access-8jlvf\") pod \"maas-api-key-cleanup-e2e-worker-w4-46b13b-29889510-cqz8m\" (UID: \"b1dda7b0-fb53-4fec-96ad-b8e04eb8b24b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889510-cqz8m" Oct 30 14:30:00.639993 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.639973 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqf49\" (UniqueName: \"kubernetes.io/projected/9246e72d-bf25-49e9-a530-78843f462b48-kube-api-access-xqf49\") pod \"maas-api-key-cleanup-29889510-jfwrx\" (UID: \"9246e72d-bf25-49e9-a530-78843f462b48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889510-jfwrx" Oct 30 14:30:00.642677 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.642442 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-34bcc5-29889510-hkg5s" Oct 30 14:30:00.666107 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.658872 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889510-cqz8m" Oct 30 14:30:00.670692 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.667172 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889510-jfwrx" Oct 30 14:30:00.697852 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.696826 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889510-ksc4p" Oct 30 14:30:00.719403 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.719257 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889510-v8wkp" Oct 30 14:30:00.823791 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.822746 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fde75d-29889510-mppvj"] Oct 30 14:30:00.890556 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.890522 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b9c4b4-29889510-dhhvd" Oct 30 14:30:00.929973 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:00.929937 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a9542c-29889510-s2w5r" Oct 30 14:30:01.084079 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:01.084041 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-93549e-29889510-w9dvh"] Oct 30 14:30:01.137474 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:30:01.137434 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc3717af_2029_422a_9230_b41ebc4dedc5.slice/crio-6ad65578699a80bef71e218899ff92eed304bc19f9c696b288f15ee881b68e55 WatchSource:0}: Error finding container 6ad65578699a80bef71e218899ff92eed304bc19f9c696b288f15ee881b68e55: Status 404 returned error can't find the container with id 6ad65578699a80bef71e218899ff92eed304bc19f9c696b288f15ee881b68e55 Oct 30 14:30:01.138046 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:01.138019 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-acefbb-29889510-7tmmg"] Oct 30 14:30:01.235604 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:01.235579 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b9c4b4-29889510-dhhvd"] Oct 30 14:30:01.239073 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:01.239049 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-34bcc5-29889510-hkg5s"] Oct 30 14:30:01.263098 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:01.263075 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a9542c-29889510-s2w5r"] Oct 30 14:30:01.266539 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:01.266515 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889510-cqz8m"] Oct 30 14:30:01.289794 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:30:01.289762 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c9a6cee_328e_461d_8e42_69c7bdf0afd6.slice/crio-099af706c7f6150a380fb7f6ce02de66c036e836c467a25f995c8f38401334de WatchSource:0}: Error finding container 099af706c7f6150a380fb7f6ce02de66c036e836c467a25f995c8f38401334de: Status 404 returned error can't find the container with id 099af706c7f6150a380fb7f6ce02de66c036e836c467a25f995c8f38401334de Oct 30 14:30:01.292239 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:01.292210 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889510-jfwrx"] Oct 30 14:30:01.310996 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:30:01.310961 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f0959c9_4c3d_4394_baa1_6fd5a31fbe78.slice/crio-0339b2aac535adaf8ab4aa0de0b596c6dd1703c26bc057480b82b4f654625cad WatchSource:0}: Error finding container 0339b2aac535adaf8ab4aa0de0b596c6dd1703c26bc057480b82b4f654625cad: Status 404 returned error can't find the container with id 0339b2aac535adaf8ab4aa0de0b596c6dd1703c26bc057480b82b4f654625cad Oct 30 14:30:01.317008 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:01.314468 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889510-ksc4p"] Oct 30 14:30:01.317176 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:30:01.317016 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb60b5431_525a_4363_a3fb_3352fe6521e9.slice/crio-fc1d09f160f284f1b8b49cc4349427b8aeba553c85ac3c145b6ee0c093f4ea74 WatchSource:0}: Error finding container fc1d09f160f284f1b8b49cc4349427b8aeba553c85ac3c145b6ee0c093f4ea74: Status 404 returned error can't find the container with id fc1d09f160f284f1b8b49cc4349427b8aeba553c85ac3c145b6ee0c093f4ea74 Oct 30 14:30:01.318723 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:01.318679 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889510-v8wkp"] Oct 30 14:30:01.336306 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:01.336261 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fde75d-29889510-mppvj" event={"ID":"6ecbf4bb-822a-4bd6-891c-c1eb6d12ecaa","Type":"ContainerStarted","Data":"3679360e6bc218ff259e7776153a4738820362b3b3d0f30eac0c747ca3a6b9ce"} Oct 30 14:30:01.336416 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:01.336318 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fde75d-29889510-mppvj" event={"ID":"6ecbf4bb-822a-4bd6-891c-c1eb6d12ecaa","Type":"ContainerStarted","Data":"93a7dac0855cb4ba0f463860fd11cd7fe31257beb85e3af70efce74891313212"} Oct 30 14:30:01.337921 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:01.337899 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889510-ksc4p" event={"ID":"e7260a80-a857-497e-bb12-28d8b7fa8bab","Type":"ContainerStarted","Data":"ebc66f77cc3288fd4e324db6735526a3ef5257316009730f7a47f62e3f535fba"} Oct 30 14:30:01.339696 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:01.339674 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-34bcc5-29889510-hkg5s" event={"ID":"f300b507-503e-47e1-a09b-d907b975c2c2","Type":"ContainerStarted","Data":"dcbf54922feadf2342be99ee96951dd90b6dac54233eff75fb4fa9e38431563b"} Oct 30 14:30:01.349379 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:01.349345 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889510-jfwrx" event={"ID":"9246e72d-bf25-49e9-a530-78843f462b48","Type":"ContainerStarted","Data":"2592041f46d2bf30961c9ec7d1ab7ad706406b0e95216b200af6eabb9c97678d"} Oct 30 14:30:01.353902 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:01.353874 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-acefbb-29889510-7tmmg" event={"ID":"fc3717af-2029-422a-9230-b41ebc4dedc5","Type":"ContainerStarted","Data":"6ad65578699a80bef71e218899ff92eed304bc19f9c696b288f15ee881b68e55"} Oct 30 14:30:01.355502 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:01.355483 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b9c4b4-29889510-dhhvd" event={"ID":"5c9a6cee-328e-461d-8e42-69c7bdf0afd6","Type":"ContainerStarted","Data":"099af706c7f6150a380fb7f6ce02de66c036e836c467a25f995c8f38401334de"} Oct 30 14:30:01.356825 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:01.356797 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a9542c-29889510-s2w5r" event={"ID":"6f0959c9-4c3d-4394-baa1-6fd5a31fbe78","Type":"ContainerStarted","Data":"0339b2aac535adaf8ab4aa0de0b596c6dd1703c26bc057480b82b4f654625cad"} Oct 30 14:30:01.358245 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:01.358220 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889510-cqz8m" event={"ID":"b1dda7b0-fb53-4fec-96ad-b8e04eb8b24b","Type":"ContainerStarted","Data":"b082e3d5e798e49ffee84c42008ca277288f6a4c8291d41177d0ff9debcd7636"} Oct 30 14:30:01.360740 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:01.360714 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-93549e-29889510-w9dvh" event={"ID":"7d6da5d4-7fd8-4b0e-9a82-40c44fc77816","Type":"ContainerStarted","Data":"93fd554f97ec0ce03914ede4ec924a8c7defe651aa58d51bdf15097a8eb95793"} Oct 30 14:30:01.395896 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:01.383685 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d15602-29889510-5jdpd"] Oct 30 14:30:01.431363 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:01.431306 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fde75d-29889510-mppvj" podStartSLOduration=1.431285354 podStartE2EDuration="1.431285354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:30:01.400394323 +0000 UTC m=+2471.144528400" watchObservedRunningTime="2026-10-30 14:30:01.431285354 +0000 UTC m=+2471.175419398" Oct 30 14:30:01.813269 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:01.813176 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:30:02.372099 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:02.372064 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889510-v8wkp" event={"ID":"b60b5431-525a-4363-a3fb-3352fe6521e9","Type":"ContainerStarted","Data":"d1cddbb3997dc1125e4a80f38bca1f4c30e466e41689cffb008977e7e9f6c12f"} Oct 30 14:30:02.372231 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:02.372163 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889510-v8wkp" event={"ID":"b60b5431-525a-4363-a3fb-3352fe6521e9","Type":"ContainerStarted","Data":"fc1d09f160f284f1b8b49cc4349427b8aeba553c85ac3c145b6ee0c093f4ea74"} Oct 30 14:30:02.373938 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:02.373912 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889510-ksc4p" event={"ID":"e7260a80-a857-497e-bb12-28d8b7fa8bab","Type":"ContainerStarted","Data":"126281833cdcf14fa39235a92f808df3f0900eb0a38ef2b3e5854a2239926173"} Oct 30 14:30:02.375663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:02.375624 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-34bcc5-29889510-hkg5s" event={"ID":"f300b507-503e-47e1-a09b-d907b975c2c2","Type":"ContainerStarted","Data":"8f123a618f3a233e8984e76b96c95499e43086d3a4143e865ffe222cfd33211c"} Oct 30 14:30:02.377414 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:02.377390 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889510-jfwrx" event={"ID":"9246e72d-bf25-49e9-a530-78843f462b48","Type":"ContainerStarted","Data":"b95145216667042d559e20eb553c7ad77bb3f9d999a7b90f144b02ee00292ac0"} Oct 30 14:30:02.379455 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:02.379435 4982 generic.go:356] "Generic (PLEG): container finished" podID="fc3717af-2029-422a-9230-b41ebc4dedc5" containerID="05c3842bdcfd3446b39c6a5c75e26f01856bc930a49ebb850167e4a579425939" exitCode=60 Oct 30 14:30:02.379554 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:02.379510 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-acefbb-29889510-7tmmg" event={"ID":"fc3717af-2029-422a-9230-b41ebc4dedc5","Type":"ContainerDied","Data":"05c3842bdcfd3446b39c6a5c75e26f01856bc930a49ebb850167e4a579425939"} Oct 30 14:30:02.379736 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:02.379713 4982 scope.go:122] "RemoveContainer" containerID="05c3842bdcfd3446b39c6a5c75e26f01856bc930a49ebb850167e4a579425939" Oct 30 14:30:02.390863 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:02.390837 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b9c4b4-29889510-dhhvd" event={"ID":"5c9a6cee-328e-461d-8e42-69c7bdf0afd6","Type":"ContainerStarted","Data":"55ee5b6855557f69000b266d374d579f048aeee4534a1c01e5d9b273ef6a0052"} Oct 30 14:30:02.400069 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:02.400048 4982 generic.go:356] "Generic (PLEG): container finished" podID="7d6da5d4-7fd8-4b0e-9a82-40c44fc77816" containerID="c2830bd78e02a200f681067e58d080431968af6db27282bc6f5a0320b4f56a9f" exitCode=60 Oct 30 14:30:02.400358 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:02.400339 4982 scope.go:122] "RemoveContainer" containerID="c2830bd78e02a200f681067e58d080431968af6db27282bc6f5a0320b4f56a9f" Oct 30 14:30:02.407882 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:02.407385 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a9542c-29889510-s2w5r" event={"ID":"6f0959c9-4c3d-4394-baa1-6fd5a31fbe78","Type":"ContainerStarted","Data":"704f9ce9a2d54d132cbf260644ba373966e967c39b50a21ecb9f8be489901403"} Oct 30 14:30:02.407882 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:02.407430 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889510-cqz8m" event={"ID":"b1dda7b0-fb53-4fec-96ad-b8e04eb8b24b","Type":"ContainerStarted","Data":"26f5192455543a1f13af0223626023bb11db69252c996256c3282f4327b97a53"} Oct 30 14:30:02.407882 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:02.407456 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-93549e-29889510-w9dvh" event={"ID":"7d6da5d4-7fd8-4b0e-9a82-40c44fc77816","Type":"ContainerDied","Data":"c2830bd78e02a200f681067e58d080431968af6db27282bc6f5a0320b4f56a9f"} Oct 30 14:30:02.407882 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:02.407482 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d15602-29889510-5jdpd" event={"ID":"bd736b21-6929-413d-8eb4-a9f891ffe64a","Type":"ContainerStarted","Data":"62ec2dcdd1a4a542d3f1be19f49b5aa713e8774ff6f1a1a54b758f1587e213d2"} Oct 30 14:30:02.407882 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:02.407504 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d15602-29889510-5jdpd" event={"ID":"bd736b21-6929-413d-8eb4-a9f891ffe64a","Type":"ContainerStarted","Data":"88e6723e1bb5baf36c8f43a188c3985165381216ff1c460addb7420185d1a0e6"} Oct 30 14:30:02.409177 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:02.409150 4982 generic.go:356] "Generic (PLEG): container finished" podID="6ecbf4bb-822a-4bd6-891c-c1eb6d12ecaa" containerID="3679360e6bc218ff259e7776153a4738820362b3b3d0f30eac0c747ca3a6b9ce" exitCode=60 Oct 30 14:30:02.409265 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:02.409208 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fde75d-29889510-mppvj" event={"ID":"6ecbf4bb-822a-4bd6-891c-c1eb6d12ecaa","Type":"ContainerDied","Data":"3679360e6bc218ff259e7776153a4738820362b3b3d0f30eac0c747ca3a6b9ce"} Oct 30 14:30:02.409744 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:02.409723 4982 scope.go:122] "RemoveContainer" containerID="3679360e6bc218ff259e7776153a4738820362b3b3d0f30eac0c747ca3a6b9ce" Oct 30 14:30:02.421737 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:02.420684 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889510-v8wkp" podStartSLOduration=2.420667601 podStartE2EDuration="2.420667601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:30:02.416274535 +0000 UTC m=+2472.160408573" watchObservedRunningTime="2026-10-30 14:30:02.420667601 +0000 UTC m=+2472.164801644" Oct 30 14:30:02.461170 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:02.461127 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b9c4b4-29889510-dhhvd" podStartSLOduration=2.461114701 podStartE2EDuration="2.461114701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:30:02.460837824 +0000 UTC m=+2472.204971867" watchObservedRunningTime="2026-10-30 14:30:02.461114701 +0000 UTC m=+2472.205248741" Oct 30 14:30:02.551944 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:02.551892 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a9542c-29889510-s2w5r" podStartSLOduration=2.5518753849999998 podStartE2EDuration="2.551875385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:30:02.547928158 +0000 UTC m=+2472.292062200" watchObservedRunningTime="2026-10-30 14:30:02.551875385 +0000 UTC m=+2472.296009426" Oct 30 14:30:02.637062 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:02.637011 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-34bcc5-29889510-hkg5s" podStartSLOduration=2.63699748 podStartE2EDuration="2.63699748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:30:02.635568269 +0000 UTC m=+2472.379702301" watchObservedRunningTime="2026-10-30 14:30:02.63699748 +0000 UTC m=+2472.381131519" Oct 30 14:30:02.708923 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:02.707835 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889510-jfwrx" podStartSLOduration=2.7078147980000002 podStartE2EDuration="2.707814798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:30:02.705728298 +0000 UTC m=+2472.449862343" watchObservedRunningTime="2026-10-30 14:30:02.707814798 +0000 UTC m=+2472.451948839" Oct 30 14:30:02.750563 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:02.750510 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889510-ksc4p" podStartSLOduration=2.750495738 podStartE2EDuration="2.750495738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:30:02.748361272 +0000 UTC m=+2472.492495312" watchObservedRunningTime="2026-10-30 14:30:02.750495738 +0000 UTC m=+2472.494629778" Oct 30 14:30:02.794935 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:02.794879 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889510-cqz8m" podStartSLOduration=2.794858803 podStartE2EDuration="2.794858803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:30:02.791660648 +0000 UTC m=+2472.535794691" watchObservedRunningTime="2026-10-30 14:30:02.794858803 +0000 UTC m=+2472.538992844" Oct 30 14:30:02.887097 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:02.884890 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d15602-29889510-5jdpd" podStartSLOduration=2.884855024 podStartE2EDuration="2.884855024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:30:02.877969814 +0000 UTC m=+2472.622103855" watchObservedRunningTime="2026-10-30 14:30:02.884855024 +0000 UTC m=+2472.628989066" Oct 30 14:30:03.420734 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:03.420691 4982 generic.go:356] "Generic (PLEG): container finished" podID="e7260a80-a857-497e-bb12-28d8b7fa8bab" containerID="126281833cdcf14fa39235a92f808df3f0900eb0a38ef2b3e5854a2239926173" exitCode=60 Oct 30 14:30:03.420918 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:03.420808 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889510-ksc4p" event={"ID":"e7260a80-a857-497e-bb12-28d8b7fa8bab","Type":"ContainerDied","Data":"126281833cdcf14fa39235a92f808df3f0900eb0a38ef2b3e5854a2239926173"} Oct 30 14:30:03.421118 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:03.421092 4982 scope.go:122] "RemoveContainer" containerID="126281833cdcf14fa39235a92f808df3f0900eb0a38ef2b3e5854a2239926173" Oct 30 14:30:03.422977 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:03.422952 4982 generic.go:356] "Generic (PLEG): container finished" podID="f300b507-503e-47e1-a09b-d907b975c2c2" containerID="8f123a618f3a233e8984e76b96c95499e43086d3a4143e865ffe222cfd33211c" exitCode=60 Oct 30 14:30:03.423618 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:03.423076 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-34bcc5-29889510-hkg5s" event={"ID":"f300b507-503e-47e1-a09b-d907b975c2c2","Type":"ContainerDied","Data":"8f123a618f3a233e8984e76b96c95499e43086d3a4143e865ffe222cfd33211c"} Oct 30 14:30:03.423618 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:03.423269 4982 scope.go:122] "RemoveContainer" containerID="8f123a618f3a233e8984e76b96c95499e43086d3a4143e865ffe222cfd33211c" Oct 30 14:30:03.425045 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:03.424882 4982 generic.go:356] "Generic (PLEG): container finished" podID="9246e72d-bf25-49e9-a530-78843f462b48" containerID="b95145216667042d559e20eb553c7ad77bb3f9d999a7b90f144b02ee00292ac0" exitCode=60 Oct 30 14:30:03.425045 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:03.424998 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889510-jfwrx" event={"ID":"9246e72d-bf25-49e9-a530-78843f462b48","Type":"ContainerDied","Data":"b95145216667042d559e20eb553c7ad77bb3f9d999a7b90f144b02ee00292ac0"} Oct 30 14:30:03.425415 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:03.425196 4982 scope.go:122] "RemoveContainer" containerID="b95145216667042d559e20eb553c7ad77bb3f9d999a7b90f144b02ee00292ac0" Oct 30 14:30:03.428186 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:03.428105 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-acefbb-29889510-7tmmg" event={"ID":"fc3717af-2029-422a-9230-b41ebc4dedc5","Type":"ContainerStarted","Data":"183898b76a0885019156b162baf93cfc02dc321e5487e4932d39ae4905e73b3a"} Oct 30 14:30:03.430263 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:03.430238 4982 generic.go:356] "Generic (PLEG): container finished" podID="5c9a6cee-328e-461d-8e42-69c7bdf0afd6" containerID="55ee5b6855557f69000b266d374d579f048aeee4534a1c01e5d9b273ef6a0052" exitCode=60 Oct 30 14:30:03.430367 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:03.430315 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b9c4b4-29889510-dhhvd" event={"ID":"5c9a6cee-328e-461d-8e42-69c7bdf0afd6","Type":"ContainerDied","Data":"55ee5b6855557f69000b266d374d579f048aeee4534a1c01e5d9b273ef6a0052"} Oct 30 14:30:03.430669 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:03.430622 4982 scope.go:122] "RemoveContainer" containerID="55ee5b6855557f69000b266d374d579f048aeee4534a1c01e5d9b273ef6a0052" Oct 30 14:30:03.431956 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:03.431933 4982 generic.go:356] "Generic (PLEG): container finished" podID="6f0959c9-4c3d-4394-baa1-6fd5a31fbe78" containerID="704f9ce9a2d54d132cbf260644ba373966e967c39b50a21ecb9f8be489901403" exitCode=60 Oct 30 14:30:03.432040 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:03.432002 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a9542c-29889510-s2w5r" event={"ID":"6f0959c9-4c3d-4394-baa1-6fd5a31fbe78","Type":"ContainerDied","Data":"704f9ce9a2d54d132cbf260644ba373966e967c39b50a21ecb9f8be489901403"} Oct 30 14:30:03.432212 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:03.432191 4982 scope.go:122] "RemoveContainer" containerID="704f9ce9a2d54d132cbf260644ba373966e967c39b50a21ecb9f8be489901403" Oct 30 14:30:03.434911 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:03.434402 4982 generic.go:356] "Generic (PLEG): container finished" podID="b1dda7b0-fb53-4fec-96ad-b8e04eb8b24b" containerID="26f5192455543a1f13af0223626023bb11db69252c996256c3282f4327b97a53" exitCode=60 Oct 30 14:30:03.434911 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:03.434431 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889510-cqz8m" event={"ID":"b1dda7b0-fb53-4fec-96ad-b8e04eb8b24b","Type":"ContainerDied","Data":"26f5192455543a1f13af0223626023bb11db69252c996256c3282f4327b97a53"} Oct 30 14:30:03.434911 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:03.434693 4982 scope.go:122] "RemoveContainer" containerID="26f5192455543a1f13af0223626023bb11db69252c996256c3282f4327b97a53" Oct 30 14:30:03.436494 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:03.436470 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-93549e-29889510-w9dvh" event={"ID":"7d6da5d4-7fd8-4b0e-9a82-40c44fc77816","Type":"ContainerStarted","Data":"00b4fdca412fd23c64a08340d45d2d2607984121653f8df33b81d092e2e4d6ad"} Oct 30 14:30:03.438112 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:03.438092 4982 generic.go:356] "Generic (PLEG): container finished" podID="bd736b21-6929-413d-8eb4-a9f891ffe64a" containerID="62ec2dcdd1a4a542d3f1be19f49b5aa713e8774ff6f1a1a54b758f1587e213d2" exitCode=60 Oct 30 14:30:03.438208 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:03.438175 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d15602-29889510-5jdpd" event={"ID":"bd736b21-6929-413d-8eb4-a9f891ffe64a","Type":"ContainerDied","Data":"62ec2dcdd1a4a542d3f1be19f49b5aa713e8774ff6f1a1a54b758f1587e213d2"} Oct 30 14:30:03.438362 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:03.438347 4982 scope.go:122] "RemoveContainer" containerID="62ec2dcdd1a4a542d3f1be19f49b5aa713e8774ff6f1a1a54b758f1587e213d2" Oct 30 14:30:03.440313 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:03.440290 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fde75d-29889510-mppvj" event={"ID":"6ecbf4bb-822a-4bd6-891c-c1eb6d12ecaa","Type":"ContainerStarted","Data":"f77f39483292f899ad6b637701e664c19753528fe4ba1155c49145524eb74fc7"} Oct 30 14:30:03.446711 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:03.446686 4982 generic.go:356] "Generic (PLEG): container finished" podID="b60b5431-525a-4363-a3fb-3352fe6521e9" containerID="d1cddbb3997dc1125e4a80f38bca1f4c30e466e41689cffb008977e7e9f6c12f" exitCode=60 Oct 30 14:30:03.446830 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:03.446726 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889510-v8wkp" event={"ID":"b60b5431-525a-4363-a3fb-3352fe6521e9","Type":"ContainerDied","Data":"d1cddbb3997dc1125e4a80f38bca1f4c30e466e41689cffb008977e7e9f6c12f"} Oct 30 14:30:03.447059 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:03.447002 4982 scope.go:122] "RemoveContainer" containerID="d1cddbb3997dc1125e4a80f38bca1f4c30e466e41689cffb008977e7e9f6c12f" Oct 30 14:30:03.600594 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:03.600543 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-acefbb-29889510-7tmmg" podStartSLOduration=3.600527797 podStartE2EDuration="3.600527797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:30:03.59918804 +0000 UTC m=+2473.343322080" watchObservedRunningTime="2026-10-30 14:30:03.600527797 +0000 UTC m=+2473.344661834" Oct 30 14:30:03.738770 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:03.738725 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-93549e-29889510-w9dvh" podStartSLOduration=3.738711155 podStartE2EDuration="3.738711155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:30:03.737247398 +0000 UTC m=+2473.481381438" watchObservedRunningTime="2026-10-30 14:30:03.738711155 +0000 UTC m=+2473.482845198" Oct 30 14:30:03.841837 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:03.841802 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:30:04.459349 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:04.459305 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889510-v8wkp" event={"ID":"b60b5431-525a-4363-a3fb-3352fe6521e9","Type":"ContainerStarted","Data":"c31661dd58496bf307968b1a8b0161906aad890506a1c07756f230aef735950c"} Oct 30 14:30:04.461325 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:04.461297 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889510-ksc4p" event={"ID":"e7260a80-a857-497e-bb12-28d8b7fa8bab","Type":"ContainerStarted","Data":"ed30e26901f71cfc6efe4db8c1fbc34b52d90ca7c3b8af0dcfb7b64f773bbe9f"} Oct 30 14:30:04.463117 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:04.463088 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-34bcc5-29889510-hkg5s" event={"ID":"f300b507-503e-47e1-a09b-d907b975c2c2","Type":"ContainerStarted","Data":"e2c77a41bf317b72f8fc6617c07497c4bee7b44ac464e2c0013bb89537db490a"} Oct 30 14:30:04.465013 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:04.464990 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889510-jfwrx" event={"ID":"9246e72d-bf25-49e9-a530-78843f462b48","Type":"ContainerStarted","Data":"ec103dd100f32b7f8c161fcf67c0283933e93455dec037f1da999db33cea3b94"} Oct 30 14:30:04.466942 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:04.466923 4982 generic.go:356] "Generic (PLEG): container finished" podID="fc3717af-2029-422a-9230-b41ebc4dedc5" containerID="183898b76a0885019156b162baf93cfc02dc321e5487e4932d39ae4905e73b3a" exitCode=60 Oct 30 14:30:04.467034 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:04.467002 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-acefbb-29889510-7tmmg" event={"ID":"fc3717af-2029-422a-9230-b41ebc4dedc5","Type":"ContainerDied","Data":"183898b76a0885019156b162baf93cfc02dc321e5487e4932d39ae4905e73b3a"} Oct 30 14:30:04.467093 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:04.467065 4982 scope.go:122] "RemoveContainer" containerID="05c3842bdcfd3446b39c6a5c75e26f01856bc930a49ebb850167e4a579425939" Oct 30 14:30:04.467193 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:04.467177 4982 scope.go:122] "RemoveContainer" containerID="183898b76a0885019156b162baf93cfc02dc321e5487e4932d39ae4905e73b3a" Oct 30 14:30:04.467456 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:30:04.467434 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-acefbb-29889510-7tmmg_odh-ai-gateway-infra(fc3717af-2029-422a-9230-b41ebc4dedc5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-acefbb-29889510-7tmmg" podUID="fc3717af-2029-422a-9230-b41ebc4dedc5" Oct 30 14:30:04.468914 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:04.468893 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b9c4b4-29889510-dhhvd" event={"ID":"5c9a6cee-328e-461d-8e42-69c7bdf0afd6","Type":"ContainerStarted","Data":"9e7ade9b479ea9318f1aca444dcfec68d74c43b333fc0f4d177c962b1e2339e7"} Oct 30 14:30:04.470741 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:04.470718 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a9542c-29889510-s2w5r" event={"ID":"6f0959c9-4c3d-4394-baa1-6fd5a31fbe78","Type":"ContainerStarted","Data":"5913e1fb85dc10e3b3326d0092d8ea31064e56a8f653066429c7b81a64c98ac5"} Oct 30 14:30:04.472531 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:04.472513 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889510-cqz8m" event={"ID":"b1dda7b0-fb53-4fec-96ad-b8e04eb8b24b","Type":"ContainerStarted","Data":"0549ec03a578e279de1b6bfbdfa25107d5a1975c5bfa955e2bb0fe36453eca20"} Oct 30 14:30:04.474401 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:04.474381 4982 generic.go:356] "Generic (PLEG): container finished" podID="7d6da5d4-7fd8-4b0e-9a82-40c44fc77816" containerID="00b4fdca412fd23c64a08340d45d2d2607984121653f8df33b81d092e2e4d6ad" exitCode=60 Oct 30 14:30:04.474508 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:04.474463 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-93549e-29889510-w9dvh" event={"ID":"7d6da5d4-7fd8-4b0e-9a82-40c44fc77816","Type":"ContainerDied","Data":"00b4fdca412fd23c64a08340d45d2d2607984121653f8df33b81d092e2e4d6ad"} Oct 30 14:30:04.474672 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:04.474618 4982 scope.go:122] "RemoveContainer" containerID="00b4fdca412fd23c64a08340d45d2d2607984121653f8df33b81d092e2e4d6ad" Oct 30 14:30:04.474865 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:30:04.474841 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-93549e-29889510-w9dvh_odh-ai-gateway-infra(7d6da5d4-7fd8-4b0e-9a82-40c44fc77816)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-93549e-29889510-w9dvh" podUID="7d6da5d4-7fd8-4b0e-9a82-40c44fc77816" Oct 30 14:30:04.476519 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:04.476496 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d15602-29889510-5jdpd" event={"ID":"bd736b21-6929-413d-8eb4-a9f891ffe64a","Type":"ContainerStarted","Data":"98efdc5b70442aabcbd62532d663463df50bca2aae6855fe5fa8bdeeb4f719a0"} Oct 30 14:30:04.478385 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:04.478362 4982 generic.go:356] "Generic (PLEG): container finished" podID="6ecbf4bb-822a-4bd6-891c-c1eb6d12ecaa" containerID="f77f39483292f899ad6b637701e664c19753528fe4ba1155c49145524eb74fc7" exitCode=60 Oct 30 14:30:04.478487 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:04.478405 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fde75d-29889510-mppvj" event={"ID":"6ecbf4bb-822a-4bd6-891c-c1eb6d12ecaa","Type":"ContainerDied","Data":"f77f39483292f899ad6b637701e664c19753528fe4ba1155c49145524eb74fc7"} Oct 30 14:30:04.478614 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:04.478598 4982 scope.go:122] "RemoveContainer" containerID="f77f39483292f899ad6b637701e664c19753528fe4ba1155c49145524eb74fc7" Oct 30 14:30:04.478856 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:30:04.478837 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-fde75d-29889510-mppvj_odh-ai-gateway-infra(6ecbf4bb-822a-4bd6-891c-c1eb6d12ecaa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fde75d-29889510-mppvj" podUID="6ecbf4bb-822a-4bd6-891c-c1eb6d12ecaa" Oct 30 14:30:04.491258 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:04.491237 4982 scope.go:122] "RemoveContainer" containerID="c2830bd78e02a200f681067e58d080431968af6db27282bc6f5a0320b4f56a9f" Oct 30 14:30:04.507825 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:04.507803 4982 scope.go:122] "RemoveContainer" containerID="3679360e6bc218ff259e7776153a4738820362b3b3d0f30eac0c747ca3a6b9ce" Oct 30 14:30:05.489795 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:05.489758 4982 generic.go:356] "Generic (PLEG): container finished" podID="e7260a80-a857-497e-bb12-28d8b7fa8bab" containerID="ed30e26901f71cfc6efe4db8c1fbc34b52d90ca7c3b8af0dcfb7b64f773bbe9f" exitCode=60 Oct 30 14:30:05.490249 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:05.489845 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889510-ksc4p" event={"ID":"e7260a80-a857-497e-bb12-28d8b7fa8bab","Type":"ContainerDied","Data":"ed30e26901f71cfc6efe4db8c1fbc34b52d90ca7c3b8af0dcfb7b64f773bbe9f"} Oct 30 14:30:05.490249 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:05.489914 4982 scope.go:122] "RemoveContainer" containerID="126281833cdcf14fa39235a92f808df3f0900eb0a38ef2b3e5854a2239926173" Oct 30 14:30:05.490249 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:05.490152 4982 scope.go:122] "RemoveContainer" containerID="ed30e26901f71cfc6efe4db8c1fbc34b52d90ca7c3b8af0dcfb7b64f773bbe9f" Oct 30 14:30:05.490572 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:30:05.490542 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-39529e-29889510-ksc4p_odh-ai-gateway-infra(e7260a80-a857-497e-bb12-28d8b7fa8bab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889510-ksc4p" podUID="e7260a80-a857-497e-bb12-28d8b7fa8bab" Oct 30 14:30:05.492570 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:05.492118 4982 generic.go:356] "Generic (PLEG): container finished" podID="f300b507-503e-47e1-a09b-d907b975c2c2" containerID="e2c77a41bf317b72f8fc6617c07497c4bee7b44ac464e2c0013bb89537db490a" exitCode=60 Oct 30 14:30:05.492570 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:05.492195 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-34bcc5-29889510-hkg5s" event={"ID":"f300b507-503e-47e1-a09b-d907b975c2c2","Type":"ContainerDied","Data":"e2c77a41bf317b72f8fc6617c07497c4bee7b44ac464e2c0013bb89537db490a"} Oct 30 14:30:05.492570 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:05.492464 4982 scope.go:122] "RemoveContainer" containerID="e2c77a41bf317b72f8fc6617c07497c4bee7b44ac464e2c0013bb89537db490a" Oct 30 14:30:05.492838 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:30:05.492814 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-34bcc5-29889510-hkg5s_odh-ai-gateway-infra(f300b507-503e-47e1-a09b-d907b975c2c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-34bcc5-29889510-hkg5s" podUID="f300b507-503e-47e1-a09b-d907b975c2c2" Oct 30 14:30:05.494282 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:05.494260 4982 generic.go:356] "Generic (PLEG): container finished" podID="9246e72d-bf25-49e9-a530-78843f462b48" containerID="ec103dd100f32b7f8c161fcf67c0283933e93455dec037f1da999db33cea3b94" exitCode=60 Oct 30 14:30:05.494408 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:05.494346 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889510-jfwrx" event={"ID":"9246e72d-bf25-49e9-a530-78843f462b48","Type":"ContainerDied","Data":"ec103dd100f32b7f8c161fcf67c0283933e93455dec037f1da999db33cea3b94"} Oct 30 14:30:05.494615 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:05.494595 4982 scope.go:122] "RemoveContainer" containerID="ec103dd100f32b7f8c161fcf67c0283933e93455dec037f1da999db33cea3b94" Oct 30 14:30:05.494952 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:30:05.494927 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889510-jfwrx_odh-ai-gateway-infra(9246e72d-bf25-49e9-a530-78843f462b48)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889510-jfwrx" podUID="9246e72d-bf25-49e9-a530-78843f462b48" Oct 30 14:30:05.496602 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:05.496583 4982 scope.go:122] "RemoveContainer" containerID="183898b76a0885019156b162baf93cfc02dc321e5487e4932d39ae4905e73b3a" Oct 30 14:30:05.496945 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:30:05.496920 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-acefbb-29889510-7tmmg_odh-ai-gateway-infra(fc3717af-2029-422a-9230-b41ebc4dedc5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-acefbb-29889510-7tmmg" podUID="fc3717af-2029-422a-9230-b41ebc4dedc5" Oct 30 14:30:05.498713 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:05.498689 4982 generic.go:356] "Generic (PLEG): container finished" podID="5c9a6cee-328e-461d-8e42-69c7bdf0afd6" containerID="9e7ade9b479ea9318f1aca444dcfec68d74c43b333fc0f4d177c962b1e2339e7" exitCode=60 Oct 30 14:30:05.498828 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:05.498773 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b9c4b4-29889510-dhhvd" event={"ID":"5c9a6cee-328e-461d-8e42-69c7bdf0afd6","Type":"ContainerDied","Data":"9e7ade9b479ea9318f1aca444dcfec68d74c43b333fc0f4d177c962b1e2339e7"} Oct 30 14:30:05.499011 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:05.498994 4982 scope.go:122] "RemoveContainer" containerID="9e7ade9b479ea9318f1aca444dcfec68d74c43b333fc0f4d177c962b1e2339e7" Oct 30 14:30:05.499346 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:30:05.499291 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-b9c4b4-29889510-dhhvd_odh-ai-gateway-infra(5c9a6cee-328e-461d-8e42-69c7bdf0afd6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b9c4b4-29889510-dhhvd" podUID="5c9a6cee-328e-461d-8e42-69c7bdf0afd6" Oct 30 14:30:05.500985 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:05.500966 4982 generic.go:356] "Generic (PLEG): container finished" podID="6f0959c9-4c3d-4394-baa1-6fd5a31fbe78" containerID="5913e1fb85dc10e3b3326d0092d8ea31064e56a8f653066429c7b81a64c98ac5" exitCode=60 Oct 30 14:30:05.501091 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:05.501049 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a9542c-29889510-s2w5r" event={"ID":"6f0959c9-4c3d-4394-baa1-6fd5a31fbe78","Type":"ContainerDied","Data":"5913e1fb85dc10e3b3326d0092d8ea31064e56a8f653066429c7b81a64c98ac5"} Oct 30 14:30:05.501303 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:05.501284 4982 scope.go:122] "RemoveContainer" containerID="5913e1fb85dc10e3b3326d0092d8ea31064e56a8f653066429c7b81a64c98ac5" Oct 30 14:30:05.501597 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:30:05.501571 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-a9542c-29889510-s2w5r_odh-ai-gateway-infra(6f0959c9-4c3d-4394-baa1-6fd5a31fbe78)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a9542c-29889510-s2w5r" podUID="6f0959c9-4c3d-4394-baa1-6fd5a31fbe78" Oct 30 14:30:05.503451 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:05.503431 4982 generic.go:356] "Generic (PLEG): container finished" podID="b1dda7b0-fb53-4fec-96ad-b8e04eb8b24b" containerID="0549ec03a578e279de1b6bfbdfa25107d5a1975c5bfa955e2bb0fe36453eca20" exitCode=60 Oct 30 14:30:05.503613 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:05.503590 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889510-cqz8m" event={"ID":"b1dda7b0-fb53-4fec-96ad-b8e04eb8b24b","Type":"ContainerDied","Data":"0549ec03a578e279de1b6bfbdfa25107d5a1975c5bfa955e2bb0fe36453eca20"} Oct 30 14:30:05.503767 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:05.503749 4982 scope.go:122] "RemoveContainer" containerID="0549ec03a578e279de1b6bfbdfa25107d5a1975c5bfa955e2bb0fe36453eca20" Oct 30 14:30:05.504052 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:30:05.504023 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-46b13b-29889510-cqz8m_odh-ai-gateway-infra(b1dda7b0-fb53-4fec-96ad-b8e04eb8b24b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889510-cqz8m" podUID="b1dda7b0-fb53-4fec-96ad-b8e04eb8b24b" Oct 30 14:30:05.505812 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:05.505791 4982 scope.go:122] "RemoveContainer" containerID="00b4fdca412fd23c64a08340d45d2d2607984121653f8df33b81d092e2e4d6ad" Oct 30 14:30:05.506076 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:30:05.506056 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-93549e-29889510-w9dvh_odh-ai-gateway-infra(7d6da5d4-7fd8-4b0e-9a82-40c44fc77816)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-93549e-29889510-w9dvh" podUID="7d6da5d4-7fd8-4b0e-9a82-40c44fc77816" Oct 30 14:30:05.507391 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:05.507369 4982 generic.go:356] "Generic (PLEG): container finished" podID="bd736b21-6929-413d-8eb4-a9f891ffe64a" containerID="98efdc5b70442aabcbd62532d663463df50bca2aae6855fe5fa8bdeeb4f719a0" exitCode=60 Oct 30 14:30:05.507504 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:05.507463 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d15602-29889510-5jdpd" event={"ID":"bd736b21-6929-413d-8eb4-a9f891ffe64a","Type":"ContainerDied","Data":"98efdc5b70442aabcbd62532d663463df50bca2aae6855fe5fa8bdeeb4f719a0"} Oct 30 14:30:05.507681 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:05.507633 4982 scope.go:122] "RemoveContainer" containerID="98efdc5b70442aabcbd62532d663463df50bca2aae6855fe5fa8bdeeb4f719a0" Oct 30 14:30:05.507893 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:30:05.507873 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-d15602-29889510-5jdpd_odh-ai-gateway-infra(bd736b21-6929-413d-8eb4-a9f891ffe64a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d15602-29889510-5jdpd" podUID="bd736b21-6929-413d-8eb4-a9f891ffe64a" Oct 30 14:30:05.507962 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:05.507904 4982 scope.go:122] "RemoveContainer" containerID="8f123a618f3a233e8984e76b96c95499e43086d3a4143e865ffe222cfd33211c" Oct 30 14:30:05.509445 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:05.509427 4982 scope.go:122] "RemoveContainer" containerID="f77f39483292f899ad6b637701e664c19753528fe4ba1155c49145524eb74fc7" Oct 30 14:30:05.509744 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:30:05.509712 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-fde75d-29889510-mppvj_odh-ai-gateway-infra(6ecbf4bb-822a-4bd6-891c-c1eb6d12ecaa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fde75d-29889510-mppvj" podUID="6ecbf4bb-822a-4bd6-891c-c1eb6d12ecaa" Oct 30 14:30:05.511316 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:05.511293 4982 generic.go:356] "Generic (PLEG): container finished" podID="b60b5431-525a-4363-a3fb-3352fe6521e9" containerID="c31661dd58496bf307968b1a8b0161906aad890506a1c07756f230aef735950c" exitCode=60 Oct 30 14:30:05.511401 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:05.511348 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889510-v8wkp" event={"ID":"b60b5431-525a-4363-a3fb-3352fe6521e9","Type":"ContainerDied","Data":"c31661dd58496bf307968b1a8b0161906aad890506a1c07756f230aef735950c"} Oct 30 14:30:05.511597 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:05.511582 4982 scope.go:122] "RemoveContainer" containerID="c31661dd58496bf307968b1a8b0161906aad890506a1c07756f230aef735950c" Oct 30 14:30:05.511852 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:30:05.511831 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889510-v8wkp_odh-ai-gateway-infra(b60b5431-525a-4363-a3fb-3352fe6521e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889510-v8wkp" podUID="b60b5431-525a-4363-a3fb-3352fe6521e9" Oct 30 14:30:05.523431 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:05.523411 4982 scope.go:122] "RemoveContainer" containerID="b95145216667042d559e20eb553c7ad77bb3f9d999a7b90f144b02ee00292ac0" Oct 30 14:30:05.537044 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:05.536947 4982 scope.go:122] "RemoveContainer" containerID="55ee5b6855557f69000b266d374d579f048aeee4534a1c01e5d9b273ef6a0052" Oct 30 14:30:05.555210 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:05.555188 4982 scope.go:122] "RemoveContainer" containerID="704f9ce9a2d54d132cbf260644ba373966e967c39b50a21ecb9f8be489901403" Oct 30 14:30:05.568196 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:05.568174 4982 scope.go:122] "RemoveContainer" containerID="26f5192455543a1f13af0223626023bb11db69252c996256c3282f4327b97a53" Oct 30 14:30:05.584484 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:05.584461 4982 scope.go:122] "RemoveContainer" containerID="62ec2dcdd1a4a542d3f1be19f49b5aa713e8774ff6f1a1a54b758f1587e213d2" Oct 30 14:30:05.599305 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:05.599279 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54495360-zbkbn"] Oct 30 14:30:05.599622 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:05.599544 4982 scope.go:122] "RemoveContainer" containerID="d1cddbb3997dc1125e4a80f38bca1f4c30e466e41689cffb008977e7e9f6c12f" Oct 30 14:30:05.606014 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:05.605991 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54495360-zbkbn" Oct 30 14:30:05.623611 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:05.623590 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:30:05.628074 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:05.628053 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54495360-zbkbn"] Oct 30 14:30:05.655967 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:05.655935 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f3320ba-10c9-4018-a33b-479d8b18d02e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54495360-zbkbn\" (UID: \"2f3320ba-10c9-4018-a33b-479d8b18d02e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54495360-zbkbn" Oct 30 14:30:05.656109 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:05.656046 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f3320ba-10c9-4018-a33b-479d8b18d02e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54495360-zbkbn\" (UID: \"2f3320ba-10c9-4018-a33b-479d8b18d02e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54495360-zbkbn" Oct 30 14:30:05.656203 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:05.656174 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk8fg\" (UniqueName: \"kubernetes.io/projected/2f3320ba-10c9-4018-a33b-479d8b18d02e-kube-api-access-fk8fg\") pod \"maas-api-revoke-sub-54495360-zbkbn\" (UID: \"2f3320ba-10c9-4018-a33b-479d8b18d02e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54495360-zbkbn" Oct 30 14:30:05.756123 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:05.755998 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:30:05.758773 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:05.758746 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f3320ba-10c9-4018-a33b-479d8b18d02e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54495360-zbkbn\" (UID: \"2f3320ba-10c9-4018-a33b-479d8b18d02e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54495360-zbkbn" Oct 30 14:30:05.759071 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:05.759050 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f3320ba-10c9-4018-a33b-479d8b18d02e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54495360-zbkbn\" (UID: \"2f3320ba-10c9-4018-a33b-479d8b18d02e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54495360-zbkbn" Oct 30 14:30:05.759267 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:05.759235 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk8fg\" (UniqueName: \"kubernetes.io/projected/2f3320ba-10c9-4018-a33b-479d8b18d02e-kube-api-access-fk8fg\") pod \"maas-api-revoke-sub-54495360-zbkbn\" (UID: \"2f3320ba-10c9-4018-a33b-479d8b18d02e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54495360-zbkbn" Oct 30 14:30:05.760938 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:05.760915 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f3320ba-10c9-4018-a33b-479d8b18d02e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54495360-zbkbn\" (UID: \"2f3320ba-10c9-4018-a33b-479d8b18d02e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54495360-zbkbn" Oct 30 14:30:05.766039 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:05.766012 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f3320ba-10c9-4018-a33b-479d8b18d02e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54495360-zbkbn\" (UID: \"2f3320ba-10c9-4018-a33b-479d8b18d02e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54495360-zbkbn" Oct 30 14:30:05.792745 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:05.792710 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk8fg\" (UniqueName: \"kubernetes.io/projected/2f3320ba-10c9-4018-a33b-479d8b18d02e-kube-api-access-fk8fg\") pod \"maas-api-revoke-sub-54495360-zbkbn\" (UID: \"2f3320ba-10c9-4018-a33b-479d8b18d02e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54495360-zbkbn" Oct 30 14:30:05.923511 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:05.923478 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54495360-zbkbn" Oct 30 14:30:06.238147 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:30:06.238112 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f3320ba_10c9_4018_a33b_479d8b18d02e.slice/crio-5bf8d7ddafddeef5c15cbf3007254f632f070ca9cddd83422bcf32cad76047f4 WatchSource:0}: Error finding container 5bf8d7ddafddeef5c15cbf3007254f632f070ca9cddd83422bcf32cad76047f4: Status 404 returned error can't find the container with id 5bf8d7ddafddeef5c15cbf3007254f632f070ca9cddd83422bcf32cad76047f4 Oct 30 14:30:06.239999 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:06.239972 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54495360-zbkbn"] Oct 30 14:30:06.523265 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:06.523179 4982 scope.go:122] "RemoveContainer" containerID="c31661dd58496bf307968b1a8b0161906aad890506a1c07756f230aef735950c" Oct 30 14:30:06.523737 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:30:06.523563 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889510-v8wkp_odh-ai-gateway-infra(b60b5431-525a-4363-a3fb-3352fe6521e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889510-v8wkp" podUID="b60b5431-525a-4363-a3fb-3352fe6521e9" Oct 30 14:30:06.524498 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:06.524466 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54495360-zbkbn" event={"ID":"2f3320ba-10c9-4018-a33b-479d8b18d02e","Type":"ContainerStarted","Data":"5bf8d7ddafddeef5c15cbf3007254f632f070ca9cddd83422bcf32cad76047f4"} Oct 30 14:30:06.526463 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:06.526439 4982 scope.go:122] "RemoveContainer" containerID="ed30e26901f71cfc6efe4db8c1fbc34b52d90ca7c3b8af0dcfb7b64f773bbe9f" Oct 30 14:30:06.526857 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:30:06.526830 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-39529e-29889510-ksc4p_odh-ai-gateway-infra(e7260a80-a857-497e-bb12-28d8b7fa8bab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889510-ksc4p" podUID="e7260a80-a857-497e-bb12-28d8b7fa8bab" Oct 30 14:30:06.528203 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:06.528183 4982 scope.go:122] "RemoveContainer" containerID="e2c77a41bf317b72f8fc6617c07497c4bee7b44ac464e2c0013bb89537db490a" Oct 30 14:30:06.528566 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:30:06.528539 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-34bcc5-29889510-hkg5s_odh-ai-gateway-infra(f300b507-503e-47e1-a09b-d907b975c2c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-34bcc5-29889510-hkg5s" podUID="f300b507-503e-47e1-a09b-d907b975c2c2" Oct 30 14:30:06.529859 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:06.529841 4982 scope.go:122] "RemoveContainer" containerID="ec103dd100f32b7f8c161fcf67c0283933e93455dec037f1da999db33cea3b94" Oct 30 14:30:06.530115 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:30:06.530086 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889510-jfwrx_odh-ai-gateway-infra(9246e72d-bf25-49e9-a530-78843f462b48)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889510-jfwrx" podUID="9246e72d-bf25-49e9-a530-78843f462b48" Oct 30 14:30:06.531546 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:06.531525 4982 scope.go:122] "RemoveContainer" containerID="9e7ade9b479ea9318f1aca444dcfec68d74c43b333fc0f4d177c962b1e2339e7" Oct 30 14:30:06.531814 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:30:06.531795 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-b9c4b4-29889510-dhhvd_odh-ai-gateway-infra(5c9a6cee-328e-461d-8e42-69c7bdf0afd6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b9c4b4-29889510-dhhvd" podUID="5c9a6cee-328e-461d-8e42-69c7bdf0afd6" Oct 30 14:30:06.533098 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:06.533082 4982 scope.go:122] "RemoveContainer" containerID="5913e1fb85dc10e3b3326d0092d8ea31064e56a8f653066429c7b81a64c98ac5" Oct 30 14:30:06.533323 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:30:06.533299 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-a9542c-29889510-s2w5r_odh-ai-gateway-infra(6f0959c9-4c3d-4394-baa1-6fd5a31fbe78)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a9542c-29889510-s2w5r" podUID="6f0959c9-4c3d-4394-baa1-6fd5a31fbe78" Oct 30 14:30:06.534684 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:06.534658 4982 scope.go:122] "RemoveContainer" containerID="0549ec03a578e279de1b6bfbdfa25107d5a1975c5bfa955e2bb0fe36453eca20" Oct 30 14:30:06.534899 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:30:06.534872 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-46b13b-29889510-cqz8m_odh-ai-gateway-infra(b1dda7b0-fb53-4fec-96ad-b8e04eb8b24b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889510-cqz8m" podUID="b1dda7b0-fb53-4fec-96ad-b8e04eb8b24b" Oct 30 14:30:06.536303 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:06.536283 4982 scope.go:122] "RemoveContainer" containerID="98efdc5b70442aabcbd62532d663463df50bca2aae6855fe5fa8bdeeb4f719a0" Oct 30 14:30:06.536505 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:30:06.536487 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-d15602-29889510-5jdpd_odh-ai-gateway-infra(bd736b21-6929-413d-8eb4-a9f891ffe64a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d15602-29889510-5jdpd" podUID="bd736b21-6929-413d-8eb4-a9f891ffe64a" Oct 30 14:30:07.556127 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:07.556034 4982 generic.go:356] "Generic (PLEG): container finished" podID="2f3320ba-10c9-4018-a33b-479d8b18d02e" containerID="8009de666b1b7c9c254d4317566cb4c88fed78e6847cfa6ec31dcd480c15b6aa" exitCode=0 Oct 30 14:30:07.556579 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:07.556146 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54495360-zbkbn" event={"ID":"2f3320ba-10c9-4018-a33b-479d8b18d02e","Type":"ContainerDied","Data":"8009de666b1b7c9c254d4317566cb4c88fed78e6847cfa6ec31dcd480c15b6aa"} Oct 30 14:30:07.814474 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:07.814111 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:30:08.744620 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:08.744595 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54495360-zbkbn" Oct 30 14:30:08.905674 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:08.905584 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f3320ba-10c9-4018-a33b-479d8b18d02e-kube-api-access-fk8fg\" (UniqueName: \"kubernetes.io/projected/2f3320ba-10c9-4018-a33b-479d8b18d02e-kube-api-access-fk8fg\") pod \"2f3320ba-10c9-4018-a33b-479d8b18d02e\" (UID: \"2f3320ba-10c9-4018-a33b-479d8b18d02e\") " Oct 30 14:30:08.905829 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:08.905796 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f3320ba-10c9-4018-a33b-479d8b18d02e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f3320ba-10c9-4018-a33b-479d8b18d02e-maas-api-xks-ca\") pod \"2f3320ba-10c9-4018-a33b-479d8b18d02e\" (UID: \"2f3320ba-10c9-4018-a33b-479d8b18d02e\") " Oct 30 14:30:08.905918 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:08.905850 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f3320ba-10c9-4018-a33b-479d8b18d02e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f3320ba-10c9-4018-a33b-479d8b18d02e-maas-api-openshift-ca\") pod \"2f3320ba-10c9-4018-a33b-479d8b18d02e\" (UID: \"2f3320ba-10c9-4018-a33b-479d8b18d02e\") " Oct 30 14:30:08.906223 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:08.906190 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f3320ba-10c9-4018-a33b-479d8b18d02e-maas-api-openshift-ca" pod "2f3320ba-10c9-4018-a33b-479d8b18d02e" (UID: "2f3320ba-10c9-4018-a33b-479d8b18d02e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:30:08.906465 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:08.906447 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f3320ba-10c9-4018-a33b-479d8b18d02e-maas-api-openshift-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:30:08.908185 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:08.908159 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f3320ba-10c9-4018-a33b-479d8b18d02e-maas-api-xks-ca" pod "2f3320ba-10c9-4018-a33b-479d8b18d02e" (UID: "2f3320ba-10c9-4018-a33b-479d8b18d02e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:30:08.908295 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:08.908215 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f3320ba-10c9-4018-a33b-479d8b18d02e-kube-api-access-fk8fg" pod "2f3320ba-10c9-4018-a33b-479d8b18d02e" (UID: "2f3320ba-10c9-4018-a33b-479d8b18d02e"). InnerVolumeSpecName "kube-api-access-fk8fg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:30:09.008182 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:09.008149 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f3320ba-10c9-4018-a33b-479d8b18d02e-maas-api-xks-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:30:09.008317 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:09.008185 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fk8fg\" (UniqueName: \"kubernetes.io/projected/2f3320ba-10c9-4018-a33b-479d8b18d02e-kube-api-access-fk8fg\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:30:09.571355 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:09.571321 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54495360-zbkbn" event={"ID":"2f3320ba-10c9-4018-a33b-479d8b18d02e","Type":"ContainerDied","Data":"5bf8d7ddafddeef5c15cbf3007254f632f070ca9cddd83422bcf32cad76047f4"} Oct 30 14:30:09.571612 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:09.571375 4982 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5bf8d7ddafddeef5c15cbf3007254f632f070ca9cddd83422bcf32cad76047f4" Oct 30 14:30:09.571612 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:09.571330 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54495360-zbkbn" Oct 30 14:30:09.726686 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:09.726145 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:30:11.669633 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:11.669581 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:30:13.439519 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:13.439477 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:30:15.120439 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:15.120400 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:30:15.392862 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:15.392777 4982 scope.go:122] "RemoveContainer" containerID="183898b76a0885019156b162baf93cfc02dc321e5487e4932d39ae4905e73b3a" Oct 30 14:30:15.393027 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:15.392961 4982 scope.go:122] "RemoveContainer" containerID="f77f39483292f899ad6b637701e664c19753528fe4ba1155c49145524eb74fc7" Oct 30 14:30:15.393121 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:15.393104 4982 scope.go:122] "RemoveContainer" containerID="9e7ade9b479ea9318f1aca444dcfec68d74c43b333fc0f4d177c962b1e2339e7" Oct 30 14:30:15.483705 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:15.478088 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:30:16.392872 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:16.392845 4982 scope.go:122] "RemoveContainer" containerID="c31661dd58496bf307968b1a8b0161906aad890506a1c07756f230aef735950c" Oct 30 14:30:16.637979 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:16.637888 4982 generic.go:356] "Generic (PLEG): container finished" podID="fc3717af-2029-422a-9230-b41ebc4dedc5" containerID="be3be95ce588703393c9ae96869ef4f0211ce7a03d7178e0ad91ab87030008fa" exitCode=60 Oct 30 14:30:16.637979 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:16.637958 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-acefbb-29889510-7tmmg" event={"ID":"fc3717af-2029-422a-9230-b41ebc4dedc5","Type":"ContainerDied","Data":"be3be95ce588703393c9ae96869ef4f0211ce7a03d7178e0ad91ab87030008fa"} Oct 30 14:30:16.638180 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:16.638021 4982 scope.go:122] "RemoveContainer" containerID="183898b76a0885019156b162baf93cfc02dc321e5487e4932d39ae4905e73b3a" Oct 30 14:30:16.638480 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:16.638459 4982 scope.go:122] "RemoveContainer" containerID="be3be95ce588703393c9ae96869ef4f0211ce7a03d7178e0ad91ab87030008fa" Oct 30 14:30:16.638844 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:30:16.638821 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-acefbb-29889510-7tmmg_odh-ai-gateway-infra(fc3717af-2029-422a-9230-b41ebc4dedc5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-acefbb-29889510-7tmmg" podUID="fc3717af-2029-422a-9230-b41ebc4dedc5" Oct 30 14:30:16.640020 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:16.640001 4982 generic.go:356] "Generic (PLEG): container finished" podID="5c9a6cee-328e-461d-8e42-69c7bdf0afd6" containerID="8f1bb8c10e04d7d178e52f5782e854b306963993e022a67c8101c7be4e65863e" exitCode=60 Oct 30 14:30:16.640110 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:16.640072 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b9c4b4-29889510-dhhvd" event={"ID":"5c9a6cee-328e-461d-8e42-69c7bdf0afd6","Type":"ContainerDied","Data":"8f1bb8c10e04d7d178e52f5782e854b306963993e022a67c8101c7be4e65863e"} Oct 30 14:30:16.640460 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:16.640438 4982 scope.go:122] "RemoveContainer" containerID="8f1bb8c10e04d7d178e52f5782e854b306963993e022a67c8101c7be4e65863e" Oct 30 14:30:16.640854 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:30:16.640828 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-b9c4b4-29889510-dhhvd_odh-ai-gateway-infra(5c9a6cee-328e-461d-8e42-69c7bdf0afd6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b9c4b4-29889510-dhhvd" podUID="5c9a6cee-328e-461d-8e42-69c7bdf0afd6" Oct 30 14:30:16.642184 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:16.642165 4982 generic.go:356] "Generic (PLEG): container finished" podID="6ecbf4bb-822a-4bd6-891c-c1eb6d12ecaa" containerID="f1ed2990332ea5e3c3fb302b13e26aa2817966b832d509614f5859ccef6da97b" exitCode=60 Oct 30 14:30:16.642288 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:16.642232 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fde75d-29889510-mppvj" event={"ID":"6ecbf4bb-822a-4bd6-891c-c1eb6d12ecaa","Type":"ContainerDied","Data":"f1ed2990332ea5e3c3fb302b13e26aa2817966b832d509614f5859ccef6da97b"} Oct 30 14:30:16.642563 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:16.642547 4982 scope.go:122] "RemoveContainer" containerID="f1ed2990332ea5e3c3fb302b13e26aa2817966b832d509614f5859ccef6da97b" Oct 30 14:30:16.642838 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:30:16.642818 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-fde75d-29889510-mppvj_odh-ai-gateway-infra(6ecbf4bb-822a-4bd6-891c-c1eb6d12ecaa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fde75d-29889510-mppvj" podUID="6ecbf4bb-822a-4bd6-891c-c1eb6d12ecaa" Oct 30 14:30:16.763293 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:16.763270 4982 scope.go:122] "RemoveContainer" containerID="9e7ade9b479ea9318f1aca444dcfec68d74c43b333fc0f4d177c962b1e2339e7" Oct 30 14:30:16.777724 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:16.777703 4982 scope.go:122] "RemoveContainer" containerID="f77f39483292f899ad6b637701e664c19753528fe4ba1155c49145524eb74fc7" Oct 30 14:30:17.365274 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:17.365199 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:30:17.394060 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:17.392219 4982 scope.go:122] "RemoveContainer" containerID="00b4fdca412fd23c64a08340d45d2d2607984121653f8df33b81d092e2e4d6ad" Oct 30 14:30:17.394060 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:17.393356 4982 scope.go:122] "RemoveContainer" containerID="ec103dd100f32b7f8c161fcf67c0283933e93455dec037f1da999db33cea3b94" Oct 30 14:30:17.653592 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:17.653513 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889510-v8wkp" event={"ID":"b60b5431-525a-4363-a3fb-3352fe6521e9","Type":"ContainerStarted","Data":"af188a5a12d1d413ba762c873069c175c3c1320558a70a86db6e6733b4044fd1"} Oct 30 14:30:17.718465 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:17.718424 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-acefbb-29889510-7tmmg"] Oct 30 14:30:17.722534 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:17.722504 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a9542c-69bcb5578b-rzp2q"] Oct 30 14:30:17.751371 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:17.751341 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a9542c-794b65877f-sp225"] Oct 30 14:30:17.772016 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:17.771968 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fde75d-29889510-mppvj"] Oct 30 14:30:17.833945 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:17.832866 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b9c4b4-29889510-dhhvd"] Oct 30 14:30:17.879571 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:17.879520 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:30:18.043029 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:18.042879 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:30:18.059260 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:18.059126 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-acefbb-29889510-7tmmg" Oct 30 14:30:18.085059 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:18.085030 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc3717af-2029-422a-9230-b41ebc4dedc5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc3717af-2029-422a-9230-b41ebc4dedc5-maas-api-service-ca\") pod \"fc3717af-2029-422a-9230-b41ebc4dedc5\" (UID: \"fc3717af-2029-422a-9230-b41ebc4dedc5\") " Oct 30 14:30:18.085324 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:18.085311 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc3717af-2029-422a-9230-b41ebc4dedc5-kube-api-access-d998f\" (UniqueName: \"kubernetes.io/projected/fc3717af-2029-422a-9230-b41ebc4dedc5-kube-api-access-d998f\") pod \"fc3717af-2029-422a-9230-b41ebc4dedc5\" (UID: \"fc3717af-2029-422a-9230-b41ebc4dedc5\") " Oct 30 14:30:18.086540 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:18.086515 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc3717af-2029-422a-9230-b41ebc4dedc5-maas-api-service-ca" pod "fc3717af-2029-422a-9230-b41ebc4dedc5" (UID: "fc3717af-2029-422a-9230-b41ebc4dedc5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:30:18.113031 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:18.112850 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc3717af-2029-422a-9230-b41ebc4dedc5-kube-api-access-d998f" pod "fc3717af-2029-422a-9230-b41ebc4dedc5" (UID: "fc3717af-2029-422a-9230-b41ebc4dedc5"). InnerVolumeSpecName "kube-api-access-d998f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:30:18.152096 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:18.151827 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fde75d-29889510-mppvj" Oct 30 14:30:18.180098 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:18.180069 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b9c4b4-29889510-dhhvd" Oct 30 14:30:18.186868 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:18.186838 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ecbf4bb-822a-4bd6-891c-c1eb6d12ecaa-kube-api-access-4jzp4\" (UniqueName: \"kubernetes.io/projected/6ecbf4bb-822a-4bd6-891c-c1eb6d12ecaa-kube-api-access-4jzp4\") pod \"6ecbf4bb-822a-4bd6-891c-c1eb6d12ecaa\" (UID: \"6ecbf4bb-822a-4bd6-891c-c1eb6d12ecaa\") " Oct 30 14:30:18.187136 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:18.187102 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ecbf4bb-822a-4bd6-891c-c1eb6d12ecaa-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ecbf4bb-822a-4bd6-891c-c1eb6d12ecaa-maas-api-service-ca\") pod \"6ecbf4bb-822a-4bd6-891c-c1eb6d12ecaa\" (UID: \"6ecbf4bb-822a-4bd6-891c-c1eb6d12ecaa\") " Oct 30 14:30:18.187483 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:18.187456 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ecbf4bb-822a-4bd6-891c-c1eb6d12ecaa-maas-api-service-ca" pod "6ecbf4bb-822a-4bd6-891c-c1eb6d12ecaa" (UID: "6ecbf4bb-822a-4bd6-891c-c1eb6d12ecaa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:30:18.187769 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:18.187743 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc3717af-2029-422a-9230-b41ebc4dedc5-maas-api-service-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:30:18.187769 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:18.187765 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ecbf4bb-822a-4bd6-891c-c1eb6d12ecaa-maas-api-service-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:30:18.187949 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:18.187775 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d998f\" (UniqueName: \"kubernetes.io/projected/fc3717af-2029-422a-9230-b41ebc4dedc5-kube-api-access-d998f\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:30:18.188863 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:18.188840 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ecbf4bb-822a-4bd6-891c-c1eb6d12ecaa-kube-api-access-4jzp4" pod "6ecbf4bb-822a-4bd6-891c-c1eb6d12ecaa" (UID: "6ecbf4bb-822a-4bd6-891c-c1eb6d12ecaa"). InnerVolumeSpecName "kube-api-access-4jzp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:30:18.288592 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:18.288552 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c9a6cee-328e-461d-8e42-69c7bdf0afd6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c9a6cee-328e-461d-8e42-69c7bdf0afd6-maas-api-service-ca\") pod \"5c9a6cee-328e-461d-8e42-69c7bdf0afd6\" (UID: \"5c9a6cee-328e-461d-8e42-69c7bdf0afd6\") " Oct 30 14:30:18.288821 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:18.288607 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c9a6cee-328e-461d-8e42-69c7bdf0afd6-kube-api-access-fbb2z\" (UniqueName: \"kubernetes.io/projected/5c9a6cee-328e-461d-8e42-69c7bdf0afd6-kube-api-access-fbb2z\") pod \"5c9a6cee-328e-461d-8e42-69c7bdf0afd6\" (UID: \"5c9a6cee-328e-461d-8e42-69c7bdf0afd6\") " Oct 30 14:30:18.288960 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:18.288935 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c9a6cee-328e-461d-8e42-69c7bdf0afd6-maas-api-service-ca" pod "5c9a6cee-328e-461d-8e42-69c7bdf0afd6" (UID: "5c9a6cee-328e-461d-8e42-69c7bdf0afd6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:30:18.289575 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:18.289545 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c9a6cee-328e-461d-8e42-69c7bdf0afd6-maas-api-service-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:30:18.289721 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:18.289579 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jzp4\" (UniqueName: \"kubernetes.io/projected/6ecbf4bb-822a-4bd6-891c-c1eb6d12ecaa-kube-api-access-4jzp4\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:30:18.291167 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:18.291139 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c9a6cee-328e-461d-8e42-69c7bdf0afd6-kube-api-access-fbb2z" pod "5c9a6cee-328e-461d-8e42-69c7bdf0afd6" (UID: "5c9a6cee-328e-461d-8e42-69c7bdf0afd6"). InnerVolumeSpecName "kube-api-access-fbb2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:30:18.391753 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:18.390847 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbb2z\" (UniqueName: \"kubernetes.io/projected/5c9a6cee-328e-461d-8e42-69c7bdf0afd6-kube-api-access-fbb2z\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:30:18.412662 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:18.410239 4982 scope.go:122] "RemoveContainer" containerID="5913e1fb85dc10e3b3326d0092d8ea31064e56a8f653066429c7b81a64c98ac5" Oct 30 14:30:18.412662 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:18.411244 4982 scope.go:122] "RemoveContainer" containerID="98efdc5b70442aabcbd62532d663463df50bca2aae6855fe5fa8bdeeb4f719a0" Oct 30 14:30:18.412662 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:18.412518 4982 scope.go:122] "RemoveContainer" containerID="e2c77a41bf317b72f8fc6617c07497c4bee7b44ac464e2c0013bb89537db490a" Oct 30 14:30:18.413262 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:18.413221 4982 scope.go:122] "RemoveContainer" containerID="0549ec03a578e279de1b6bfbdfa25107d5a1975c5bfa955e2bb0fe36453eca20" Oct 30 14:30:18.666818 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:18.666786 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-acefbb-29889510-7tmmg" Oct 30 14:30:18.666990 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:18.666812 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-acefbb-29889510-7tmmg" event={"ID":"fc3717af-2029-422a-9230-b41ebc4dedc5","Type":"ContainerDied","Data":"6ad65578699a80bef71e218899ff92eed304bc19f9c696b288f15ee881b68e55"} Oct 30 14:30:18.666990 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:18.666881 4982 scope.go:122] "RemoveContainer" containerID="be3be95ce588703393c9ae96869ef4f0211ce7a03d7178e0ad91ab87030008fa" Oct 30 14:30:18.668818 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:18.668790 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b9c4b4-29889510-dhhvd" Oct 30 14:30:18.669266 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:18.669243 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b9c4b4-29889510-dhhvd" event={"ID":"5c9a6cee-328e-461d-8e42-69c7bdf0afd6","Type":"ContainerDied","Data":"099af706c7f6150a380fb7f6ce02de66c036e836c467a25f995c8f38401334de"} Oct 30 14:30:18.671856 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:18.671831 4982 generic.go:356] "Generic (PLEG): container finished" podID="7d6da5d4-7fd8-4b0e-9a82-40c44fc77816" containerID="d77a4d929e29d20ab9ed79bf0430dd609925c95620b49d87813338f3c83798b1" exitCode=60 Oct 30 14:30:18.671971 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:18.671908 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-93549e-29889510-w9dvh" event={"ID":"7d6da5d4-7fd8-4b0e-9a82-40c44fc77816","Type":"ContainerDied","Data":"d77a4d929e29d20ab9ed79bf0430dd609925c95620b49d87813338f3c83798b1"} Oct 30 14:30:18.672373 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:18.672355 4982 scope.go:122] "RemoveContainer" containerID="d77a4d929e29d20ab9ed79bf0430dd609925c95620b49d87813338f3c83798b1" Oct 30 14:30:18.672683 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:30:18.672662 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-93549e-29889510-w9dvh_odh-ai-gateway-infra(7d6da5d4-7fd8-4b0e-9a82-40c44fc77816)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-93549e-29889510-w9dvh" podUID="7d6da5d4-7fd8-4b0e-9a82-40c44fc77816" Oct 30 14:30:18.673397 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:18.673379 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fde75d-29889510-mppvj" Oct 30 14:30:18.673486 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:18.673418 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fde75d-29889510-mppvj" event={"ID":"6ecbf4bb-822a-4bd6-891c-c1eb6d12ecaa","Type":"ContainerDied","Data":"93a7dac0855cb4ba0f463860fd11cd7fe31257beb85e3af70efce74891313212"} Oct 30 14:30:18.675243 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:18.675219 4982 generic.go:356] "Generic (PLEG): container finished" podID="b60b5431-525a-4363-a3fb-3352fe6521e9" containerID="af188a5a12d1d413ba762c873069c175c3c1320558a70a86db6e6733b4044fd1" exitCode=60 Oct 30 14:30:18.675342 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:18.675289 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889510-v8wkp" event={"ID":"b60b5431-525a-4363-a3fb-3352fe6521e9","Type":"ContainerDied","Data":"af188a5a12d1d413ba762c873069c175c3c1320558a70a86db6e6733b4044fd1"} Oct 30 14:30:18.675703 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:18.675686 4982 scope.go:122] "RemoveContainer" containerID="af188a5a12d1d413ba762c873069c175c3c1320558a70a86db6e6733b4044fd1" Oct 30 14:30:18.676285 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:30:18.676011 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889510-v8wkp_odh-ai-gateway-infra(b60b5431-525a-4363-a3fb-3352fe6521e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889510-v8wkp" podUID="b60b5431-525a-4363-a3fb-3352fe6521e9" Oct 30 14:30:18.677542 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:18.677522 4982 generic.go:356] "Generic (PLEG): container finished" podID="9246e72d-bf25-49e9-a530-78843f462b48" containerID="f56ceddad24d4715647179074df7c8486309f95a9b3216186b06eecbeb5292a8" exitCode=60 Oct 30 14:30:18.677626 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:18.677545 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889510-jfwrx" event={"ID":"9246e72d-bf25-49e9-a530-78843f462b48","Type":"ContainerDied","Data":"f56ceddad24d4715647179074df7c8486309f95a9b3216186b06eecbeb5292a8"} Oct 30 14:30:18.677900 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:18.677884 4982 scope.go:122] "RemoveContainer" containerID="f56ceddad24d4715647179074df7c8486309f95a9b3216186b06eecbeb5292a8" Oct 30 14:30:18.678116 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:30:18.678091 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889510-jfwrx_odh-ai-gateway-infra(9246e72d-bf25-49e9-a530-78843f462b48)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889510-jfwrx" podUID="9246e72d-bf25-49e9-a530-78843f462b48" Oct 30 14:30:18.685853 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:18.681214 4982 scope.go:122] "RemoveContainer" containerID="8f1bb8c10e04d7d178e52f5782e854b306963993e022a67c8101c7be4e65863e" Oct 30 14:30:18.696970 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:18.696881 4982 scope.go:122] "RemoveContainer" containerID="00b4fdca412fd23c64a08340d45d2d2607984121653f8df33b81d092e2e4d6ad" Oct 30 14:30:18.712257 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:18.712221 4982 scope.go:122] "RemoveContainer" containerID="f1ed2990332ea5e3c3fb302b13e26aa2817966b832d509614f5859ccef6da97b" Oct 30 14:30:18.724819 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:18.724789 4982 scope.go:122] "RemoveContainer" containerID="c31661dd58496bf307968b1a8b0161906aad890506a1c07756f230aef735950c" Oct 30 14:30:18.735335 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:18.735310 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b9c4b4-29889510-dhhvd"] Oct 30 14:30:18.739594 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:18.739574 4982 scope.go:122] "RemoveContainer" containerID="ec103dd100f32b7f8c161fcf67c0283933e93455dec037f1da999db33cea3b94" Oct 30 14:30:18.752479 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:18.751809 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b9c4b4-29889510-dhhvd"] Oct 30 14:30:18.760188 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:18.760156 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889510-v8wkp"] Oct 30 14:30:18.791665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:18.791620 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-acefbb-29889510-7tmmg"] Oct 30 14:30:18.799095 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:18.799063 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-acefbb-29889510-7tmmg"] Oct 30 14:30:18.972159 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:18.972076 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fde75d-29889510-mppvj"] Oct 30 14:30:18.981248 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:18.981099 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fde75d-29889510-mppvj"] Oct 30 14:30:19.392901 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:19.392867 4982 scope.go:122] "RemoveContainer" containerID="ed30e26901f71cfc6efe4db8c1fbc34b52d90ca7c3b8af0dcfb7b64f773bbe9f" Oct 30 14:30:19.461803 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:19.461776 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:30:19.697409 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:19.697380 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-34bcc5-29889510-hkg5s" event={"ID":"f300b507-503e-47e1-a09b-d907b975c2c2","Type":"ContainerStarted","Data":"2ba8a88414b30205bfd901027f2b5a9370db13bc6c7941e4ab4ca4724a1b7f1a"} Oct 30 14:30:19.712194 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:19.712163 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a9542c-29889510-s2w5r" event={"ID":"6f0959c9-4c3d-4394-baa1-6fd5a31fbe78","Type":"ContainerStarted","Data":"491776f5552824883c573ce2e8392fead760154d508ccbc248c9a0031baab0a4"} Oct 30 14:30:19.712593 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:19.712556 4982 scope.go:122] "RemoveContainer" containerID="491776f5552824883c573ce2e8392fead760154d508ccbc248c9a0031baab0a4" Oct 30 14:30:19.713054 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:30:19.713010 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-a9542c-29889510-s2w5r_odh-ai-gateway-infra(6f0959c9-4c3d-4394-baa1-6fd5a31fbe78)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a9542c-29889510-s2w5r" podUID="6f0959c9-4c3d-4394-baa1-6fd5a31fbe78" Oct 30 14:30:19.716898 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:19.716873 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889510-cqz8m" event={"ID":"b1dda7b0-fb53-4fec-96ad-b8e04eb8b24b","Type":"ContainerStarted","Data":"793afb86ec29df4f0323cd02b2a0852ad9ea65fcb7cf27f5cd3e7e6b00d77ce4"} Oct 30 14:30:19.721196 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:19.721174 4982 generic.go:356] "Generic (PLEG): container finished" podID="bd736b21-6929-413d-8eb4-a9f891ffe64a" containerID="25ad5a059e092e59457b2fec421c263db926d12532db64eb493a96891ee3f8f1" exitCode=60 Oct 30 14:30:19.721368 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:19.721219 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d15602-29889510-5jdpd" event={"ID":"bd736b21-6929-413d-8eb4-a9f891ffe64a","Type":"ContainerDied","Data":"25ad5a059e092e59457b2fec421c263db926d12532db64eb493a96891ee3f8f1"} Oct 30 14:30:19.721498 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:19.721485 4982 scope.go:122] "RemoveContainer" containerID="98efdc5b70442aabcbd62532d663463df50bca2aae6855fe5fa8bdeeb4f719a0" Oct 30 14:30:19.722032 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:19.722011 4982 scope.go:122] "RemoveContainer" containerID="25ad5a059e092e59457b2fec421c263db926d12532db64eb493a96891ee3f8f1" Oct 30 14:30:19.722386 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:30:19.722362 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-d15602-29889510-5jdpd_odh-ai-gateway-infra(bd736b21-6929-413d-8eb4-a9f891ffe64a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d15602-29889510-5jdpd" podUID="bd736b21-6929-413d-8eb4-a9f891ffe64a" Oct 30 14:30:19.900362 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:19.900291 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-93549e-29889510-w9dvh"] Oct 30 14:30:19.928864 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:19.927434 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889510-v8wkp" Oct 30 14:30:19.940900 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:19.940870 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889510-jfwrx"] Oct 30 14:30:20.045932 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:20.045896 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b60b5431-525a-4363-a3fb-3352fe6521e9-kube-api-access-ljzdg\" (UniqueName: \"kubernetes.io/projected/b60b5431-525a-4363-a3fb-3352fe6521e9-kube-api-access-ljzdg\") pod \"b60b5431-525a-4363-a3fb-3352fe6521e9\" (UID: \"b60b5431-525a-4363-a3fb-3352fe6521e9\") " Oct 30 14:30:20.046065 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:20.045977 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b60b5431-525a-4363-a3fb-3352fe6521e9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b60b5431-525a-4363-a3fb-3352fe6521e9-maas-api-service-ca\") pod \"b60b5431-525a-4363-a3fb-3352fe6521e9\" (UID: \"b60b5431-525a-4363-a3fb-3352fe6521e9\") " Oct 30 14:30:20.047000 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:20.046971 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b60b5431-525a-4363-a3fb-3352fe6521e9-maas-api-service-ca" pod "b60b5431-525a-4363-a3fb-3352fe6521e9" (UID: "b60b5431-525a-4363-a3fb-3352fe6521e9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:30:20.050044 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:20.050015 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b60b5431-525a-4363-a3fb-3352fe6521e9-kube-api-access-ljzdg" pod "b60b5431-525a-4363-a3fb-3352fe6521e9" (UID: "b60b5431-525a-4363-a3fb-3352fe6521e9"). InnerVolumeSpecName "kube-api-access-ljzdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:30:20.090122 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:20.090090 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-93549e-29889510-w9dvh" Oct 30 14:30:20.148072 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:20.148033 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d6da5d4-7fd8-4b0e-9a82-40c44fc77816-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d6da5d4-7fd8-4b0e-9a82-40c44fc77816-maas-api-service-ca\") pod \"7d6da5d4-7fd8-4b0e-9a82-40c44fc77816\" (UID: \"7d6da5d4-7fd8-4b0e-9a82-40c44fc77816\") " Oct 30 14:30:20.148265 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:20.148169 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d6da5d4-7fd8-4b0e-9a82-40c44fc77816-kube-api-access-w6nc2\" (UniqueName: \"kubernetes.io/projected/7d6da5d4-7fd8-4b0e-9a82-40c44fc77816-kube-api-access-w6nc2\") pod \"7d6da5d4-7fd8-4b0e-9a82-40c44fc77816\" (UID: \"7d6da5d4-7fd8-4b0e-9a82-40c44fc77816\") " Oct 30 14:30:20.148940 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:20.148903 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljzdg\" (UniqueName: \"kubernetes.io/projected/b60b5431-525a-4363-a3fb-3352fe6521e9-kube-api-access-ljzdg\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:30:20.148940 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:20.148937 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b60b5431-525a-4363-a3fb-3352fe6521e9-maas-api-service-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:30:20.150059 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:20.150034 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d6da5d4-7fd8-4b0e-9a82-40c44fc77816-maas-api-service-ca" pod "7d6da5d4-7fd8-4b0e-9a82-40c44fc77816" (UID: "7d6da5d4-7fd8-4b0e-9a82-40c44fc77816"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:30:20.152315 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:20.152249 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d6da5d4-7fd8-4b0e-9a82-40c44fc77816-kube-api-access-w6nc2" pod "7d6da5d4-7fd8-4b0e-9a82-40c44fc77816" (UID: "7d6da5d4-7fd8-4b0e-9a82-40c44fc77816"). InnerVolumeSpecName "kube-api-access-w6nc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:30:20.191939 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:20.191911 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889510-jfwrx" Oct 30 14:30:20.249469 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:20.249435 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9246e72d-bf25-49e9-a530-78843f462b48-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9246e72d-bf25-49e9-a530-78843f462b48-maas-api-service-ca\") pod \"9246e72d-bf25-49e9-a530-78843f462b48\" (UID: \"9246e72d-bf25-49e9-a530-78843f462b48\") " Oct 30 14:30:20.249654 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:20.249585 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9246e72d-bf25-49e9-a530-78843f462b48-kube-api-access-xqf49\" (UniqueName: \"kubernetes.io/projected/9246e72d-bf25-49e9-a530-78843f462b48-kube-api-access-xqf49\") pod \"9246e72d-bf25-49e9-a530-78843f462b48\" (UID: \"9246e72d-bf25-49e9-a530-78843f462b48\") " Oct 30 14:30:20.249917 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:20.249883 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9246e72d-bf25-49e9-a530-78843f462b48-maas-api-service-ca" pod "9246e72d-bf25-49e9-a530-78843f462b48" (UID: "9246e72d-bf25-49e9-a530-78843f462b48"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:30:20.250057 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:20.250006 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9246e72d-bf25-49e9-a530-78843f462b48-maas-api-service-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:30:20.250057 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:20.250026 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d6da5d4-7fd8-4b0e-9a82-40c44fc77816-maas-api-service-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:30:20.250057 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:20.250042 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6nc2\" (UniqueName: \"kubernetes.io/projected/7d6da5d4-7fd8-4b0e-9a82-40c44fc77816-kube-api-access-w6nc2\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:30:20.252022 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:20.251990 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9246e72d-bf25-49e9-a530-78843f462b48-kube-api-access-xqf49" pod "9246e72d-bf25-49e9-a530-78843f462b48" (UID: "9246e72d-bf25-49e9-a530-78843f462b48"). InnerVolumeSpecName "kube-api-access-xqf49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:30:20.351349 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:20.351297 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqf49\" (UniqueName: \"kubernetes.io/projected/9246e72d-bf25-49e9-a530-78843f462b48-kube-api-access-xqf49\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:30:20.357548 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:20.357519 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:30:20.401580 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:20.401550 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c9a6cee-328e-461d-8e42-69c7bdf0afd6" path="/var/lib/kubelet/pods/5c9a6cee-328e-461d-8e42-69c7bdf0afd6/volumes" Oct 30 14:30:20.402133 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:20.402116 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ecbf4bb-822a-4bd6-891c-c1eb6d12ecaa" path="/var/lib/kubelet/pods/6ecbf4bb-822a-4bd6-891c-c1eb6d12ecaa/volumes" Oct 30 14:30:20.402546 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:20.402509 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc3717af-2029-422a-9230-b41ebc4dedc5" path="/var/lib/kubelet/pods/fc3717af-2029-422a-9230-b41ebc4dedc5/volumes" Oct 30 14:30:20.733187 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:20.733104 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889510-v8wkp" Oct 30 14:30:20.733187 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:20.733126 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889510-v8wkp" event={"ID":"b60b5431-525a-4363-a3fb-3352fe6521e9","Type":"ContainerDied","Data":"fc1d09f160f284f1b8b49cc4349427b8aeba553c85ac3c145b6ee0c093f4ea74"} Oct 30 14:30:20.733712 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:20.733219 4982 scope.go:122] "RemoveContainer" containerID="af188a5a12d1d413ba762c873069c175c3c1320558a70a86db6e6733b4044fd1" Oct 30 14:30:20.735695 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:20.735632 4982 generic.go:356] "Generic (PLEG): container finished" podID="e7260a80-a857-497e-bb12-28d8b7fa8bab" containerID="699adcb5e3009c4c830186c8b60a8d610db14fd6c9e8e94f91201c4e9cd09e29" exitCode=60 Oct 30 14:30:20.735816 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:20.735671 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889510-ksc4p" event={"ID":"e7260a80-a857-497e-bb12-28d8b7fa8bab","Type":"ContainerDied","Data":"699adcb5e3009c4c830186c8b60a8d610db14fd6c9e8e94f91201c4e9cd09e29"} Oct 30 14:30:20.736174 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:20.736155 4982 scope.go:122] "RemoveContainer" containerID="699adcb5e3009c4c830186c8b60a8d610db14fd6c9e8e94f91201c4e9cd09e29" Oct 30 14:30:20.736527 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:30:20.736498 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-39529e-29889510-ksc4p_odh-ai-gateway-infra(e7260a80-a857-497e-bb12-28d8b7fa8bab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889510-ksc4p" podUID="e7260a80-a857-497e-bb12-28d8b7fa8bab" Oct 30 14:30:20.737928 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:20.737904 4982 generic.go:356] "Generic (PLEG): container finished" podID="f300b507-503e-47e1-a09b-d907b975c2c2" containerID="2ba8a88414b30205bfd901027f2b5a9370db13bc6c7941e4ab4ca4724a1b7f1a" exitCode=60 Oct 30 14:30:20.738030 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:20.737978 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-34bcc5-29889510-hkg5s" event={"ID":"f300b507-503e-47e1-a09b-d907b975c2c2","Type":"ContainerDied","Data":"2ba8a88414b30205bfd901027f2b5a9370db13bc6c7941e4ab4ca4724a1b7f1a"} Oct 30 14:30:20.738359 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:20.738342 4982 scope.go:122] "RemoveContainer" containerID="2ba8a88414b30205bfd901027f2b5a9370db13bc6c7941e4ab4ca4724a1b7f1a" Oct 30 14:30:20.738667 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:30:20.738610 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-34bcc5-29889510-hkg5s_odh-ai-gateway-infra(f300b507-503e-47e1-a09b-d907b975c2c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-34bcc5-29889510-hkg5s" podUID="f300b507-503e-47e1-a09b-d907b975c2c2" Oct 30 14:30:20.739521 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:20.739503 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889510-jfwrx" Oct 30 14:30:20.739618 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:20.739521 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889510-jfwrx" event={"ID":"9246e72d-bf25-49e9-a530-78843f462b48","Type":"ContainerDied","Data":"2592041f46d2bf30961c9ec7d1ab7ad706406b0e95216b200af6eabb9c97678d"} Oct 30 14:30:20.741885 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:20.741861 4982 generic.go:356] "Generic (PLEG): container finished" podID="6f0959c9-4c3d-4394-baa1-6fd5a31fbe78" containerID="491776f5552824883c573ce2e8392fead760154d508ccbc248c9a0031baab0a4" exitCode=60 Oct 30 14:30:20.742201 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:20.741936 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a9542c-29889510-s2w5r" event={"ID":"6f0959c9-4c3d-4394-baa1-6fd5a31fbe78","Type":"ContainerDied","Data":"491776f5552824883c573ce2e8392fead760154d508ccbc248c9a0031baab0a4"} Oct 30 14:30:20.742596 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:20.742579 4982 scope.go:122] "RemoveContainer" containerID="491776f5552824883c573ce2e8392fead760154d508ccbc248c9a0031baab0a4" Oct 30 14:30:20.742900 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:30:20.742865 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-a9542c-29889510-s2w5r_odh-ai-gateway-infra(6f0959c9-4c3d-4394-baa1-6fd5a31fbe78)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a9542c-29889510-s2w5r" podUID="6f0959c9-4c3d-4394-baa1-6fd5a31fbe78" Oct 30 14:30:20.743878 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:20.743859 4982 generic.go:356] "Generic (PLEG): container finished" podID="b1dda7b0-fb53-4fec-96ad-b8e04eb8b24b" containerID="793afb86ec29df4f0323cd02b2a0852ad9ea65fcb7cf27f5cd3e7e6b00d77ce4" exitCode=60 Oct 30 14:30:20.743963 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:20.743930 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889510-cqz8m" event={"ID":"b1dda7b0-fb53-4fec-96ad-b8e04eb8b24b","Type":"ContainerDied","Data":"793afb86ec29df4f0323cd02b2a0852ad9ea65fcb7cf27f5cd3e7e6b00d77ce4"} Oct 30 14:30:20.744274 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:20.744257 4982 scope.go:122] "RemoveContainer" containerID="793afb86ec29df4f0323cd02b2a0852ad9ea65fcb7cf27f5cd3e7e6b00d77ce4" Oct 30 14:30:20.744552 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:30:20.744531 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-46b13b-29889510-cqz8m_odh-ai-gateway-infra(b1dda7b0-fb53-4fec-96ad-b8e04eb8b24b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889510-cqz8m" podUID="b1dda7b0-fb53-4fec-96ad-b8e04eb8b24b" Oct 30 14:30:20.745320 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:20.745303 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-93549e-29889510-w9dvh" Oct 30 14:30:20.745427 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:20.745324 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-93549e-29889510-w9dvh" event={"ID":"7d6da5d4-7fd8-4b0e-9a82-40c44fc77816","Type":"ContainerDied","Data":"93fd554f97ec0ce03914ede4ec924a8c7defe651aa58d51bdf15097a8eb95793"} Oct 30 14:30:20.745775 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:20.745761 4982 scope.go:122] "RemoveContainer" containerID="ed30e26901f71cfc6efe4db8c1fbc34b52d90ca7c3b8af0dcfb7b64f773bbe9f" Oct 30 14:30:20.759054 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:20.759037 4982 scope.go:122] "RemoveContainer" containerID="e2c77a41bf317b72f8fc6617c07497c4bee7b44ac464e2c0013bb89537db490a" Oct 30 14:30:20.772786 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:20.772763 4982 scope.go:122] "RemoveContainer" containerID="f56ceddad24d4715647179074df7c8486309f95a9b3216186b06eecbeb5292a8" Oct 30 14:30:20.783690 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:20.783669 4982 scope.go:122] "RemoveContainer" containerID="5913e1fb85dc10e3b3326d0092d8ea31064e56a8f653066429c7b81a64c98ac5" Oct 30 14:30:20.791540 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:20.791518 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889510-v8wkp"] Oct 30 14:30:20.797907 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:20.797887 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a7bb0c-29889510-v8wkp"] Oct 30 14:30:20.806172 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:20.806148 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d15602-29889510-5jdpd"] Oct 30 14:30:20.807147 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:20.807125 4982 scope.go:122] "RemoveContainer" containerID="0549ec03a578e279de1b6bfbdfa25107d5a1975c5bfa955e2bb0fe36453eca20" Oct 30 14:30:20.836076 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:20.836044 4982 scope.go:122] "RemoveContainer" containerID="d77a4d929e29d20ab9ed79bf0430dd609925c95620b49d87813338f3c83798b1" Oct 30 14:30:20.852468 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:20.852438 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a9542c-29889510-s2w5r"] Oct 30 14:30:20.893368 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:20.890701 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-93549e-29889510-w9dvh"] Oct 30 14:30:20.904691 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:20.899043 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-93549e-29889510-w9dvh"] Oct 30 14:30:20.904691 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:20.903870 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-34bcc5-29889510-hkg5s"] Oct 30 14:30:20.971807 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:20.971776 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889510-cqz8m"] Oct 30 14:30:21.002143 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:20.999701 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889510-jfwrx"] Oct 30 14:30:21.006351 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:21.006324 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889510-jfwrx"] Oct 30 14:30:21.030009 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:21.029984 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d15602-29889510-5jdpd" Oct 30 14:30:21.066118 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:21.066084 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd736b21-6929-413d-8eb4-a9f891ffe64a-kube-api-access-lslhp\" (UniqueName: \"kubernetes.io/projected/bd736b21-6929-413d-8eb4-a9f891ffe64a-kube-api-access-lslhp\") pod \"bd736b21-6929-413d-8eb4-a9f891ffe64a\" (UID: \"bd736b21-6929-413d-8eb4-a9f891ffe64a\") " Oct 30 14:30:21.066296 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:21.066144 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd736b21-6929-413d-8eb4-a9f891ffe64a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd736b21-6929-413d-8eb4-a9f891ffe64a-maas-api-service-ca\") pod \"bd736b21-6929-413d-8eb4-a9f891ffe64a\" (UID: \"bd736b21-6929-413d-8eb4-a9f891ffe64a\") " Oct 30 14:30:21.067157 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:21.067126 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd736b21-6929-413d-8eb4-a9f891ffe64a-maas-api-service-ca" pod "bd736b21-6929-413d-8eb4-a9f891ffe64a" (UID: "bd736b21-6929-413d-8eb4-a9f891ffe64a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:30:21.076607 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:21.076576 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd736b21-6929-413d-8eb4-a9f891ffe64a-kube-api-access-lslhp" pod "bd736b21-6929-413d-8eb4-a9f891ffe64a" (UID: "bd736b21-6929-413d-8eb4-a9f891ffe64a"). InnerVolumeSpecName "kube-api-access-lslhp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:30:21.169079 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:21.169040 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lslhp\" (UniqueName: \"kubernetes.io/projected/bd736b21-6929-413d-8eb4-a9f891ffe64a-kube-api-access-lslhp\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:30:21.169079 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:21.169079 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd736b21-6929-413d-8eb4-a9f891ffe64a-maas-api-service-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:30:21.611578 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:21.611522 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:30:21.759341 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:21.759309 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d15602-29889510-5jdpd" Oct 30 14:30:21.759721 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:21.759316 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d15602-29889510-5jdpd" event={"ID":"bd736b21-6929-413d-8eb4-a9f891ffe64a","Type":"ContainerDied","Data":"88e6723e1bb5baf36c8f43a188c3985165381216ff1c460addb7420185d1a0e6"} Oct 30 14:30:21.759721 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:21.759498 4982 scope.go:122] "RemoveContainer" containerID="25ad5a059e092e59457b2fec421c263db926d12532db64eb493a96891ee3f8f1" Oct 30 14:30:21.855204 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:21.855165 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d15602-29889510-5jdpd"] Oct 30 14:30:21.860516 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:21.860390 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d15602-29889510-5jdpd"] Oct 30 14:30:21.977361 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:21.977328 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889510-cqz8m" Oct 30 14:30:22.087575 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:22.087545 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a9542c-29889510-s2w5r" Oct 30 14:30:22.088106 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:22.088076 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1dda7b0-fb53-4fec-96ad-b8e04eb8b24b-kube-api-access-8jlvf\" (UniqueName: \"kubernetes.io/projected/b1dda7b0-fb53-4fec-96ad-b8e04eb8b24b-kube-api-access-8jlvf\") pod \"b1dda7b0-fb53-4fec-96ad-b8e04eb8b24b\" (UID: \"b1dda7b0-fb53-4fec-96ad-b8e04eb8b24b\") " Oct 30 14:30:22.091191 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:22.091160 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1dda7b0-fb53-4fec-96ad-b8e04eb8b24b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1dda7b0-fb53-4fec-96ad-b8e04eb8b24b-maas-api-service-ca\") pod \"b1dda7b0-fb53-4fec-96ad-b8e04eb8b24b\" (UID: \"b1dda7b0-fb53-4fec-96ad-b8e04eb8b24b\") " Oct 30 14:30:22.092451 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:22.092422 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1dda7b0-fb53-4fec-96ad-b8e04eb8b24b-maas-api-service-ca" pod "b1dda7b0-fb53-4fec-96ad-b8e04eb8b24b" (UID: "b1dda7b0-fb53-4fec-96ad-b8e04eb8b24b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:30:22.095134 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:22.095096 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889510-ksc4p"] Oct 30 14:30:22.098312 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:22.098276 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1dda7b0-fb53-4fec-96ad-b8e04eb8b24b-kube-api-access-8jlvf" pod "b1dda7b0-fb53-4fec-96ad-b8e04eb8b24b" (UID: "b1dda7b0-fb53-4fec-96ad-b8e04eb8b24b"). InnerVolumeSpecName "kube-api-access-8jlvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:30:22.100073 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:22.100053 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-34bcc5-29889510-hkg5s" Oct 30 14:30:22.195677 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:22.195605 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f300b507-503e-47e1-a09b-d907b975c2c2-kube-api-access-r6d5d\" (UniqueName: \"kubernetes.io/projected/f300b507-503e-47e1-a09b-d907b975c2c2-kube-api-access-r6d5d\") pod \"f300b507-503e-47e1-a09b-d907b975c2c2\" (UID: \"f300b507-503e-47e1-a09b-d907b975c2c2\") " Oct 30 14:30:22.195874 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:22.195695 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f0959c9-4c3d-4394-baa1-6fd5a31fbe78-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f0959c9-4c3d-4394-baa1-6fd5a31fbe78-maas-api-service-ca\") pod \"6f0959c9-4c3d-4394-baa1-6fd5a31fbe78\" (UID: \"6f0959c9-4c3d-4394-baa1-6fd5a31fbe78\") " Oct 30 14:30:22.195874 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:22.195755 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f0959c9-4c3d-4394-baa1-6fd5a31fbe78-kube-api-access-xvfm6\" (UniqueName: \"kubernetes.io/projected/6f0959c9-4c3d-4394-baa1-6fd5a31fbe78-kube-api-access-xvfm6\") pod \"6f0959c9-4c3d-4394-baa1-6fd5a31fbe78\" (UID: \"6f0959c9-4c3d-4394-baa1-6fd5a31fbe78\") " Oct 30 14:30:22.196007 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:22.195885 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f300b507-503e-47e1-a09b-d907b975c2c2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f300b507-503e-47e1-a09b-d907b975c2c2-maas-api-service-ca\") pod \"f300b507-503e-47e1-a09b-d907b975c2c2\" (UID: \"f300b507-503e-47e1-a09b-d907b975c2c2\") " Oct 30 14:30:22.196589 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:22.196565 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jlvf\" (UniqueName: \"kubernetes.io/projected/b1dda7b0-fb53-4fec-96ad-b8e04eb8b24b-kube-api-access-8jlvf\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:30:22.196760 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:22.196738 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f0959c9-4c3d-4394-baa1-6fd5a31fbe78-maas-api-service-ca" pod "6f0959c9-4c3d-4394-baa1-6fd5a31fbe78" (UID: "6f0959c9-4c3d-4394-baa1-6fd5a31fbe78"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:30:22.196903 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:22.196887 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1dda7b0-fb53-4fec-96ad-b8e04eb8b24b-maas-api-service-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:30:22.198713 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:22.197545 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f300b507-503e-47e1-a09b-d907b975c2c2-maas-api-service-ca" pod "f300b507-503e-47e1-a09b-d907b975c2c2" (UID: "f300b507-503e-47e1-a09b-d907b975c2c2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:30:22.200046 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:22.200021 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f300b507-503e-47e1-a09b-d907b975c2c2-kube-api-access-r6d5d" pod "f300b507-503e-47e1-a09b-d907b975c2c2" (UID: "f300b507-503e-47e1-a09b-d907b975c2c2"). InnerVolumeSpecName "kube-api-access-r6d5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:30:22.206830 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:22.206784 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f0959c9-4c3d-4394-baa1-6fd5a31fbe78-kube-api-access-xvfm6" pod "6f0959c9-4c3d-4394-baa1-6fd5a31fbe78" (UID: "6f0959c9-4c3d-4394-baa1-6fd5a31fbe78"). InnerVolumeSpecName "kube-api-access-xvfm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:30:22.296313 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:22.296286 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889510-ksc4p" Oct 30 14:30:22.298472 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:22.298444 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvfm6\" (UniqueName: \"kubernetes.io/projected/6f0959c9-4c3d-4394-baa1-6fd5a31fbe78-kube-api-access-xvfm6\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:30:22.298472 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:22.298473 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f300b507-503e-47e1-a09b-d907b975c2c2-maas-api-service-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:30:22.298628 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:22.298482 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6d5d\" (UniqueName: \"kubernetes.io/projected/f300b507-503e-47e1-a09b-d907b975c2c2-kube-api-access-r6d5d\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:30:22.298628 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:22.298490 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f0959c9-4c3d-4394-baa1-6fd5a31fbe78-maas-api-service-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:30:22.399424 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:22.399384 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7260a80-a857-497e-bb12-28d8b7fa8bab-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7260a80-a857-497e-bb12-28d8b7fa8bab-maas-api-service-ca\") pod \"e7260a80-a857-497e-bb12-28d8b7fa8bab\" (UID: \"e7260a80-a857-497e-bb12-28d8b7fa8bab\") " Oct 30 14:30:22.399565 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:22.399487 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7260a80-a857-497e-bb12-28d8b7fa8bab-kube-api-access-jhhg9\" (UniqueName: \"kubernetes.io/projected/e7260a80-a857-497e-bb12-28d8b7fa8bab-kube-api-access-jhhg9\") pod \"e7260a80-a857-497e-bb12-28d8b7fa8bab\" (UID: \"e7260a80-a857-497e-bb12-28d8b7fa8bab\") " Oct 30 14:30:22.401480 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:22.401445 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7260a80-a857-497e-bb12-28d8b7fa8bab-maas-api-service-ca" pod "e7260a80-a857-497e-bb12-28d8b7fa8bab" (UID: "e7260a80-a857-497e-bb12-28d8b7fa8bab"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:30:22.402310 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:22.402285 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7260a80-a857-497e-bb12-28d8b7fa8bab-kube-api-access-jhhg9" pod "e7260a80-a857-497e-bb12-28d8b7fa8bab" (UID: "e7260a80-a857-497e-bb12-28d8b7fa8bab"). InnerVolumeSpecName "kube-api-access-jhhg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:30:22.414107 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:22.414080 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d6da5d4-7fd8-4b0e-9a82-40c44fc77816" path="/var/lib/kubelet/pods/7d6da5d4-7fd8-4b0e-9a82-40c44fc77816/volumes" Oct 30 14:30:22.414676 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:22.414659 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9246e72d-bf25-49e9-a530-78843f462b48" path="/var/lib/kubelet/pods/9246e72d-bf25-49e9-a530-78843f462b48/volumes" Oct 30 14:30:22.415187 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:22.415172 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b60b5431-525a-4363-a3fb-3352fe6521e9" path="/var/lib/kubelet/pods/b60b5431-525a-4363-a3fb-3352fe6521e9/volumes" Oct 30 14:30:22.416136 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:22.416118 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd736b21-6929-413d-8eb4-a9f891ffe64a" path="/var/lib/kubelet/pods/bd736b21-6929-413d-8eb4-a9f891ffe64a/volumes" Oct 30 14:30:22.501812 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:22.501785 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7260a80-a857-497e-bb12-28d8b7fa8bab-maas-api-service-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:30:22.501812 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:22.501811 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhhg9\" (UniqueName: \"kubernetes.io/projected/e7260a80-a857-497e-bb12-28d8b7fa8bab-kube-api-access-jhhg9\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:30:22.723030 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:22.722944 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-a9542c-69bcb5578b-rzp2q" podUID="73f0efb5-d969-4b91-8558-75d5ebe12a43" containerName="payload-processing" containerID="cri-o://2e9387560467b28eae38822c7ae72f2cfa3e72cde643a9697a62fd0751d45910" gracePeriod=25 Oct 30 14:30:22.751873 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:22.751840 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a9542c-794b65877f-sp225" podUID="cdf6bd0e-f6ff-49b9-8f1a-b90052fe7787" containerName="payload-pre-processing" containerID="cri-o://480cf0ec02d620b0a9ad9188b060c296e9bd12ee799147bb2fc1f907ba55c234" gracePeriod=25 Oct 30 14:30:22.779650 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:22.779607 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889510-ksc4p" event={"ID":"e7260a80-a857-497e-bb12-28d8b7fa8bab","Type":"ContainerDied","Data":"ebc66f77cc3288fd4e324db6735526a3ef5257316009730f7a47f62e3f535fba"} Oct 30 14:30:22.779650 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:22.779628 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889510-ksc4p" Oct 30 14:30:22.780092 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:22.779687 4982 scope.go:122] "RemoveContainer" containerID="699adcb5e3009c4c830186c8b60a8d610db14fd6c9e8e94f91201c4e9cd09e29" Oct 30 14:30:22.781310 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:22.781162 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-34bcc5-29889510-hkg5s" Oct 30 14:30:22.781310 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:22.781232 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-34bcc5-29889510-hkg5s" event={"ID":"f300b507-503e-47e1-a09b-d907b975c2c2","Type":"ContainerDied","Data":"dcbf54922feadf2342be99ee96951dd90b6dac54233eff75fb4fa9e38431563b"} Oct 30 14:30:22.783073 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:22.783052 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a9542c-29889510-s2w5r" event={"ID":"6f0959c9-4c3d-4394-baa1-6fd5a31fbe78","Type":"ContainerDied","Data":"0339b2aac535adaf8ab4aa0de0b596c6dd1703c26bc057480b82b4f654625cad"} Oct 30 14:30:22.783238 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:22.783072 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a9542c-29889510-s2w5r" Oct 30 14:30:22.784631 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:22.784610 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889510-cqz8m" event={"ID":"b1dda7b0-fb53-4fec-96ad-b8e04eb8b24b","Type":"ContainerDied","Data":"b082e3d5e798e49ffee84c42008ca277288f6a4c8291d41177d0ff9debcd7636"} Oct 30 14:30:22.784757 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:22.784692 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889510-cqz8m" Oct 30 14:30:22.792808 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:22.792786 4982 scope.go:122] "RemoveContainer" containerID="2ba8a88414b30205bfd901027f2b5a9370db13bc6c7941e4ab4ca4724a1b7f1a" Oct 30 14:30:22.803546 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:22.803518 4982 scope.go:122] "RemoveContainer" containerID="491776f5552824883c573ce2e8392fead760154d508ccbc248c9a0031baab0a4" Oct 30 14:30:22.815149 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:22.815128 4982 scope.go:122] "RemoveContainer" containerID="793afb86ec29df4f0323cd02b2a0852ad9ea65fcb7cf27f5cd3e7e6b00d77ce4" Oct 30 14:30:22.838186 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:22.838159 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a9542c-29889510-s2w5r"] Oct 30 14:30:22.851072 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:22.851044 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a9542c-29889510-s2w5r"] Oct 30 14:30:22.890721 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:22.890690 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-34bcc5-29889510-hkg5s"] Oct 30 14:30:22.897268 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:22.897239 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-34bcc5-29889510-hkg5s"] Oct 30 14:30:22.934654 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:22.934620 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889510-ksc4p"] Oct 30 14:30:22.939396 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:22.939373 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889510-ksc4p"] Oct 30 14:30:22.979092 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:22.979020 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889510-cqz8m"] Oct 30 14:30:22.984613 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:22.984587 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889510-cqz8m"] Oct 30 14:30:23.561799 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:23.561766 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:30:24.404418 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:24.404377 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f0959c9-4c3d-4394-baa1-6fd5a31fbe78" path="/var/lib/kubelet/pods/6f0959c9-4c3d-4394-baa1-6fd5a31fbe78/volumes" Oct 30 14:30:24.404949 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:24.404928 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1dda7b0-fb53-4fec-96ad-b8e04eb8b24b" path="/var/lib/kubelet/pods/b1dda7b0-fb53-4fec-96ad-b8e04eb8b24b/volumes" Oct 30 14:30:24.405372 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:24.405355 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7260a80-a857-497e-bb12-28d8b7fa8bab" path="/var/lib/kubelet/pods/e7260a80-a857-497e-bb12-28d8b7fa8bab/volumes" Oct 30 14:30:24.406134 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:24.406112 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f300b507-503e-47e1-a09b-d907b975c2c2" path="/var/lib/kubelet/pods/f300b507-503e-47e1-a09b-d907b975c2c2/volumes" Oct 30 14:30:24.595423 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:24.595387 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:30:25.422283 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:25.422242 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:30:27.361932 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:27.360783 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:30:28.016612 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:30:28.016565 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:30:29.394366 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:29.394334 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:30:29.540175 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:29.539280 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:30:30.894955 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:30.894903 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-a9542c-69bcb5578b-rzp2q" podUID="73f0efb5-d969-4b91-8558-75d5ebe12a43" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:30:30.895458 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:30.895399 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a9542c-794b65877f-sp225" podUID="cdf6bd0e-f6ff-49b9-8f1a-b90052fe7787" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:30:31.439779 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:31.439742 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:30:32.187150 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:32.187100 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:30:33.325071 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:33.325016 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:30:34.035818 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:34.035781 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20ce18b4-prz95"] Oct 30 14:30:34.048740 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:34.048712 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20ce18b4-prz95"] Oct 30 14:30:34.401594 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:34.401561 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0255e279-3912-48f3-8710-7ba964b2af0a" path="/var/lib/kubelet/pods/0255e279-3912-48f3-8710-7ba964b2af0a/volumes" Oct 30 14:30:35.301972 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:35.301904 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:30:37.106477 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:37.106442 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:30:38.928528 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:38.928493 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:30:40.889861 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:40.889826 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:30:40.894876 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:40.894844 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a9542c-794b65877f-sp225" podUID="cdf6bd0e-f6ff-49b9-8f1a-b90052fe7787" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:30:40.895031 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:40.894943 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-a9542c-69bcb5578b-rzp2q" podUID="73f0efb5-d969-4b91-8558-75d5ebe12a43" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:30:42.318130 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.318092 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d15602-847fx"] Oct 30 14:30:42.318730 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.318714 4982 cpu_manager.go:414] "containerMap: removing container" podUID="6f0959c9-4c3d-4394-baa1-6fd5a31fbe78" containerName="cleanup" Oct 30 14:30:42.318780 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.318731 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f0959c9-4c3d-4394-baa1-6fd5a31fbe78" containerName="cleanup" Oct 30 14:30:42.318780 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.318741 4982 cpu_manager.go:414] "containerMap: removing container" podUID="b1dda7b0-fb53-4fec-96ad-b8e04eb8b24b" containerName="cleanup" Oct 30 14:30:42.318780 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.318747 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1dda7b0-fb53-4fec-96ad-b8e04eb8b24b" containerName="cleanup" Oct 30 14:30:42.318780 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.318756 4982 cpu_manager.go:414] "containerMap: removing container" podUID="f300b507-503e-47e1-a09b-d907b975c2c2" containerName="cleanup" Oct 30 14:30:42.318780 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.318762 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f300b507-503e-47e1-a09b-d907b975c2c2" containerName="cleanup" Oct 30 14:30:42.318780 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.318773 4982 cpu_manager.go:414] "containerMap: removing container" podUID="bd736b21-6929-413d-8eb4-a9f891ffe64a" containerName="cleanup" Oct 30 14:30:42.318780 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.318778 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd736b21-6929-413d-8eb4-a9f891ffe64a" containerName="cleanup" Oct 30 14:30:42.318992 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.318785 4982 cpu_manager.go:414] "containerMap: removing container" podUID="b1dda7b0-fb53-4fec-96ad-b8e04eb8b24b" containerName="cleanup" Oct 30 14:30:42.318992 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.318790 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1dda7b0-fb53-4fec-96ad-b8e04eb8b24b" containerName="cleanup" Oct 30 14:30:42.318992 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.318797 4982 cpu_manager.go:414] "containerMap: removing container" podUID="fc3717af-2029-422a-9230-b41ebc4dedc5" containerName="cleanup" Oct 30 14:30:42.318992 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.318801 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc3717af-2029-422a-9230-b41ebc4dedc5" containerName="cleanup" Oct 30 14:30:42.318992 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.318807 4982 cpu_manager.go:414] "containerMap: removing container" podUID="f300b507-503e-47e1-a09b-d907b975c2c2" containerName="cleanup" Oct 30 14:30:42.318992 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.318811 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f300b507-503e-47e1-a09b-d907b975c2c2" containerName="cleanup" Oct 30 14:30:42.318992 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.318818 4982 cpu_manager.go:414] "containerMap: removing container" podUID="6f0959c9-4c3d-4394-baa1-6fd5a31fbe78" containerName="cleanup" Oct 30 14:30:42.318992 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.318823 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f0959c9-4c3d-4394-baa1-6fd5a31fbe78" containerName="cleanup" Oct 30 14:30:42.318992 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.318830 4982 cpu_manager.go:414] "containerMap: removing container" podUID="6ecbf4bb-822a-4bd6-891c-c1eb6d12ecaa" containerName="cleanup" Oct 30 14:30:42.318992 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.318835 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ecbf4bb-822a-4bd6-891c-c1eb6d12ecaa" containerName="cleanup" Oct 30 14:30:42.318992 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.318841 4982 cpu_manager.go:414] "containerMap: removing container" podUID="7d6da5d4-7fd8-4b0e-9a82-40c44fc77816" containerName="cleanup" Oct 30 14:30:42.318992 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.318845 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d6da5d4-7fd8-4b0e-9a82-40c44fc77816" containerName="cleanup" Oct 30 14:30:42.318992 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.318852 4982 cpu_manager.go:414] "containerMap: removing container" podUID="9246e72d-bf25-49e9-a530-78843f462b48" containerName="cleanup" Oct 30 14:30:42.318992 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.318856 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9246e72d-bf25-49e9-a530-78843f462b48" containerName="cleanup" Oct 30 14:30:42.318992 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.318865 4982 cpu_manager.go:414] "containerMap: removing container" podUID="f300b507-503e-47e1-a09b-d907b975c2c2" containerName="cleanup" Oct 30 14:30:42.318992 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.318869 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f300b507-503e-47e1-a09b-d907b975c2c2" containerName="cleanup" Oct 30 14:30:42.318992 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.318875 4982 cpu_manager.go:414] "containerMap: removing container" podUID="e7260a80-a857-497e-bb12-28d8b7fa8bab" containerName="cleanup" Oct 30 14:30:42.318992 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.318879 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7260a80-a857-497e-bb12-28d8b7fa8bab" containerName="cleanup" Oct 30 14:30:42.318992 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.318886 4982 cpu_manager.go:414] "containerMap: removing container" podUID="6ecbf4bb-822a-4bd6-891c-c1eb6d12ecaa" containerName="cleanup" Oct 30 14:30:42.318992 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.318890 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ecbf4bb-822a-4bd6-891c-c1eb6d12ecaa" containerName="cleanup" Oct 30 14:30:42.318992 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.318896 4982 cpu_manager.go:414] "containerMap: removing container" podUID="9246e72d-bf25-49e9-a530-78843f462b48" containerName="cleanup" Oct 30 14:30:42.318992 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.318900 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9246e72d-bf25-49e9-a530-78843f462b48" containerName="cleanup" Oct 30 14:30:42.318992 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.318908 4982 cpu_manager.go:414] "containerMap: removing container" podUID="7d6da5d4-7fd8-4b0e-9a82-40c44fc77816" containerName="cleanup" Oct 30 14:30:42.318992 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.318912 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d6da5d4-7fd8-4b0e-9a82-40c44fc77816" containerName="cleanup" Oct 30 14:30:42.318992 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.318920 4982 cpu_manager.go:414] "containerMap: removing container" podUID="5c9a6cee-328e-461d-8e42-69c7bdf0afd6" containerName="cleanup" Oct 30 14:30:42.318992 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.318923 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c9a6cee-328e-461d-8e42-69c7bdf0afd6" containerName="cleanup" Oct 30 14:30:42.318992 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.318930 4982 cpu_manager.go:414] "containerMap: removing container" podUID="b60b5431-525a-4363-a3fb-3352fe6521e9" containerName="cleanup" Oct 30 14:30:42.318992 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.318934 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b60b5431-525a-4363-a3fb-3352fe6521e9" containerName="cleanup" Oct 30 14:30:42.318992 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.318942 4982 cpu_manager.go:414] "containerMap: removing container" podUID="fc3717af-2029-422a-9230-b41ebc4dedc5" containerName="cleanup" Oct 30 14:30:42.318992 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.318946 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc3717af-2029-422a-9230-b41ebc4dedc5" containerName="cleanup" Oct 30 14:30:42.318992 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.318959 4982 cpu_manager.go:414] "containerMap: removing container" podUID="b60b5431-525a-4363-a3fb-3352fe6521e9" containerName="cleanup" Oct 30 14:30:42.318992 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.318963 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b60b5431-525a-4363-a3fb-3352fe6521e9" containerName="cleanup" Oct 30 14:30:42.318992 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.318970 4982 cpu_manager.go:414] "containerMap: removing container" podUID="6f0959c9-4c3d-4394-baa1-6fd5a31fbe78" containerName="cleanup" Oct 30 14:30:42.318992 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.318974 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f0959c9-4c3d-4394-baa1-6fd5a31fbe78" containerName="cleanup" Oct 30 14:30:42.318992 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.318981 4982 cpu_manager.go:414] "containerMap: removing container" podUID="e7260a80-a857-497e-bb12-28d8b7fa8bab" containerName="cleanup" Oct 30 14:30:42.318992 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.318986 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7260a80-a857-497e-bb12-28d8b7fa8bab" containerName="cleanup" Oct 30 14:30:42.318992 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.318992 4982 cpu_manager.go:414] "containerMap: removing container" podUID="bd736b21-6929-413d-8eb4-a9f891ffe64a" containerName="cleanup" Oct 30 14:30:42.318992 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.318997 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd736b21-6929-413d-8eb4-a9f891ffe64a" containerName="cleanup" Oct 30 14:30:42.318992 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.319005 4982 cpu_manager.go:414] "containerMap: removing container" podUID="e7260a80-a857-497e-bb12-28d8b7fa8bab" containerName="cleanup" Oct 30 14:30:42.318992 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.319010 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7260a80-a857-497e-bb12-28d8b7fa8bab" containerName="cleanup" Oct 30 14:30:42.320146 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.319018 4982 cpu_manager.go:414] "containerMap: removing container" podUID="b60b5431-525a-4363-a3fb-3352fe6521e9" containerName="cleanup" Oct 30 14:30:42.320146 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.319023 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b60b5431-525a-4363-a3fb-3352fe6521e9" containerName="cleanup" Oct 30 14:30:42.320146 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.319028 4982 cpu_manager.go:414] "containerMap: removing container" podUID="5c9a6cee-328e-461d-8e42-69c7bdf0afd6" containerName="cleanup" Oct 30 14:30:42.320146 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.319032 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c9a6cee-328e-461d-8e42-69c7bdf0afd6" containerName="cleanup" Oct 30 14:30:42.320146 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.319047 4982 cpu_manager.go:414] "containerMap: removing container" podUID="2f3320ba-10c9-4018-a33b-479d8b18d02e" containerName="revoke-keys" Oct 30 14:30:42.320146 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.319051 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f3320ba-10c9-4018-a33b-479d8b18d02e" containerName="revoke-keys" Oct 30 14:30:42.320146 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.319139 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="9246e72d-bf25-49e9-a530-78843f462b48" containerName="cleanup" Oct 30 14:30:42.320146 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.319147 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c9a6cee-328e-461d-8e42-69c7bdf0afd6" containerName="cleanup" Oct 30 14:30:42.320146 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.319152 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f3320ba-10c9-4018-a33b-479d8b18d02e" containerName="revoke-keys" Oct 30 14:30:42.320146 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.319159 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d6da5d4-7fd8-4b0e-9a82-40c44fc77816" containerName="cleanup" Oct 30 14:30:42.320146 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.319165 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ecbf4bb-822a-4bd6-891c-c1eb6d12ecaa" containerName="cleanup" Oct 30 14:30:42.320146 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.319171 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f0959c9-4c3d-4394-baa1-6fd5a31fbe78" containerName="cleanup" Oct 30 14:30:42.320146 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.319176 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1dda7b0-fb53-4fec-96ad-b8e04eb8b24b" containerName="cleanup" Oct 30 14:30:42.320146 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.319182 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="f300b507-503e-47e1-a09b-d907b975c2c2" containerName="cleanup" Oct 30 14:30:42.320146 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.319187 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd736b21-6929-413d-8eb4-a9f891ffe64a" containerName="cleanup" Oct 30 14:30:42.320146 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.319192 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1dda7b0-fb53-4fec-96ad-b8e04eb8b24b" containerName="cleanup" Oct 30 14:30:42.320146 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.319198 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc3717af-2029-422a-9230-b41ebc4dedc5" containerName="cleanup" Oct 30 14:30:42.320146 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.319202 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="f300b507-503e-47e1-a09b-d907b975c2c2" containerName="cleanup" Oct 30 14:30:42.320146 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.319207 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f0959c9-4c3d-4394-baa1-6fd5a31fbe78" containerName="cleanup" Oct 30 14:30:42.320146 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.319211 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ecbf4bb-822a-4bd6-891c-c1eb6d12ecaa" containerName="cleanup" Oct 30 14:30:42.320146 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.319217 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d6da5d4-7fd8-4b0e-9a82-40c44fc77816" containerName="cleanup" Oct 30 14:30:42.320146 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.319222 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="9246e72d-bf25-49e9-a530-78843f462b48" containerName="cleanup" Oct 30 14:30:42.320146 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.319227 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd736b21-6929-413d-8eb4-a9f891ffe64a" containerName="cleanup" Oct 30 14:30:42.320146 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.319231 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7260a80-a857-497e-bb12-28d8b7fa8bab" containerName="cleanup" Oct 30 14:30:42.320146 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.319236 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ecbf4bb-822a-4bd6-891c-c1eb6d12ecaa" containerName="cleanup" Oct 30 14:30:42.320146 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.319241 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="9246e72d-bf25-49e9-a530-78843f462b48" containerName="cleanup" Oct 30 14:30:42.320146 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.319246 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c9a6cee-328e-461d-8e42-69c7bdf0afd6" containerName="cleanup" Oct 30 14:30:42.320146 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.319251 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="b60b5431-525a-4363-a3fb-3352fe6521e9" containerName="cleanup" Oct 30 14:30:42.320146 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.319257 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="b60b5431-525a-4363-a3fb-3352fe6521e9" containerName="cleanup" Oct 30 14:30:42.320146 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.319264 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7260a80-a857-497e-bb12-28d8b7fa8bab" containerName="cleanup" Oct 30 14:30:42.320146 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.319270 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="b60b5431-525a-4363-a3fb-3352fe6521e9" containerName="cleanup" Oct 30 14:30:42.320146 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.319274 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc3717af-2029-422a-9230-b41ebc4dedc5" containerName="cleanup" Oct 30 14:30:42.324558 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.324516 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d15602-847fx" Oct 30 14:30:42.344802 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.344773 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d15602-847fx"] Oct 30 14:30:42.352335 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.352313 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:30:42.352479 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.352373 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:30:42.403295 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.403259 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdl29\" (UniqueName: \"kubernetes.io/projected/2a2022f7-1c6c-4e2d-bfc8-11f02ce57c19-kube-api-access-fdl29\") pod \"maas-api-revoke-keys-e2e-inf-a-d15602-847fx\" (UID: \"2a2022f7-1c6c-4e2d-bfc8-11f02ce57c19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d15602-847fx" Oct 30 14:30:42.403483 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.403367 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a2022f7-1c6c-4e2d-bfc8-11f02ce57c19-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d15602-847fx\" (UID: \"2a2022f7-1c6c-4e2d-bfc8-11f02ce57c19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d15602-847fx" Oct 30 14:30:42.403483 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.403426 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a2022f7-1c6c-4e2d-bfc8-11f02ce57c19-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d15602-847fx\" (UID: \"2a2022f7-1c6c-4e2d-bfc8-11f02ce57c19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d15602-847fx" Oct 30 14:30:42.505276 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.505241 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdl29\" (UniqueName: \"kubernetes.io/projected/2a2022f7-1c6c-4e2d-bfc8-11f02ce57c19-kube-api-access-fdl29\") pod \"maas-api-revoke-keys-e2e-inf-a-d15602-847fx\" (UID: \"2a2022f7-1c6c-4e2d-bfc8-11f02ce57c19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d15602-847fx" Oct 30 14:30:42.505468 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.505439 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a2022f7-1c6c-4e2d-bfc8-11f02ce57c19-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d15602-847fx\" (UID: \"2a2022f7-1c6c-4e2d-bfc8-11f02ce57c19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d15602-847fx" Oct 30 14:30:42.505547 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.505532 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a2022f7-1c6c-4e2d-bfc8-11f02ce57c19-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d15602-847fx\" (UID: \"2a2022f7-1c6c-4e2d-bfc8-11f02ce57c19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d15602-847fx" Oct 30 14:30:42.506311 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.506286 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a2022f7-1c6c-4e2d-bfc8-11f02ce57c19-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d15602-847fx\" (UID: \"2a2022f7-1c6c-4e2d-bfc8-11f02ce57c19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d15602-847fx" Oct 30 14:30:42.508067 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.508041 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a2022f7-1c6c-4e2d-bfc8-11f02ce57c19-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d15602-847fx\" (UID: \"2a2022f7-1c6c-4e2d-bfc8-11f02ce57c19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d15602-847fx" Oct 30 14:30:42.531162 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.531136 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdl29\" (UniqueName: \"kubernetes.io/projected/2a2022f7-1c6c-4e2d-bfc8-11f02ce57c19-kube-api-access-fdl29\") pod \"maas-api-revoke-keys-e2e-inf-a-d15602-847fx\" (UID: \"2a2022f7-1c6c-4e2d-bfc8-11f02ce57c19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d15602-847fx" Oct 30 14:30:42.641172 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.641093 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d15602-847fx" Oct 30 14:30:42.701664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.697106 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:30:42.998953 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:42.998916 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d15602-847fx"] Oct 30 14:30:43.032133 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:30:43.031785 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a2022f7_1c6c_4e2d_bfc8_11f02ce57c19.slice/crio-d389eeaa0d4c728f8062340be33c8d5b544cda56d0e8e6effc9badcc4903d5d8 WatchSource:0}: Error finding container d389eeaa0d4c728f8062340be33c8d5b544cda56d0e8e6effc9badcc4903d5d8: Status 404 returned error can't find the container with id d389eeaa0d4c728f8062340be33c8d5b544cda56d0e8e6effc9badcc4903d5d8 Oct 30 14:30:43.044654 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:43.044614 4982 generic.go:356] "Generic (PLEG): container finished" podID="cdf6bd0e-f6ff-49b9-8f1a-b90052fe7787" containerID="480cf0ec02d620b0a9ad9188b060c296e9bd12ee799147bb2fc1f907ba55c234" exitCode=0 Oct 30 14:30:43.049341 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:43.049313 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a9542c-69bcb5578b-rzp2q" Oct 30 14:30:43.050738 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:43.044771 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a9542c-794b65877f-sp225" event={"ID":"cdf6bd0e-f6ff-49b9-8f1a-b90052fe7787","Type":"ContainerDied","Data":"480cf0ec02d620b0a9ad9188b060c296e9bd12ee799147bb2fc1f907ba55c234"} Oct 30 14:30:43.056626 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:43.056605 4982 generic.go:356] "Generic (PLEG): container finished" podID="73f0efb5-d969-4b91-8558-75d5ebe12a43" containerID="2e9387560467b28eae38822c7ae72f2cfa3e72cde643a9697a62fd0751d45910" exitCode=0 Oct 30 14:30:43.056786 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:43.056710 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a9542c-69bcb5578b-rzp2q" event={"ID":"73f0efb5-d969-4b91-8558-75d5ebe12a43","Type":"ContainerDied","Data":"2e9387560467b28eae38822c7ae72f2cfa3e72cde643a9697a62fd0751d45910"} Oct 30 14:30:43.056850 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:43.056785 4982 scope.go:122] "RemoveContainer" containerID="2e9387560467b28eae38822c7ae72f2cfa3e72cde643a9697a62fd0751d45910" Oct 30 14:30:43.094212 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:43.094185 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a9542c-794b65877f-sp225" Oct 30 14:30:43.235960 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:43.235860 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdf6bd0e-f6ff-49b9-8f1a-b90052fe7787-kube-api-access-wt2mc\" (UniqueName: \"kubernetes.io/projected/cdf6bd0e-f6ff-49b9-8f1a-b90052fe7787-kube-api-access-wt2mc\") pod \"cdf6bd0e-f6ff-49b9-8f1a-b90052fe7787\" (UID: \"cdf6bd0e-f6ff-49b9-8f1a-b90052fe7787\") " Oct 30 14:30:43.235960 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:43.235928 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73f0efb5-d969-4b91-8558-75d5ebe12a43-kube-api-access-mhkjc\" (UniqueName: \"kubernetes.io/projected/73f0efb5-d969-4b91-8558-75d5ebe12a43-kube-api-access-mhkjc\") pod \"73f0efb5-d969-4b91-8558-75d5ebe12a43\" (UID: \"73f0efb5-d969-4b91-8558-75d5ebe12a43\") " Oct 30 14:30:43.236192 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:43.235999 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdf6bd0e-f6ff-49b9-8f1a-b90052fe7787-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdf6bd0e-f6ff-49b9-8f1a-b90052fe7787-plugins-config-volume\") pod \"cdf6bd0e-f6ff-49b9-8f1a-b90052fe7787\" (UID: \"cdf6bd0e-f6ff-49b9-8f1a-b90052fe7787\") " Oct 30 14:30:43.236192 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:43.236024 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73f0efb5-d969-4b91-8558-75d5ebe12a43-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73f0efb5-d969-4b91-8558-75d5ebe12a43-plugins-config-volume\") pod \"73f0efb5-d969-4b91-8558-75d5ebe12a43\" (UID: \"73f0efb5-d969-4b91-8558-75d5ebe12a43\") " Oct 30 14:30:43.236418 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:43.236385 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdf6bd0e-f6ff-49b9-8f1a-b90052fe7787-plugins-config-volume" pod "cdf6bd0e-f6ff-49b9-8f1a-b90052fe7787" (UID: "cdf6bd0e-f6ff-49b9-8f1a-b90052fe7787"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:30:43.236559 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:43.236482 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73f0efb5-d969-4b91-8558-75d5ebe12a43-plugins-config-volume" pod "73f0efb5-d969-4b91-8558-75d5ebe12a43" (UID: "73f0efb5-d969-4b91-8558-75d5ebe12a43"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:30:43.236820 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:43.236801 4982 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdf6bd0e-f6ff-49b9-8f1a-b90052fe7787-plugins-config-volume\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:30:43.236907 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:43.236821 4982 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73f0efb5-d969-4b91-8558-75d5ebe12a43-plugins-config-volume\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:30:43.280428 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:43.280387 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdf6bd0e-f6ff-49b9-8f1a-b90052fe7787-kube-api-access-wt2mc" pod "cdf6bd0e-f6ff-49b9-8f1a-b90052fe7787" (UID: "cdf6bd0e-f6ff-49b9-8f1a-b90052fe7787"). InnerVolumeSpecName "kube-api-access-wt2mc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:30:43.280578 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:43.280453 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73f0efb5-d969-4b91-8558-75d5ebe12a43-kube-api-access-mhkjc" pod "73f0efb5-d969-4b91-8558-75d5ebe12a43" (UID: "73f0efb5-d969-4b91-8558-75d5ebe12a43"). InnerVolumeSpecName "kube-api-access-mhkjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:30:43.338209 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:43.338172 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wt2mc\" (UniqueName: \"kubernetes.io/projected/cdf6bd0e-f6ff-49b9-8f1a-b90052fe7787-kube-api-access-wt2mc\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:30:43.338209 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:43.338202 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhkjc\" (UniqueName: \"kubernetes.io/projected/73f0efb5-d969-4b91-8558-75d5ebe12a43-kube-api-access-mhkjc\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:30:44.066360 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:44.066326 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a9542c-794b65877f-sp225" Oct 30 14:30:44.066548 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:44.066326 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a9542c-794b65877f-sp225" event={"ID":"cdf6bd0e-f6ff-49b9-8f1a-b90052fe7787","Type":"ContainerDied","Data":"c88c89395d1070360857e8bb252eabb7cfb333cf5074ed6142548dbbb0eed53a"} Oct 30 14:30:44.066548 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:44.066481 4982 scope.go:122] "RemoveContainer" containerID="480cf0ec02d620b0a9ad9188b060c296e9bd12ee799147bb2fc1f907ba55c234" Oct 30 14:30:44.068103 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:44.068070 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d15602-847fx" event={"ID":"2a2022f7-1c6c-4e2d-bfc8-11f02ce57c19","Type":"ContainerStarted","Data":"3cda605315726171249c02fb3573e294d7fed2d6afc8a03e58b8bc94bfbce9e1"} Oct 30 14:30:44.068238 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:44.068124 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d15602-847fx" event={"ID":"2a2022f7-1c6c-4e2d-bfc8-11f02ce57c19","Type":"ContainerStarted","Data":"d389eeaa0d4c728f8062340be33c8d5b544cda56d0e8e6effc9badcc4903d5d8"} Oct 30 14:30:44.069235 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:44.069211 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a9542c-69bcb5578b-rzp2q" event={"ID":"73f0efb5-d969-4b91-8558-75d5ebe12a43","Type":"ContainerDied","Data":"729d426411d174bde80748c9fa96f95b6fa79fc5e6c5c04e4eeb39d253fb352b"} Oct 30 14:30:44.069358 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:44.069216 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a9542c-69bcb5578b-rzp2q" Oct 30 14:30:44.135270 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:44.135221 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d15602-847fx" podStartSLOduration=2.13520692 podStartE2EDuration="2.13520692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:30:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:30:44.132729673 +0000 UTC m=+2513.876863714" watchObservedRunningTime="2026-10-30 14:30:44.13520692 +0000 UTC m=+2513.879340956" Oct 30 14:30:44.177359 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:44.177328 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a9542c-794b65877f-sp225"] Oct 30 14:30:44.183191 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:44.183160 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a9542c-794b65877f-sp225"] Oct 30 14:30:44.221164 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:44.221131 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a9542c-69bcb5578b-rzp2q"] Oct 30 14:30:44.227155 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:44.227129 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a9542c-69bcb5578b-rzp2q"] Oct 30 14:30:44.404075 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:44.403989 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73f0efb5-d969-4b91-8558-75d5ebe12a43" path="/var/lib/kubelet/pods/73f0efb5-d969-4b91-8558-75d5ebe12a43/volumes" Oct 30 14:30:44.404632 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:44.404607 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdf6bd0e-f6ff-49b9-8f1a-b90052fe7787" path="/var/lib/kubelet/pods/cdf6bd0e-f6ff-49b9-8f1a-b90052fe7787/volumes" Oct 30 14:30:44.566095 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:44.566060 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:30:44.715295 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:44.715157 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:30:45.080927 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:45.080888 4982 generic.go:356] "Generic (PLEG): container finished" podID="2a2022f7-1c6c-4e2d-bfc8-11f02ce57c19" containerID="3cda605315726171249c02fb3573e294d7fed2d6afc8a03e58b8bc94bfbce9e1" exitCode=0 Oct 30 14:30:45.081167 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:45.081144 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d15602-847fx" event={"ID":"2a2022f7-1c6c-4e2d-bfc8-11f02ce57c19","Type":"ContainerDied","Data":"3cda605315726171249c02fb3573e294d7fed2d6afc8a03e58b8bc94bfbce9e1"} Oct 30 14:30:45.311751 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:45.311689 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:30:46.060499 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:46.060461 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjqps"] Oct 30 14:30:46.060993 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:46.060705 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjqps" podUID="9c00c972-1e46-4182-aaf4-163a6960fb70" containerName="discovery" containerID="cri-o://5f7b659e00513b90380612c899dfa9f10e29ddeb3d97ec4166160c8f991158b6" gracePeriod=30 Oct 30 14:30:46.317394 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:46.316153 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d15602-847fx" Oct 30 14:30:46.387230 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:46.387189 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a2022f7-1c6c-4e2d-bfc8-11f02ce57c19-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a2022f7-1c6c-4e2d-bfc8-11f02ce57c19-maas-api-openshift-ca\") pod \"2a2022f7-1c6c-4e2d-bfc8-11f02ce57c19\" (UID: \"2a2022f7-1c6c-4e2d-bfc8-11f02ce57c19\") " Oct 30 14:30:46.387666 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:46.387561 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a2022f7-1c6c-4e2d-bfc8-11f02ce57c19-maas-api-openshift-ca" pod "2a2022f7-1c6c-4e2d-bfc8-11f02ce57c19" (UID: "2a2022f7-1c6c-4e2d-bfc8-11f02ce57c19"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:30:46.387787 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:46.387769 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a2022f7-1c6c-4e2d-bfc8-11f02ce57c19-kube-api-access-fdl29\" (UniqueName: \"kubernetes.io/projected/2a2022f7-1c6c-4e2d-bfc8-11f02ce57c19-kube-api-access-fdl29\") pod \"2a2022f7-1c6c-4e2d-bfc8-11f02ce57c19\" (UID: \"2a2022f7-1c6c-4e2d-bfc8-11f02ce57c19\") " Oct 30 14:30:46.387868 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:46.387851 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a2022f7-1c6c-4e2d-bfc8-11f02ce57c19-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a2022f7-1c6c-4e2d-bfc8-11f02ce57c19-maas-api-xks-ca\") pod \"2a2022f7-1c6c-4e2d-bfc8-11f02ce57c19\" (UID: \"2a2022f7-1c6c-4e2d-bfc8-11f02ce57c19\") " Oct 30 14:30:46.388564 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:46.388533 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a2022f7-1c6c-4e2d-bfc8-11f02ce57c19-maas-api-openshift-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:30:46.391203 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:46.391172 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a2022f7-1c6c-4e2d-bfc8-11f02ce57c19-maas-api-xks-ca" pod "2a2022f7-1c6c-4e2d-bfc8-11f02ce57c19" (UID: "2a2022f7-1c6c-4e2d-bfc8-11f02ce57c19"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:30:46.408758 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:46.408732 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a2022f7-1c6c-4e2d-bfc8-11f02ce57c19-kube-api-access-fdl29" pod "2a2022f7-1c6c-4e2d-bfc8-11f02ce57c19" (UID: "2a2022f7-1c6c-4e2d-bfc8-11f02ce57c19"). InnerVolumeSpecName "kube-api-access-fdl29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:30:46.439198 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:46.439176 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjqps" Oct 30 14:30:46.489822 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:46.489781 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c00c972-1e46-4182-aaf4-163a6960fb70-cacerts\" (UniqueName: \"kubernetes.io/secret/9c00c972-1e46-4182-aaf4-163a6960fb70-cacerts\") pod \"9c00c972-1e46-4182-aaf4-163a6960fb70\" (UID: \"9c00c972-1e46-4182-aaf4-163a6960fb70\") " Oct 30 14:30:46.490038 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:46.489906 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c00c972-1e46-4182-aaf4-163a6960fb70-kube-api-access-5zz5l\" (UniqueName: \"kubernetes.io/projected/9c00c972-1e46-4182-aaf4-163a6960fb70-kube-api-access-5zz5l\") pod \"9c00c972-1e46-4182-aaf4-163a6960fb70\" (UID: \"9c00c972-1e46-4182-aaf4-163a6960fb70\") " Oct 30 14:30:46.490038 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:46.489964 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c00c972-1e46-4182-aaf4-163a6960fb70-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9c00c972-1e46-4182-aaf4-163a6960fb70-istio-kubeconfig\") pod \"9c00c972-1e46-4182-aaf4-163a6960fb70\" (UID: \"9c00c972-1e46-4182-aaf4-163a6960fb70\") " Oct 30 14:30:46.490227 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:46.490127 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c00c972-1e46-4182-aaf4-163a6960fb70-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9c00c972-1e46-4182-aaf4-163a6960fb70-istio-csr-dns-cert\") pod \"9c00c972-1e46-4182-aaf4-163a6960fb70\" (UID: \"9c00c972-1e46-4182-aaf4-163a6960fb70\") " Oct 30 14:30:46.490227 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:46.490194 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c00c972-1e46-4182-aaf4-163a6960fb70-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9c00c972-1e46-4182-aaf4-163a6960fb70-istio-csr-ca-configmap\") pod \"9c00c972-1e46-4182-aaf4-163a6960fb70\" (UID: \"9c00c972-1e46-4182-aaf4-163a6960fb70\") " Oct 30 14:30:46.490394 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:46.490257 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c00c972-1e46-4182-aaf4-163a6960fb70-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c00c972-1e46-4182-aaf4-163a6960fb70-local-certs\") pod \"9c00c972-1e46-4182-aaf4-163a6960fb70\" (UID: \"9c00c972-1e46-4182-aaf4-163a6960fb70\") " Oct 30 14:30:46.490394 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:46.490291 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c00c972-1e46-4182-aaf4-163a6960fb70-istio-token\" (UniqueName: \"kubernetes.io/projected/9c00c972-1e46-4182-aaf4-163a6960fb70-istio-token\") pod \"9c00c972-1e46-4182-aaf4-163a6960fb70\" (UID: \"9c00c972-1e46-4182-aaf4-163a6960fb70\") " Oct 30 14:30:46.491667 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:46.491079 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdl29\" (UniqueName: \"kubernetes.io/projected/2a2022f7-1c6c-4e2d-bfc8-11f02ce57c19-kube-api-access-fdl29\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:30:46.491667 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:46.491102 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a2022f7-1c6c-4e2d-bfc8-11f02ce57c19-maas-api-xks-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:30:46.498567 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:46.498229 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c00c972-1e46-4182-aaf4-163a6960fb70-istio-token" pod "9c00c972-1e46-4182-aaf4-163a6960fb70" (UID: "9c00c972-1e46-4182-aaf4-163a6960fb70"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:30:46.498567 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:46.498533 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c00c972-1e46-4182-aaf4-163a6960fb70-istio-csr-ca-configmap" pod "9c00c972-1e46-4182-aaf4-163a6960fb70" (UID: "9c00c972-1e46-4182-aaf4-163a6960fb70"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:30:46.500559 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:46.500532 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c00c972-1e46-4182-aaf4-163a6960fb70-istio-kubeconfig" pod "9c00c972-1e46-4182-aaf4-163a6960fb70" (UID: "9c00c972-1e46-4182-aaf4-163a6960fb70"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:30:46.500994 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:46.500948 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c00c972-1e46-4182-aaf4-163a6960fb70-istio-csr-dns-cert" pod "9c00c972-1e46-4182-aaf4-163a6960fb70" (UID: "9c00c972-1e46-4182-aaf4-163a6960fb70"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:30:46.505101 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:46.505080 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c00c972-1e46-4182-aaf4-163a6960fb70-cacerts" pod "9c00c972-1e46-4182-aaf4-163a6960fb70" (UID: "9c00c972-1e46-4182-aaf4-163a6960fb70"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:30:46.505405 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:46.505105 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c00c972-1e46-4182-aaf4-163a6960fb70-kube-api-access-5zz5l" pod "9c00c972-1e46-4182-aaf4-163a6960fb70" (UID: "9c00c972-1e46-4182-aaf4-163a6960fb70"). InnerVolumeSpecName "kube-api-access-5zz5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:30:46.507309 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:46.507272 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c00c972-1e46-4182-aaf4-163a6960fb70-local-certs" pod "9c00c972-1e46-4182-aaf4-163a6960fb70" (UID: "9c00c972-1e46-4182-aaf4-163a6960fb70"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:30:46.550864 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:46.550818 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:30:46.592742 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:46.592661 4982 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9c00c972-1e46-4182-aaf4-163a6960fb70-cacerts\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:30:46.592742 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:46.592687 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zz5l\" (UniqueName: \"kubernetes.io/projected/9c00c972-1e46-4182-aaf4-163a6960fb70-kube-api-access-5zz5l\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:30:46.592742 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:46.592704 4982 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9c00c972-1e46-4182-aaf4-163a6960fb70-istio-kubeconfig\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:30:46.592742 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:46.592718 4982 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9c00c972-1e46-4182-aaf4-163a6960fb70-istio-csr-dns-cert\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:30:46.592742 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:46.592732 4982 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9c00c972-1e46-4182-aaf4-163a6960fb70-istio-csr-ca-configmap\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:30:46.593002 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:46.592744 4982 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c00c972-1e46-4182-aaf4-163a6960fb70-local-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:30:46.593002 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:46.592757 4982 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c00c972-1e46-4182-aaf4-163a6960fb70-istio-token\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:30:47.036206 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:47.036123 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d46a90e-mr46c"] Oct 30 14:30:47.039449 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:47.039419 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d46a90e-mr46c"] Oct 30 14:30:47.103579 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:47.103547 4982 generic.go:356] "Generic (PLEG): container finished" podID="9c00c972-1e46-4182-aaf4-163a6960fb70" containerID="5f7b659e00513b90380612c899dfa9f10e29ddeb3d97ec4166160c8f991158b6" exitCode=0 Oct 30 14:30:47.103997 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:47.103613 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjqps" Oct 30 14:30:47.103997 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:47.103625 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjqps" event={"ID":"9c00c972-1e46-4182-aaf4-163a6960fb70","Type":"ContainerDied","Data":"5f7b659e00513b90380612c899dfa9f10e29ddeb3d97ec4166160c8f991158b6"} Oct 30 14:30:47.103997 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:47.103749 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjqps" event={"ID":"9c00c972-1e46-4182-aaf4-163a6960fb70","Type":"ContainerDied","Data":"bfefdf0ff84a3d647c7ad4414dd5a7b4ee161672e175f4470fc491e264375ca9"} Oct 30 14:30:47.103997 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:47.103782 4982 scope.go:122] "RemoveContainer" containerID="5f7b659e00513b90380612c899dfa9f10e29ddeb3d97ec4166160c8f991158b6" Oct 30 14:30:47.106274 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:47.106253 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d15602-847fx" event={"ID":"2a2022f7-1c6c-4e2d-bfc8-11f02ce57c19","Type":"ContainerDied","Data":"d389eeaa0d4c728f8062340be33c8d5b544cda56d0e8e6effc9badcc4903d5d8"} Oct 30 14:30:47.106421 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:47.106404 4982 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d389eeaa0d4c728f8062340be33c8d5b544cda56d0e8e6effc9badcc4903d5d8" Oct 30 14:30:47.106514 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:47.106468 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d15602-847fx" Oct 30 14:30:47.116476 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:47.116451 4982 scope.go:122] "RemoveContainer" containerID="5f7b659e00513b90380612c899dfa9f10e29ddeb3d97ec4166160c8f991158b6" Oct 30 14:30:47.116917 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:30:47.116887 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f7b659e00513b90380612c899dfa9f10e29ddeb3d97ec4166160c8f991158b6\": container with ID starting with 5f7b659e00513b90380612c899dfa9f10e29ddeb3d97ec4166160c8f991158b6 not found: ID does not exist" containerID="5f7b659e00513b90380612c899dfa9f10e29ddeb3d97ec4166160c8f991158b6" Oct 30 14:30:47.117035 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:47.116926 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f7b659e00513b90380612c899dfa9f10e29ddeb3d97ec4166160c8f991158b6"} err="failed to get container status \"5f7b659e00513b90380612c899dfa9f10e29ddeb3d97ec4166160c8f991158b6\": rpc error: code = NotFound desc = could not find container \"5f7b659e00513b90380612c899dfa9f10e29ddeb3d97ec4166160c8f991158b6\": container with ID starting with 5f7b659e00513b90380612c899dfa9f10e29ddeb3d97ec4166160c8f991158b6 not found: ID does not exist" Oct 30 14:30:47.176362 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:47.176323 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjqps"] Oct 30 14:30:47.189485 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:47.189459 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjqps"] Oct 30 14:30:48.401787 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:48.401752 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cceaa42-4daf-4c92-b4d8-1a0bdefc49e8" path="/var/lib/kubelet/pods/6cceaa42-4daf-4c92-b4d8-1a0bdefc49e8/volumes" Oct 30 14:30:48.402536 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:48.402513 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c00c972-1e46-4182-aaf4-163a6960fb70" path="/var/lib/kubelet/pods/9c00c972-1e46-4182-aaf4-163a6960fb70/volumes" Oct 30 14:30:48.498806 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:48.498742 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:30:50.102612 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:50.102545 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:30:50.640950 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:50.640901 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:30:50.789737 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:50.789690 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:30:50.941908 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:50.941822 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:30:51.306505 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:51.306371 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:30:51.338478 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:51.338454 4982 scope.go:122] "RemoveContainer" containerID="5f89b70cf548c575dd647c7763ab5fcc62977089edeef38dfe057dc7acf230bc" Oct 30 14:30:52.942748 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:52.942715 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:30:53.096959 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:53.096924 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:30:54.781868 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:54.781829 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:30:55.058154 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:55.058124 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cab7d423-bf5jd"] Oct 30 14:30:55.058807 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:55.058791 4982 cpu_manager.go:414] "containerMap: removing container" podUID="9c00c972-1e46-4182-aaf4-163a6960fb70" containerName="discovery" Oct 30 14:30:55.058853 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:55.058809 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c00c972-1e46-4182-aaf4-163a6960fb70" containerName="discovery" Oct 30 14:30:55.058853 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:55.058821 4982 cpu_manager.go:414] "containerMap: removing container" podUID="5c9a6cee-328e-461d-8e42-69c7bdf0afd6" containerName="cleanup" Oct 30 14:30:55.058853 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:55.058828 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c9a6cee-328e-461d-8e42-69c7bdf0afd6" containerName="cleanup" Oct 30 14:30:55.058853 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:55.058852 4982 cpu_manager.go:414] "containerMap: removing container" podUID="2a2022f7-1c6c-4e2d-bfc8-11f02ce57c19" containerName="revoke-keys" Oct 30 14:30:55.058990 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:55.058858 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a2022f7-1c6c-4e2d-bfc8-11f02ce57c19" containerName="revoke-keys" Oct 30 14:30:55.058990 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:55.058872 4982 cpu_manager.go:414] "containerMap: removing container" podUID="bd736b21-6929-413d-8eb4-a9f891ffe64a" containerName="cleanup" Oct 30 14:30:55.058990 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:55.058877 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd736b21-6929-413d-8eb4-a9f891ffe64a" containerName="cleanup" Oct 30 14:30:55.058990 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:55.058884 4982 cpu_manager.go:414] "containerMap: removing container" podUID="b1dda7b0-fb53-4fec-96ad-b8e04eb8b24b" containerName="cleanup" Oct 30 14:30:55.058990 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:55.058889 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1dda7b0-fb53-4fec-96ad-b8e04eb8b24b" containerName="cleanup" Oct 30 14:30:55.058990 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:55.058904 4982 cpu_manager.go:414] "containerMap: removing container" podUID="6ecbf4bb-822a-4bd6-891c-c1eb6d12ecaa" containerName="cleanup" Oct 30 14:30:55.058990 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:55.058911 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ecbf4bb-822a-4bd6-891c-c1eb6d12ecaa" containerName="cleanup" Oct 30 14:30:55.058990 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:55.058921 4982 cpu_manager.go:414] "containerMap: removing container" podUID="9246e72d-bf25-49e9-a530-78843f462b48" containerName="cleanup" Oct 30 14:30:55.058990 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:55.058926 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9246e72d-bf25-49e9-a530-78843f462b48" containerName="cleanup" Oct 30 14:30:55.058990 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:55.058935 4982 cpu_manager.go:414] "containerMap: removing container" podUID="7d6da5d4-7fd8-4b0e-9a82-40c44fc77816" containerName="cleanup" Oct 30 14:30:55.058990 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:55.058939 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d6da5d4-7fd8-4b0e-9a82-40c44fc77816" containerName="cleanup" Oct 30 14:30:55.058990 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:55.058950 4982 cpu_manager.go:414] "containerMap: removing container" podUID="fc3717af-2029-422a-9230-b41ebc4dedc5" containerName="cleanup" Oct 30 14:30:55.058990 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:55.058954 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc3717af-2029-422a-9230-b41ebc4dedc5" containerName="cleanup" Oct 30 14:30:55.058990 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:55.058968 4982 cpu_manager.go:414] "containerMap: removing container" podUID="cdf6bd0e-f6ff-49b9-8f1a-b90052fe7787" containerName="payload-pre-processing" Oct 30 14:30:55.058990 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:55.058973 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdf6bd0e-f6ff-49b9-8f1a-b90052fe7787" containerName="payload-pre-processing" Oct 30 14:30:55.058990 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:55.058982 4982 cpu_manager.go:414] "containerMap: removing container" podUID="73f0efb5-d969-4b91-8558-75d5ebe12a43" containerName="payload-processing" Oct 30 14:30:55.058990 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:55.058986 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73f0efb5-d969-4b91-8558-75d5ebe12a43" containerName="payload-processing" Oct 30 14:30:55.059476 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:55.059083 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a2022f7-1c6c-4e2d-bfc8-11f02ce57c19" containerName="revoke-keys" Oct 30 14:30:55.059476 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:55.059095 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd736b21-6929-413d-8eb4-a9f891ffe64a" containerName="cleanup" Oct 30 14:30:55.059476 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:55.059101 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1dda7b0-fb53-4fec-96ad-b8e04eb8b24b" containerName="cleanup" Oct 30 14:30:55.059476 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:55.059108 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="f300b507-503e-47e1-a09b-d907b975c2c2" containerName="cleanup" Oct 30 14:30:55.059476 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:55.059113 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f0959c9-4c3d-4394-baa1-6fd5a31fbe78" containerName="cleanup" Oct 30 14:30:55.059476 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:55.059121 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d6da5d4-7fd8-4b0e-9a82-40c44fc77816" containerName="cleanup" Oct 30 14:30:55.059476 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:55.059127 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7260a80-a857-497e-bb12-28d8b7fa8bab" containerName="cleanup" Oct 30 14:30:55.059476 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:55.059133 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c9a6cee-328e-461d-8e42-69c7bdf0afd6" containerName="cleanup" Oct 30 14:30:55.059476 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:55.059142 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdf6bd0e-f6ff-49b9-8f1a-b90052fe7787" containerName="payload-pre-processing" Oct 30 14:30:55.059476 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:55.059150 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="73f0efb5-d969-4b91-8558-75d5ebe12a43" containerName="payload-processing" Oct 30 14:30:55.059476 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:55.059162 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c00c972-1e46-4182-aaf4-163a6960fb70" containerName="discovery" Oct 30 14:30:55.059476 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:55.059168 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc3717af-2029-422a-9230-b41ebc4dedc5" containerName="cleanup" Oct 30 14:30:55.064440 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:55.064396 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab7d423-bf5jd" Oct 30 14:30:55.098601 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:55.098565 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cab7d423-bf5jd"] Oct 30 14:30:55.104317 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:55.104286 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:30:55.104476 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:55.104428 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:30:55.177290 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:55.177252 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0783d93b-e60a-4769-b757-430aa9bf97a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cab7d423-bf5jd\" (UID: \"0783d93b-e60a-4769-b757-430aa9bf97a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab7d423-bf5jd" Oct 30 14:30:55.177505 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:55.177325 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0783d93b-e60a-4769-b757-430aa9bf97a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cab7d423-bf5jd\" (UID: \"0783d93b-e60a-4769-b757-430aa9bf97a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab7d423-bf5jd" Oct 30 14:30:55.177505 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:55.177477 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnfvl\" (UniqueName: \"kubernetes.io/projected/0783d93b-e60a-4769-b757-430aa9bf97a7-kube-api-access-pnfvl\") pod \"maas-api-revoke-sub-cab7d423-bf5jd\" (UID: \"0783d93b-e60a-4769-b757-430aa9bf97a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab7d423-bf5jd" Oct 30 14:30:55.278827 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:55.278795 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnfvl\" (UniqueName: \"kubernetes.io/projected/0783d93b-e60a-4769-b757-430aa9bf97a7-kube-api-access-pnfvl\") pod \"maas-api-revoke-sub-cab7d423-bf5jd\" (UID: \"0783d93b-e60a-4769-b757-430aa9bf97a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab7d423-bf5jd" Oct 30 14:30:55.278980 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:55.278896 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0783d93b-e60a-4769-b757-430aa9bf97a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cab7d423-bf5jd\" (UID: \"0783d93b-e60a-4769-b757-430aa9bf97a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab7d423-bf5jd" Oct 30 14:30:55.278980 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:55.278937 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0783d93b-e60a-4769-b757-430aa9bf97a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cab7d423-bf5jd\" (UID: \"0783d93b-e60a-4769-b757-430aa9bf97a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab7d423-bf5jd" Oct 30 14:30:55.279532 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:55.279510 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0783d93b-e60a-4769-b757-430aa9bf97a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cab7d423-bf5jd\" (UID: \"0783d93b-e60a-4769-b757-430aa9bf97a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab7d423-bf5jd" Oct 30 14:30:55.281260 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:55.281228 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0783d93b-e60a-4769-b757-430aa9bf97a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cab7d423-bf5jd\" (UID: \"0783d93b-e60a-4769-b757-430aa9bf97a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab7d423-bf5jd" Oct 30 14:30:55.308688 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:55.308614 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnfvl\" (UniqueName: \"kubernetes.io/projected/0783d93b-e60a-4769-b757-430aa9bf97a7-kube-api-access-pnfvl\") pod \"maas-api-revoke-sub-cab7d423-bf5jd\" (UID: \"0783d93b-e60a-4769-b757-430aa9bf97a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab7d423-bf5jd" Oct 30 14:30:55.379784 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:55.379753 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab7d423-bf5jd" Oct 30 14:30:55.692685 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:30:55.690931 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0783d93b_e60a_4769_b757_430aa9bf97a7.slice/crio-03a2d2505a9b1dc34cf50918af7ba474f5375dd7e50aafaa2ea952e35980621c WatchSource:0}: Error finding container 03a2d2505a9b1dc34cf50918af7ba474f5375dd7e50aafaa2ea952e35980621c: Status 404 returned error can't find the container with id 03a2d2505a9b1dc34cf50918af7ba474f5375dd7e50aafaa2ea952e35980621c Oct 30 14:30:55.692685 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:55.692014 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cab7d423-bf5jd"] Oct 30 14:30:56.188135 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:56.188097 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab7d423-bf5jd" event={"ID":"0783d93b-e60a-4769-b757-430aa9bf97a7","Type":"ContainerStarted","Data":"18caf6b5fb83db8f935ce4ed7cc69fe8b3ddeb0f56fba3279c28fd5515a0566e"} Oct 30 14:30:56.188604 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:56.188156 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab7d423-bf5jd" event={"ID":"0783d93b-e60a-4769-b757-430aa9bf97a7","Type":"ContainerStarted","Data":"03a2d2505a9b1dc34cf50918af7ba474f5375dd7e50aafaa2ea952e35980621c"} Oct 30 14:30:56.233592 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:56.233538 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab7d423-bf5jd" podStartSLOduration=1.233525152 podStartE2EDuration="1.233525152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:30:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:30:56.231289287 +0000 UTC m=+2525.975423327" watchObservedRunningTime="2026-10-30 14:30:56.233525152 +0000 UTC m=+2525.977659193" Oct 30 14:30:56.586830 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:56.586786 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:30:57.197782 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:57.197749 4982 generic.go:356] "Generic (PLEG): container finished" podID="0783d93b-e60a-4769-b757-430aa9bf97a7" containerID="18caf6b5fb83db8f935ce4ed7cc69fe8b3ddeb0f56fba3279c28fd5515a0566e" exitCode=0 Oct 30 14:30:57.198250 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:57.197833 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab7d423-bf5jd" event={"ID":"0783d93b-e60a-4769-b757-430aa9bf97a7","Type":"ContainerDied","Data":"18caf6b5fb83db8f935ce4ed7cc69fe8b3ddeb0f56fba3279c28fd5515a0566e"} Oct 30 14:30:58.012524 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:30:58.012482 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:30:58.382024 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:58.381994 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab7d423-bf5jd" Oct 30 14:30:58.528810 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:58.528720 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0783d93b-e60a-4769-b757-430aa9bf97a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0783d93b-e60a-4769-b757-430aa9bf97a7-maas-api-openshift-ca\") pod \"0783d93b-e60a-4769-b757-430aa9bf97a7\" (UID: \"0783d93b-e60a-4769-b757-430aa9bf97a7\") " Oct 30 14:30:58.528810 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:58.528775 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0783d93b-e60a-4769-b757-430aa9bf97a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0783d93b-e60a-4769-b757-430aa9bf97a7-maas-api-xks-ca\") pod \"0783d93b-e60a-4769-b757-430aa9bf97a7\" (UID: \"0783d93b-e60a-4769-b757-430aa9bf97a7\") " Oct 30 14:30:58.529065 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:58.528867 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0783d93b-e60a-4769-b757-430aa9bf97a7-kube-api-access-pnfvl\" (UniqueName: \"kubernetes.io/projected/0783d93b-e60a-4769-b757-430aa9bf97a7-kube-api-access-pnfvl\") pod \"0783d93b-e60a-4769-b757-430aa9bf97a7\" (UID: \"0783d93b-e60a-4769-b757-430aa9bf97a7\") " Oct 30 14:30:58.529120 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:58.529088 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0783d93b-e60a-4769-b757-430aa9bf97a7-maas-api-openshift-ca" pod "0783d93b-e60a-4769-b757-430aa9bf97a7" (UID: "0783d93b-e60a-4769-b757-430aa9bf97a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:30:58.529442 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:58.529419 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0783d93b-e60a-4769-b757-430aa9bf97a7-maas-api-openshift-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:30:58.530922 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:58.530897 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0783d93b-e60a-4769-b757-430aa9bf97a7-maas-api-xks-ca" pod "0783d93b-e60a-4769-b757-430aa9bf97a7" (UID: "0783d93b-e60a-4769-b757-430aa9bf97a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:30:58.531042 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:58.530998 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0783d93b-e60a-4769-b757-430aa9bf97a7-kube-api-access-pnfvl" pod "0783d93b-e60a-4769-b757-430aa9bf97a7" (UID: "0783d93b-e60a-4769-b757-430aa9bf97a7"). InnerVolumeSpecName "kube-api-access-pnfvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:30:58.630997 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:58.630961 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0783d93b-e60a-4769-b757-430aa9bf97a7-maas-api-xks-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:30:58.630997 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:58.630989 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnfvl\" (UniqueName: \"kubernetes.io/projected/0783d93b-e60a-4769-b757-430aa9bf97a7-kube-api-access-pnfvl\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:30:58.646417 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:58.646366 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:30:58.879292 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:58.879246 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:30:59.046455 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:59.046414 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:30:59.228363 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:59.228268 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab7d423-bf5jd" event={"ID":"0783d93b-e60a-4769-b757-430aa9bf97a7","Type":"ContainerDied","Data":"03a2d2505a9b1dc34cf50918af7ba474f5375dd7e50aafaa2ea952e35980621c"} Oct 30 14:30:59.228363 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:59.228347 4982 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03a2d2505a9b1dc34cf50918af7ba474f5375dd7e50aafaa2ea952e35980621c" Oct 30 14:30:59.228620 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:59.228598 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab7d423-bf5jd" Oct 30 14:30:59.256121 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:30:59.256081 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:31:00.076936 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:00.076896 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:31:00.550676 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:00.550573 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:31:00.742006 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:00.741975 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj"] Oct 30 14:31:00.742621 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:00.742602 4982 cpu_manager.go:414] "containerMap: removing container" podUID="0783d93b-e60a-4769-b757-430aa9bf97a7" containerName="revoke-keys" Oct 30 14:31:00.742621 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:00.742618 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0783d93b-e60a-4769-b757-430aa9bf97a7" containerName="revoke-keys" Oct 30 14:31:00.742814 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:00.742774 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="0783d93b-e60a-4769-b757-430aa9bf97a7" containerName="revoke-keys" Oct 30 14:31:00.746762 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:00.746736 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj" Oct 30 14:31:00.779914 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:00.779885 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj"] Oct 30 14:31:00.783723 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:00.783695 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-396d70-openshift-default-dockercfg-pnzlz\"" Oct 30 14:31:00.878450 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:00.878397 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2de9f651-0083-42cd-89b2-6d28d35bb449-istio-envoy\") pod \"e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj\" (UID: \"2de9f651-0083-42cd-89b2-6d28d35bb449\") " pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj" Oct 30 14:31:00.878737 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:00.878708 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2de9f651-0083-42cd-89b2-6d28d35bb449-credential-socket\") pod \"e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj\" (UID: \"2de9f651-0083-42cd-89b2-6d28d35bb449\") " pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj" Oct 30 14:31:00.879100 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:00.879077 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2de9f651-0083-42cd-89b2-6d28d35bb449-istiod-ca-cert\") pod \"e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj\" (UID: \"2de9f651-0083-42cd-89b2-6d28d35bb449\") " pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj" Oct 30 14:31:00.879302 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:00.879275 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2de9f651-0083-42cd-89b2-6d28d35bb449-workload-certs\") pod \"e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj\" (UID: \"2de9f651-0083-42cd-89b2-6d28d35bb449\") " pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj" Oct 30 14:31:00.879488 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:00.879475 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2de9f651-0083-42cd-89b2-6d28d35bb449-workload-socket\") pod \"e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj\" (UID: \"2de9f651-0083-42cd-89b2-6d28d35bb449\") " pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj" Oct 30 14:31:00.879616 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:00.879604 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2de9f651-0083-42cd-89b2-6d28d35bb449-istio-token\") pod \"e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj\" (UID: \"2de9f651-0083-42cd-89b2-6d28d35bb449\") " pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj" Oct 30 14:31:00.883481 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:00.883456 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2de9f651-0083-42cd-89b2-6d28d35bb449-istio-data\") pod \"e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj\" (UID: \"2de9f651-0083-42cd-89b2-6d28d35bb449\") " pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj" Oct 30 14:31:00.883717 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:00.883698 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2de9f651-0083-42cd-89b2-6d28d35bb449-istio-podinfo\") pod \"e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj\" (UID: \"2de9f651-0083-42cd-89b2-6d28d35bb449\") " pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj" Oct 30 14:31:00.883866 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:00.883848 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r97zz\" (UniqueName: \"kubernetes.io/projected/2de9f651-0083-42cd-89b2-6d28d35bb449-kube-api-access-r97zz\") pod \"e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj\" (UID: \"2de9f651-0083-42cd-89b2-6d28d35bb449\") " pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj" Oct 30 14:31:00.885237 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:00.885202 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:31:00.991622 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:00.991581 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2de9f651-0083-42cd-89b2-6d28d35bb449-istiod-ca-cert\") pod \"e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj\" (UID: \"2de9f651-0083-42cd-89b2-6d28d35bb449\") " pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj" Oct 30 14:31:00.991956 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:00.991928 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2de9f651-0083-42cd-89b2-6d28d35bb449-workload-certs\") pod \"e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj\" (UID: \"2de9f651-0083-42cd-89b2-6d28d35bb449\") " pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj" Oct 30 14:31:00.992179 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:00.992155 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2de9f651-0083-42cd-89b2-6d28d35bb449-workload-socket\") pod \"e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj\" (UID: \"2de9f651-0083-42cd-89b2-6d28d35bb449\") " pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj" Oct 30 14:31:00.992273 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:00.992201 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2de9f651-0083-42cd-89b2-6d28d35bb449-istio-token\") pod \"e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj\" (UID: \"2de9f651-0083-42cd-89b2-6d28d35bb449\") " pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj" Oct 30 14:31:00.992330 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:00.992284 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2de9f651-0083-42cd-89b2-6d28d35bb449-istio-data\") pod \"e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj\" (UID: \"2de9f651-0083-42cd-89b2-6d28d35bb449\") " pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj" Oct 30 14:31:00.992330 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:00.992322 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2de9f651-0083-42cd-89b2-6d28d35bb449-istio-podinfo\") pod \"e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj\" (UID: \"2de9f651-0083-42cd-89b2-6d28d35bb449\") " pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj" Oct 30 14:31:00.992452 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:00.992368 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r97zz\" (UniqueName: \"kubernetes.io/projected/2de9f651-0083-42cd-89b2-6d28d35bb449-kube-api-access-r97zz\") pod \"e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj\" (UID: \"2de9f651-0083-42cd-89b2-6d28d35bb449\") " pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj" Oct 30 14:31:00.992452 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:00.992431 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2de9f651-0083-42cd-89b2-6d28d35bb449-istio-envoy\") pod \"e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj\" (UID: \"2de9f651-0083-42cd-89b2-6d28d35bb449\") " pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj" Oct 30 14:31:00.992874 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:00.992485 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2de9f651-0083-42cd-89b2-6d28d35bb449-credential-socket\") pod \"e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj\" (UID: \"2de9f651-0083-42cd-89b2-6d28d35bb449\") " pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj" Oct 30 14:31:00.992874 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:00.992506 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2de9f651-0083-42cd-89b2-6d28d35bb449-istiod-ca-cert\") pod \"e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj\" (UID: \"2de9f651-0083-42cd-89b2-6d28d35bb449\") " pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj" Oct 30 14:31:00.992986 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:00.992872 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2de9f651-0083-42cd-89b2-6d28d35bb449-credential-socket\") pod \"e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj\" (UID: \"2de9f651-0083-42cd-89b2-6d28d35bb449\") " pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj" Oct 30 14:31:00.992986 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:00.992960 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2de9f651-0083-42cd-89b2-6d28d35bb449-istio-data\") pod \"e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj\" (UID: \"2de9f651-0083-42cd-89b2-6d28d35bb449\") " pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj" Oct 30 14:31:00.993215 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:00.993192 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2de9f651-0083-42cd-89b2-6d28d35bb449-workload-certs\") pod \"e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj\" (UID: \"2de9f651-0083-42cd-89b2-6d28d35bb449\") " pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj" Oct 30 14:31:00.993473 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:00.993409 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2de9f651-0083-42cd-89b2-6d28d35bb449-workload-socket\") pod \"e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj\" (UID: \"2de9f651-0083-42cd-89b2-6d28d35bb449\") " pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj" Oct 30 14:31:00.996329 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:00.996305 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2de9f651-0083-42cd-89b2-6d28d35bb449-istio-envoy\") pod \"e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj\" (UID: \"2de9f651-0083-42cd-89b2-6d28d35bb449\") " pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj" Oct 30 14:31:00.998148 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:00.998125 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2de9f651-0083-42cd-89b2-6d28d35bb449-istio-podinfo\") pod \"e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj\" (UID: \"2de9f651-0083-42cd-89b2-6d28d35bb449\") " pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj" Oct 30 14:31:01.024679 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:01.022306 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2de9f651-0083-42cd-89b2-6d28d35bb449-istio-token\") pod \"e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj\" (UID: \"2de9f651-0083-42cd-89b2-6d28d35bb449\") " pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj" Oct 30 14:31:01.027250 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:01.027218 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r97zz\" (UniqueName: \"kubernetes.io/projected/2de9f651-0083-42cd-89b2-6d28d35bb449-kube-api-access-r97zz\") pod \"e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj\" (UID: \"2de9f651-0083-42cd-89b2-6d28d35bb449\") " pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj" Oct 30 14:31:01.064287 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:01.064229 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:31:01.065246 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:01.065224 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj" Oct 30 14:31:01.398829 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:31:01.398798 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2de9f651_0083_42cd_89b2_6d28d35bb449.slice/crio-40bb0d63f3a67212de14391f8e86eeab7c060397645bac2dcba7a5acd570a0a1 WatchSource:0}: Error finding container 40bb0d63f3a67212de14391f8e86eeab7c060397645bac2dcba7a5acd570a0a1: Status 404 returned error can't find the container with id 40bb0d63f3a67212de14391f8e86eeab7c060397645bac2dcba7a5acd570a0a1 Oct 30 14:31:01.399241 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:01.398923 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj"] Oct 30 14:31:01.401933 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:01.401901 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:31:01.402035 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:01.401962 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:31:01.402035 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:01.401989 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:31:02.254099 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:02.254056 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj" event={"ID":"2de9f651-0083-42cd-89b2-6d28d35bb449","Type":"ContainerStarted","Data":"e5559289b82187bb9c7a4148b85b8a8b2bbe15ed976c90af906c8283cf10f3c4"} Oct 30 14:31:02.254258 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:02.254115 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj" event={"ID":"2de9f651-0083-42cd-89b2-6d28d35bb449","Type":"ContainerStarted","Data":"40bb0d63f3a67212de14391f8e86eeab7c060397645bac2dcba7a5acd570a0a1"} Oct 30 14:31:02.316269 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:02.316222 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj" podStartSLOduration=2.3162102620000002 podStartE2EDuration="2.316210262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:31:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:31:02.315809383 +0000 UTC m=+2532.059943424" watchObservedRunningTime="2026-10-30 14:31:02.316210262 +0000 UTC m=+2532.060344302" Oct 30 14:31:02.529177 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:02.529092 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:31:03.065944 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:03.065905 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj" Oct 30 14:31:03.073575 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:03.073545 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj" Oct 30 14:31:03.261127 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:03.261090 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj" Oct 30 14:31:03.262070 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:03.262047 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj" Oct 30 14:31:04.559562 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:04.559532 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:31:06.361248 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:06.361212 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:31:06.536389 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:06.536325 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:31:06.976160 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:06.976123 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-34bcc5/test-model-34bcc5-kserve-788cf96895-psz95"] Oct 30 14:31:06.992465 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:06.992432 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:31:07.441158 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:07.441122 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-409364/test-model-409364-kserve-588bbd6c75-k96rk"] Oct 30 14:31:07.461445 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:07.461405 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18b719b9-qrvcd"] Oct 30 14:31:07.466870 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:07.466845 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18b719b9-qrvcd" Oct 30 14:31:07.496680 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:07.487965 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18b719b9-qrvcd"] Oct 30 14:31:07.496680 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:07.495616 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:31:07.496680 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:07.495761 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:31:07.502701 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:07.502666 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:31:07.669246 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:07.662070 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c88117c-d932-43c2-8a2a-9402c12ce0f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18b719b9-qrvcd\" (UID: \"9c88117c-d932-43c2-8a2a-9402c12ce0f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18b719b9-qrvcd" Oct 30 14:31:07.669696 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:07.669663 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nztrw\" (UniqueName: \"kubernetes.io/projected/9c88117c-d932-43c2-8a2a-9402c12ce0f7-kube-api-access-nztrw\") pod \"maas-api-revoke-sub-18b719b9-qrvcd\" (UID: \"9c88117c-d932-43c2-8a2a-9402c12ce0f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18b719b9-qrvcd" Oct 30 14:31:07.669908 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:07.669890 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c88117c-d932-43c2-8a2a-9402c12ce0f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18b719b9-qrvcd\" (UID: \"9c88117c-d932-43c2-8a2a-9402c12ce0f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18b719b9-qrvcd" Oct 30 14:31:07.771251 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:07.771160 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nztrw\" (UniqueName: \"kubernetes.io/projected/9c88117c-d932-43c2-8a2a-9402c12ce0f7-kube-api-access-nztrw\") pod \"maas-api-revoke-sub-18b719b9-qrvcd\" (UID: \"9c88117c-d932-43c2-8a2a-9402c12ce0f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18b719b9-qrvcd" Oct 30 14:31:07.771251 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:07.771238 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c88117c-d932-43c2-8a2a-9402c12ce0f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18b719b9-qrvcd\" (UID: \"9c88117c-d932-43c2-8a2a-9402c12ce0f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18b719b9-qrvcd" Oct 30 14:31:07.771437 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:07.771422 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c88117c-d932-43c2-8a2a-9402c12ce0f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18b719b9-qrvcd\" (UID: \"9c88117c-d932-43c2-8a2a-9402c12ce0f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18b719b9-qrvcd" Oct 30 14:31:07.772271 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:07.772244 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c88117c-d932-43c2-8a2a-9402c12ce0f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18b719b9-qrvcd\" (UID: \"9c88117c-d932-43c2-8a2a-9402c12ce0f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18b719b9-qrvcd" Oct 30 14:31:07.774566 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:07.774543 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c88117c-d932-43c2-8a2a-9402c12ce0f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18b719b9-qrvcd\" (UID: \"9c88117c-d932-43c2-8a2a-9402c12ce0f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18b719b9-qrvcd" Oct 30 14:31:07.818906 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:07.809334 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nztrw\" (UniqueName: \"kubernetes.io/projected/9c88117c-d932-43c2-8a2a-9402c12ce0f7-kube-api-access-nztrw\") pod \"maas-api-revoke-sub-18b719b9-qrvcd\" (UID: \"9c88117c-d932-43c2-8a2a-9402c12ce0f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18b719b9-qrvcd" Oct 30 14:31:07.875130 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:07.872707 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18b719b9-qrvcd" Oct 30 14:31:08.249441 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:31:08.249380 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c88117c_d932_43c2_8a2a_9402c12ce0f7.slice/crio-ab035680a638986334ff5a984dd14655efd5b531cd53f0030f1aa5c3ec0f0a33 WatchSource:0}: Error finding container ab035680a638986334ff5a984dd14655efd5b531cd53f0030f1aa5c3ec0f0a33: Status 404 returned error can't find the container with id ab035680a638986334ff5a984dd14655efd5b531cd53f0030f1aa5c3ec0f0a33 Oct 30 14:31:08.249966 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:08.249933 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18b719b9-qrvcd"] Oct 30 14:31:08.329747 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:08.329701 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18b719b9-qrvcd" event={"ID":"9c88117c-d932-43c2-8a2a-9402c12ce0f7","Type":"ContainerStarted","Data":"ab035680a638986334ff5a984dd14655efd5b531cd53f0030f1aa5c3ec0f0a33"} Oct 30 14:31:08.398676 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:08.394963 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:31:09.340396 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:09.340361 4982 generic.go:356] "Generic (PLEG): container finished" podID="9c88117c-d932-43c2-8a2a-9402c12ce0f7" containerID="a12fcf491f1e9ef73ea04451090d06b541442570767df0445a074a65c06fa9f4" exitCode=0 Oct 30 14:31:09.340798 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:09.340398 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18b719b9-qrvcd" event={"ID":"9c88117c-d932-43c2-8a2a-9402c12ce0f7","Type":"ContainerDied","Data":"a12fcf491f1e9ef73ea04451090d06b541442570767df0445a074a65c06fa9f4"} Oct 30 14:31:10.437668 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:10.437609 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:31:10.557738 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:10.557707 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18b719b9-qrvcd" Oct 30 14:31:10.632888 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:10.632803 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c88117c-d932-43c2-8a2a-9402c12ce0f7-kube-api-access-nztrw\" (UniqueName: \"kubernetes.io/projected/9c88117c-d932-43c2-8a2a-9402c12ce0f7-kube-api-access-nztrw\") pod \"9c88117c-d932-43c2-8a2a-9402c12ce0f7\" (UID: \"9c88117c-d932-43c2-8a2a-9402c12ce0f7\") " Oct 30 14:31:10.633043 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:10.632927 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c88117c-d932-43c2-8a2a-9402c12ce0f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c88117c-d932-43c2-8a2a-9402c12ce0f7-maas-api-openshift-ca\") pod \"9c88117c-d932-43c2-8a2a-9402c12ce0f7\" (UID: \"9c88117c-d932-43c2-8a2a-9402c12ce0f7\") " Oct 30 14:31:10.633043 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:10.632963 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c88117c-d932-43c2-8a2a-9402c12ce0f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c88117c-d932-43c2-8a2a-9402c12ce0f7-maas-api-xks-ca\") pod \"9c88117c-d932-43c2-8a2a-9402c12ce0f7\" (UID: \"9c88117c-d932-43c2-8a2a-9402c12ce0f7\") " Oct 30 14:31:10.634337 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:10.634295 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c88117c-d932-43c2-8a2a-9402c12ce0f7-maas-api-openshift-ca" pod "9c88117c-d932-43c2-8a2a-9402c12ce0f7" (UID: "9c88117c-d932-43c2-8a2a-9402c12ce0f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:31:10.635660 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:10.635109 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c88117c-d932-43c2-8a2a-9402c12ce0f7-kube-api-access-nztrw" pod "9c88117c-d932-43c2-8a2a-9402c12ce0f7" (UID: "9c88117c-d932-43c2-8a2a-9402c12ce0f7"). InnerVolumeSpecName "kube-api-access-nztrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:31:10.642301 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:10.636126 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c88117c-d932-43c2-8a2a-9402c12ce0f7-maas-api-xks-ca" pod "9c88117c-d932-43c2-8a2a-9402c12ce0f7" (UID: "9c88117c-d932-43c2-8a2a-9402c12ce0f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:31:10.735499 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:10.735461 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c88117c-d932-43c2-8a2a-9402c12ce0f7-maas-api-openshift-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:31:10.735499 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:10.735502 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c88117c-d932-43c2-8a2a-9402c12ce0f7-maas-api-xks-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:31:10.735714 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:10.735521 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nztrw\" (UniqueName: \"kubernetes.io/projected/9c88117c-d932-43c2-8a2a-9402c12ce0f7-kube-api-access-nztrw\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:31:11.357658 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:11.357614 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18b719b9-qrvcd" Oct 30 14:31:11.357848 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:11.357657 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18b719b9-qrvcd" event={"ID":"9c88117c-d932-43c2-8a2a-9402c12ce0f7","Type":"ContainerDied","Data":"ab035680a638986334ff5a984dd14655efd5b531cd53f0030f1aa5c3ec0f0a33"} Oct 30 14:31:11.357848 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:11.357752 4982 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab035680a638986334ff5a984dd14655efd5b531cd53f0030f1aa5c3ec0f0a33" Oct 30 14:31:12.242821 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:12.242785 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:31:12.367426 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:12.367394 4982 generic.go:356] "Generic (PLEG): container finished" podID="30019687-440e-44fd-8fd8-0d2902927ca0" containerID="883719cef92123e240e8eaca0d0210e852634519c879e191779b6e08cd6a87f7" exitCode=0 Oct 30 14:31:12.367607 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:12.367476 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2" event={"ID":"30019687-440e-44fd-8fd8-0d2902927ca0","Type":"ContainerDied","Data":"883719cef92123e240e8eaca0d0210e852634519c879e191779b6e08cd6a87f7"} Oct 30 14:31:12.368003 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:12.367966 4982 scope.go:122] "RemoveContainer" containerID="883719cef92123e240e8eaca0d0210e852634519c879e191779b6e08cd6a87f7" Oct 30 14:31:12.369814 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:12.369787 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:31:12.369887 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:12.369853 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:31:12.369922 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:12.369887 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:31:13.378107 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:13.378074 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2" event={"ID":"30019687-440e-44fd-8fd8-0d2902927ca0","Type":"ContainerStarted","Data":"84fd2193c786eeb72126d7ad2ce8da8b2ca3e283eb48540c6bee55e1d4380633"} Oct 30 14:31:13.952353 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:13.952308 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2" Oct 30 14:31:13.952843 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:13.952818 4982 patch_prober.go:29] interesting pod/e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 30 14:31:13.952945 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:13.952865 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2" podUID="30019687-440e-44fd-8fd8-0d2902927ca0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 30 14:31:14.317075 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:14.317035 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:31:14.825226 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:14.825171 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:31:14.952723 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:14.952688 4982 patch_prober.go:29] interesting pod/e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 30 14:31:14.952917 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:14.952739 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2" podUID="30019687-440e-44fd-8fd8-0d2902927ca0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 30 14:31:14.970746 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:14.970711 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:31:15.952130 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:15.952091 4982 patch_prober.go:29] interesting pod/e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 30 14:31:15.952523 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:15.952165 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2" podUID="30019687-440e-44fd-8fd8-0d2902927ca0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 30 14:31:16.363016 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:16.362974 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:31:16.678329 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:16.678235 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:31:16.755764 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:16.755735 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-396d70-86964fb4d6-zmxfm"] Oct 30 14:31:16.756388 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:16.756374 4982 cpu_manager.go:414] "containerMap: removing container" podUID="9c88117c-d932-43c2-8a2a-9402c12ce0f7" containerName="revoke-keys" Oct 30 14:31:16.756434 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:16.756390 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c88117c-d932-43c2-8a2a-9402c12ce0f7" containerName="revoke-keys" Oct 30 14:31:16.756523 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:16.756513 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c88117c-d932-43c2-8a2a-9402c12ce0f7" containerName="revoke-keys" Oct 30 14:31:16.760370 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:16.760351 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-396d70-86964fb4d6-zmxfm" Oct 30 14:31:16.801662 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:16.797886 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-396d70\"" Oct 30 14:31:16.824666 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:16.824079 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-396d70-86964fb4d6-zmxfm"] Oct 30 14:31:16.832662 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:16.831152 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-396d70-56874f4975-w4bw5"] Oct 30 14:31:16.835696 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:16.835498 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-396d70-56874f4975-w4bw5" Oct 30 14:31:16.847461 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:16.847371 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:31:16.862732 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:16.862629 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-396d70-56874f4975-w4bw5"] Oct 30 14:31:16.863124 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:16.863105 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-396d70-dockercfg-lht6q\"" Oct 30 14:31:16.877043 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:16.877003 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8343dab-b9ab-4453-bfa3-aada32b3ebf0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-396d70-56874f4975-w4bw5\" (UID: \"b8343dab-b9ab-4453-bfa3-aada32b3ebf0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-396d70-56874f4975-w4bw5" Oct 30 14:31:16.877194 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:16.877174 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2f4s\" (UniqueName: \"kubernetes.io/projected/973f4a7b-21e1-4e4d-8190-56032de67388-kube-api-access-q2f4s\") pod \"payload-processing-e2e-shared-a-w2-396d70-86964fb4d6-zmxfm\" (UID: \"973f4a7b-21e1-4e4d-8190-56032de67388\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-396d70-86964fb4d6-zmxfm" Oct 30 14:31:16.877270 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:16.877211 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctlps\" (UniqueName: \"kubernetes.io/projected/b8343dab-b9ab-4453-bfa3-aada32b3ebf0-kube-api-access-ctlps\") pod \"payload-pre-processing-e2e-shared-a-w2-396d70-56874f4975-w4bw5\" (UID: \"b8343dab-b9ab-4453-bfa3-aada32b3ebf0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-396d70-56874f4975-w4bw5" Oct 30 14:31:16.877331 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:16.877302 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/973f4a7b-21e1-4e4d-8190-56032de67388-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-396d70-86964fb4d6-zmxfm\" (UID: \"973f4a7b-21e1-4e4d-8190-56032de67388\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-396d70-86964fb4d6-zmxfm" Oct 30 14:31:16.984716 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:16.979334 4982 patch_prober.go:29] interesting pod/e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 30 14:31:16.984716 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:16.979393 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2" podUID="30019687-440e-44fd-8fd8-0d2902927ca0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 30 14:31:16.984716 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:16.981242 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2f4s\" (UniqueName: \"kubernetes.io/projected/973f4a7b-21e1-4e4d-8190-56032de67388-kube-api-access-q2f4s\") pod \"payload-processing-e2e-shared-a-w2-396d70-86964fb4d6-zmxfm\" (UID: \"973f4a7b-21e1-4e4d-8190-56032de67388\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-396d70-86964fb4d6-zmxfm" Oct 30 14:31:16.984716 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:16.981320 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctlps\" (UniqueName: \"kubernetes.io/projected/b8343dab-b9ab-4453-bfa3-aada32b3ebf0-kube-api-access-ctlps\") pod \"payload-pre-processing-e2e-shared-a-w2-396d70-56874f4975-w4bw5\" (UID: \"b8343dab-b9ab-4453-bfa3-aada32b3ebf0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-396d70-56874f4975-w4bw5" Oct 30 14:31:16.984716 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:16.981415 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/973f4a7b-21e1-4e4d-8190-56032de67388-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-396d70-86964fb4d6-zmxfm\" (UID: \"973f4a7b-21e1-4e4d-8190-56032de67388\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-396d70-86964fb4d6-zmxfm" Oct 30 14:31:16.984716 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:16.981587 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8343dab-b9ab-4453-bfa3-aada32b3ebf0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-396d70-56874f4975-w4bw5\" (UID: \"b8343dab-b9ab-4453-bfa3-aada32b3ebf0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-396d70-56874f4975-w4bw5" Oct 30 14:31:17.021521 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:17.006309 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/973f4a7b-21e1-4e4d-8190-56032de67388-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-396d70-86964fb4d6-zmxfm\" (UID: \"973f4a7b-21e1-4e4d-8190-56032de67388\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-396d70-86964fb4d6-zmxfm" Oct 30 14:31:17.021521 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:17.007526 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8343dab-b9ab-4453-bfa3-aada32b3ebf0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-396d70-56874f4975-w4bw5\" (UID: \"b8343dab-b9ab-4453-bfa3-aada32b3ebf0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-396d70-56874f4975-w4bw5" Oct 30 14:31:17.115759 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:17.095553 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctlps\" (UniqueName: \"kubernetes.io/projected/b8343dab-b9ab-4453-bfa3-aada32b3ebf0-kube-api-access-ctlps\") pod \"payload-pre-processing-e2e-shared-a-w2-396d70-56874f4975-w4bw5\" (UID: \"b8343dab-b9ab-4453-bfa3-aada32b3ebf0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-396d70-56874f4975-w4bw5" Oct 30 14:31:17.146109 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:17.146060 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2f4s\" (UniqueName: \"kubernetes.io/projected/973f4a7b-21e1-4e4d-8190-56032de67388-kube-api-access-q2f4s\") pod \"payload-processing-e2e-shared-a-w2-396d70-86964fb4d6-zmxfm\" (UID: \"973f4a7b-21e1-4e4d-8190-56032de67388\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-396d70-86964fb4d6-zmxfm" Oct 30 14:31:17.152847 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:17.152819 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-396d70-56874f4975-w4bw5" Oct 30 14:31:17.373714 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:17.373676 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-396d70-86964fb4d6-zmxfm" Oct 30 14:31:17.530699 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:17.530441 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-396d70-56874f4975-w4bw5"] Oct 30 14:31:17.742761 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:17.742733 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-396d70-86964fb4d6-zmxfm"] Oct 30 14:31:17.952775 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:17.952735 4982 patch_prober.go:29] interesting pod/e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 30 14:31:17.952943 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:17.952801 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2" podUID="30019687-440e-44fd-8fd8-0d2902927ca0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 30 14:31:18.157288 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:18.157143 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:31:18.472350 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:18.472244 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-396d70-86964fb4d6-zmxfm" event={"ID":"973f4a7b-21e1-4e4d-8190-56032de67388","Type":"ContainerStarted","Data":"dc8275981ecb1e7a37a746490d9d950946536e02c7dd9cfc2a25f60eeaec2c34"} Oct 30 14:31:18.472693 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:18.472662 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-396d70-86964fb4d6-zmxfm" event={"ID":"973f4a7b-21e1-4e4d-8190-56032de67388","Type":"ContainerStarted","Data":"87d68f3808213c7f29f98fca14e0ce0ade36c84b6b3ee2ee5d362802823a186c"} Oct 30 14:31:18.473886 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:18.473862 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-396d70-86964fb4d6-zmxfm" Oct 30 14:31:18.481377 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:18.481347 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-396d70-56874f4975-w4bw5" event={"ID":"b8343dab-b9ab-4453-bfa3-aada32b3ebf0","Type":"ContainerStarted","Data":"4b21cc04f93430c749d2d445ef0195bdb38c0db93fb5477ab5c175a4086ebfa9"} Oct 30 14:31:18.481567 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:18.481549 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-396d70-56874f4975-w4bw5" event={"ID":"b8343dab-b9ab-4453-bfa3-aada32b3ebf0","Type":"ContainerStarted","Data":"15f3ae1d2e300de3536033c249997622b888c9042a723f516a43364f5d25065b"} Oct 30 14:31:18.482458 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:18.482436 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-396d70-56874f4975-w4bw5" Oct 30 14:31:18.556857 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:18.552980 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-396d70-86964fb4d6-zmxfm" podStartSLOduration=2.552959137 podStartE2EDuration="2.552959137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:31:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:31:18.538006574 +0000 UTC m=+2548.282140639" watchObservedRunningTime="2026-10-30 14:31:18.552959137 +0000 UTC m=+2548.297093181" Oct 30 14:31:18.592406 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:18.592350 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-396d70-56874f4975-w4bw5" podStartSLOduration=2.592324011 podStartE2EDuration="2.592324011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:31:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:31:18.591917595 +0000 UTC m=+2548.336051635" watchObservedRunningTime="2026-10-30 14:31:18.592324011 +0000 UTC m=+2548.336458053" Oct 30 14:31:18.743358 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:18.743275 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:31:18.857871 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:18.857839 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d15602-847fx"] Oct 30 14:31:18.862170 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:18.862144 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d15602-847fx"] Oct 30 14:31:18.894370 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:18.894337 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:31:18.971663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:18.970990 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2" Oct 30 14:31:18.977472 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:18.972567 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2" Oct 30 14:31:18.977472 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:18.973553 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2" Oct 30 14:31:19.119673 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:19.118983 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:31:19.233956 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:19.233906 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-34bcc5-fbwcx"] Oct 30 14:31:19.253761 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:19.251790 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-34bcc5-fbwcx"] Oct 30 14:31:19.253761 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:19.252028 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-34bcc5-fbwcx" Oct 30 14:31:19.272665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:19.270518 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:31:19.272665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:19.270789 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:31:19.294668 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:19.286908 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f304f8d-a25b-4dfb-a750-3d8efc5c44fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-34bcc5-fbwcx\" (UID: \"7f304f8d-a25b-4dfb-a750-3d8efc5c44fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-34bcc5-fbwcx" Oct 30 14:31:19.294668 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:19.286990 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tlqc\" (UniqueName: \"kubernetes.io/projected/7f304f8d-a25b-4dfb-a750-3d8efc5c44fc-kube-api-access-2tlqc\") pod \"maas-api-revoke-keys-e2e-inf-a-34bcc5-fbwcx\" (UID: \"7f304f8d-a25b-4dfb-a750-3d8efc5c44fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-34bcc5-fbwcx" Oct 30 14:31:19.294668 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:19.287109 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f304f8d-a25b-4dfb-a750-3d8efc5c44fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-34bcc5-fbwcx\" (UID: \"7f304f8d-a25b-4dfb-a750-3d8efc5c44fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-34bcc5-fbwcx" Oct 30 14:31:19.388721 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:19.388227 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f304f8d-a25b-4dfb-a750-3d8efc5c44fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-34bcc5-fbwcx\" (UID: \"7f304f8d-a25b-4dfb-a750-3d8efc5c44fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-34bcc5-fbwcx" Oct 30 14:31:19.388721 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:19.388301 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tlqc\" (UniqueName: \"kubernetes.io/projected/7f304f8d-a25b-4dfb-a750-3d8efc5c44fc-kube-api-access-2tlqc\") pod \"maas-api-revoke-keys-e2e-inf-a-34bcc5-fbwcx\" (UID: \"7f304f8d-a25b-4dfb-a750-3d8efc5c44fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-34bcc5-fbwcx" Oct 30 14:31:19.388721 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:19.388364 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f304f8d-a25b-4dfb-a750-3d8efc5c44fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-34bcc5-fbwcx\" (UID: \"7f304f8d-a25b-4dfb-a750-3d8efc5c44fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-34bcc5-fbwcx" Oct 30 14:31:19.389281 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:19.389205 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f304f8d-a25b-4dfb-a750-3d8efc5c44fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-34bcc5-fbwcx\" (UID: \"7f304f8d-a25b-4dfb-a750-3d8efc5c44fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-34bcc5-fbwcx" Oct 30 14:31:19.389948 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:19.389917 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:31:19.392453 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:19.392430 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f304f8d-a25b-4dfb-a750-3d8efc5c44fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-34bcc5-fbwcx\" (UID: \"7f304f8d-a25b-4dfb-a750-3d8efc5c44fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-34bcc5-fbwcx" Oct 30 14:31:19.468036 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:19.465309 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tlqc\" (UniqueName: \"kubernetes.io/projected/7f304f8d-a25b-4dfb-a750-3d8efc5c44fc-kube-api-access-2tlqc\") pod \"maas-api-revoke-keys-e2e-inf-a-34bcc5-fbwcx\" (UID: \"7f304f8d-a25b-4dfb-a750-3d8efc5c44fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-34bcc5-fbwcx" Oct 30 14:31:19.507089 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:19.507057 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-b9c4b4/test-model-b9c4b4-kserve-f588646c5-52gfq"] Oct 30 14:31:19.585170 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:19.584446 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-34bcc5-fbwcx" Oct 30 14:31:19.586288 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:19.586247 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:31:19.971159 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:31:19.971131 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f304f8d_a25b_4dfb_a750_3d8efc5c44fc.slice/crio-66ddcddb3d0bda246c799bb7a7f911b3228dc2f4fd4af3f05a6dc4fd8776080e WatchSource:0}: Error finding container 66ddcddb3d0bda246c799bb7a7f911b3228dc2f4fd4af3f05a6dc4fd8776080e: Status 404 returned error can't find the container with id 66ddcddb3d0bda246c799bb7a7f911b3228dc2f4fd4af3f05a6dc4fd8776080e Oct 30 14:31:19.972573 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:19.972543 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-34bcc5-fbwcx"] Oct 30 14:31:20.404370 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:20.404327 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a2022f7-1c6c-4e2d-bfc8-11f02ce57c19" path="/var/lib/kubelet/pods/2a2022f7-1c6c-4e2d-bfc8-11f02ce57c19/volumes" Oct 30 14:31:20.497908 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:20.497868 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-34bcc5-fbwcx" event={"ID":"7f304f8d-a25b-4dfb-a750-3d8efc5c44fc","Type":"ContainerStarted","Data":"c0e20d9281483b5d4f8e912d9aeeeb652cf41ab81912e710b876babe4e789f8e"} Oct 30 14:31:20.498096 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:20.497933 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-34bcc5-fbwcx" event={"ID":"7f304f8d-a25b-4dfb-a750-3d8efc5c44fc","Type":"ContainerStarted","Data":"66ddcddb3d0bda246c799bb7a7f911b3228dc2f4fd4af3f05a6dc4fd8776080e"} Oct 30 14:31:20.544119 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:20.544014 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-34bcc5-fbwcx" podStartSLOduration=1.543998907 podStartE2EDuration="1.543998907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:31:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:31:20.542193587 +0000 UTC m=+2550.286327627" watchObservedRunningTime="2026-10-30 14:31:20.543998907 +0000 UTC m=+2550.288132948" Oct 30 14:31:21.228997 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:21.228866 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:31:21.506802 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:21.506716 4982 generic.go:356] "Generic (PLEG): container finished" podID="7f304f8d-a25b-4dfb-a750-3d8efc5c44fc" containerID="c0e20d9281483b5d4f8e912d9aeeeb652cf41ab81912e710b876babe4e789f8e" exitCode=0 Oct 30 14:31:21.506802 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:21.506755 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-34bcc5-fbwcx" event={"ID":"7f304f8d-a25b-4dfb-a750-3d8efc5c44fc","Type":"ContainerDied","Data":"c0e20d9281483b5d4f8e912d9aeeeb652cf41ab81912e710b876babe4e789f8e"} Oct 30 14:31:22.162615 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:22.162578 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-34bcc5/test-model-34bcc5-kserve-788cf96895-psz95" podUID="42c0836c-fae4-4eff-a0f9-298aafae924b" containerName="main" containerID="cri-o://2fced0d76febf14fcaf0666501ab137d9d3e8f13dde4770f92d249ceffbb2f4b" gracePeriod=45 Oct 30 14:31:22.466564 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:22.466528 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-409364/test-model-409364-kserve-588bbd6c75-k96rk" podUID="b106e9f4-a3ff-46b4-8880-5ac6b7c16372" containerName="main" containerID="cri-o://45da45ebe86de633560c645409aff8dec28d47a488b6eb059bfbdde3fbe66034" gracePeriod=45 Oct 30 14:31:22.471026 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:22.471001 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-34bcc5/test-model-34bcc5-kserve-788cf96895-psz95" Oct 30 14:31:22.520673 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:22.517681 4982 generic.go:356] "Generic (PLEG): container finished" podID="42c0836c-fae4-4eff-a0f9-298aafae924b" containerID="2fced0d76febf14fcaf0666501ab137d9d3e8f13dde4770f92d249ceffbb2f4b" exitCode=0 Oct 30 14:31:22.520673 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:22.518133 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-34bcc5/test-model-34bcc5-kserve-788cf96895-psz95" Oct 30 14:31:22.520673 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:22.518236 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-34bcc5/test-model-34bcc5-kserve-788cf96895-psz95" event={"ID":"42c0836c-fae4-4eff-a0f9-298aafae924b","Type":"ContainerDied","Data":"2fced0d76febf14fcaf0666501ab137d9d3e8f13dde4770f92d249ceffbb2f4b"} Oct 30 14:31:22.520673 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:22.518270 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-34bcc5/test-model-34bcc5-kserve-788cf96895-psz95" event={"ID":"42c0836c-fae4-4eff-a0f9-298aafae924b","Type":"ContainerDied","Data":"5fda6af4cf8ed3dbbd3f80b840ebb534456aa4d83a8e38399c88b5c84001f4fe"} Oct 30 14:31:22.520673 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:22.518290 4982 scope.go:122] "RemoveContainer" containerID="2fced0d76febf14fcaf0666501ab137d9d3e8f13dde4770f92d249ceffbb2f4b" Oct 30 14:31:22.547050 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:22.547012 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42c0836c-fae4-4eff-a0f9-298aafae924b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42c0836c-fae4-4eff-a0f9-298aafae924b-tmp-dir\") pod \"42c0836c-fae4-4eff-a0f9-298aafae924b\" (UID: \"42c0836c-fae4-4eff-a0f9-298aafae924b\") " Oct 30 14:31:22.547281 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:22.547073 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42c0836c-fae4-4eff-a0f9-298aafae924b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/42c0836c-fae4-4eff-a0f9-298aafae924b-dshm\") pod \"42c0836c-fae4-4eff-a0f9-298aafae924b\" (UID: \"42c0836c-fae4-4eff-a0f9-298aafae924b\") " Oct 30 14:31:22.547281 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:22.547127 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42c0836c-fae4-4eff-a0f9-298aafae924b-tls-certs\" (UniqueName: \"kubernetes.io/secret/42c0836c-fae4-4eff-a0f9-298aafae924b-tls-certs\") pod \"42c0836c-fae4-4eff-a0f9-298aafae924b\" (UID: \"42c0836c-fae4-4eff-a0f9-298aafae924b\") " Oct 30 14:31:22.547281 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:22.547179 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42c0836c-fae4-4eff-a0f9-298aafae924b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42c0836c-fae4-4eff-a0f9-298aafae924b-model-cache\") pod \"42c0836c-fae4-4eff-a0f9-298aafae924b\" (UID: \"42c0836c-fae4-4eff-a0f9-298aafae924b\") " Oct 30 14:31:22.547535 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:22.547440 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42c0836c-fae4-4eff-a0f9-298aafae924b-kube-api-access-7r85n\" (UniqueName: \"kubernetes.io/projected/42c0836c-fae4-4eff-a0f9-298aafae924b-kube-api-access-7r85n\") pod \"42c0836c-fae4-4eff-a0f9-298aafae924b\" (UID: \"42c0836c-fae4-4eff-a0f9-298aafae924b\") " Oct 30 14:31:22.547535 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:22.547487 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42c0836c-fae4-4eff-a0f9-298aafae924b-home\" (UniqueName: \"kubernetes.io/empty-dir/42c0836c-fae4-4eff-a0f9-298aafae924b-home\") pod \"42c0836c-fae4-4eff-a0f9-298aafae924b\" (UID: \"42c0836c-fae4-4eff-a0f9-298aafae924b\") " Oct 30 14:31:22.550118 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:22.549977 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42c0836c-fae4-4eff-a0f9-298aafae924b-tmp-dir" pod "42c0836c-fae4-4eff-a0f9-298aafae924b" (UID: "42c0836c-fae4-4eff-a0f9-298aafae924b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:31:22.553182 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:22.553134 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42c0836c-fae4-4eff-a0f9-298aafae924b-tls-certs" pod "42c0836c-fae4-4eff-a0f9-298aafae924b" (UID: "42c0836c-fae4-4eff-a0f9-298aafae924b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:31:22.553317 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:22.553180 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42c0836c-fae4-4eff-a0f9-298aafae924b-dshm" pod "42c0836c-fae4-4eff-a0f9-298aafae924b" (UID: "42c0836c-fae4-4eff-a0f9-298aafae924b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:31:22.553987 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:22.553960 4982 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42c0836c-fae4-4eff-a0f9-298aafae924b-tmp-dir\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:31:22.554106 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:22.553992 4982 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42c0836c-fae4-4eff-a0f9-298aafae924b-dshm\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:31:22.554106 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:22.554007 4982 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42c0836c-fae4-4eff-a0f9-298aafae924b-tls-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:31:22.554703 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:22.554678 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42c0836c-fae4-4eff-a0f9-298aafae924b-model-cache" pod "42c0836c-fae4-4eff-a0f9-298aafae924b" (UID: "42c0836c-fae4-4eff-a0f9-298aafae924b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:31:22.554996 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:22.554975 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42c0836c-fae4-4eff-a0f9-298aafae924b-home" pod "42c0836c-fae4-4eff-a0f9-298aafae924b" (UID: "42c0836c-fae4-4eff-a0f9-298aafae924b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:31:22.556384 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:22.556358 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42c0836c-fae4-4eff-a0f9-298aafae924b-kube-api-access-7r85n" pod "42c0836c-fae4-4eff-a0f9-298aafae924b" (UID: "42c0836c-fae4-4eff-a0f9-298aafae924b"). InnerVolumeSpecName "kube-api-access-7r85n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:31:22.588125 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:22.588096 4982 scope.go:122] "RemoveContainer" containerID="2fced0d76febf14fcaf0666501ab137d9d3e8f13dde4770f92d249ceffbb2f4b" Oct 30 14:31:22.590149 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:31:22.590111 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fced0d76febf14fcaf0666501ab137d9d3e8f13dde4770f92d249ceffbb2f4b\": container with ID starting with 2fced0d76febf14fcaf0666501ab137d9d3e8f13dde4770f92d249ceffbb2f4b not found: ID does not exist" containerID="2fced0d76febf14fcaf0666501ab137d9d3e8f13dde4770f92d249ceffbb2f4b" Oct 30 14:31:22.590605 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:22.590570 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fced0d76febf14fcaf0666501ab137d9d3e8f13dde4770f92d249ceffbb2f4b"} err="failed to get container status \"2fced0d76febf14fcaf0666501ab137d9d3e8f13dde4770f92d249ceffbb2f4b\": rpc error: code = NotFound desc = could not find container \"2fced0d76febf14fcaf0666501ab137d9d3e8f13dde4770f92d249ceffbb2f4b\": container with ID starting with 2fced0d76febf14fcaf0666501ab137d9d3e8f13dde4770f92d249ceffbb2f4b not found: ID does not exist" Oct 30 14:31:22.656047 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:22.655985 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r85n\" (UniqueName: \"kubernetes.io/projected/42c0836c-fae4-4eff-a0f9-298aafae924b-kube-api-access-7r85n\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:31:22.656047 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:22.656015 4982 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42c0836c-fae4-4eff-a0f9-298aafae924b-home\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:31:22.656047 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:22.656026 4982 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42c0836c-fae4-4eff-a0f9-298aafae924b-model-cache\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:31:22.703542 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:22.703515 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-34bcc5-fbwcx" Oct 30 14:31:22.757283 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:22.757247 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f304f8d-a25b-4dfb-a750-3d8efc5c44fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f304f8d-a25b-4dfb-a750-3d8efc5c44fc-maas-api-openshift-ca\") pod \"7f304f8d-a25b-4dfb-a750-3d8efc5c44fc\" (UID: \"7f304f8d-a25b-4dfb-a750-3d8efc5c44fc\") " Oct 30 14:31:22.757463 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:22.757435 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f304f8d-a25b-4dfb-a750-3d8efc5c44fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f304f8d-a25b-4dfb-a750-3d8efc5c44fc-maas-api-xks-ca\") pod \"7f304f8d-a25b-4dfb-a750-3d8efc5c44fc\" (UID: \"7f304f8d-a25b-4dfb-a750-3d8efc5c44fc\") " Oct 30 14:31:22.757536 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:22.757471 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f304f8d-a25b-4dfb-a750-3d8efc5c44fc-kube-api-access-2tlqc\" (UniqueName: \"kubernetes.io/projected/7f304f8d-a25b-4dfb-a750-3d8efc5c44fc-kube-api-access-2tlqc\") pod \"7f304f8d-a25b-4dfb-a750-3d8efc5c44fc\" (UID: \"7f304f8d-a25b-4dfb-a750-3d8efc5c44fc\") " Oct 30 14:31:22.758008 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:22.757981 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f304f8d-a25b-4dfb-a750-3d8efc5c44fc-maas-api-openshift-ca" pod "7f304f8d-a25b-4dfb-a750-3d8efc5c44fc" (UID: "7f304f8d-a25b-4dfb-a750-3d8efc5c44fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:31:22.763111 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:22.763087 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f304f8d-a25b-4dfb-a750-3d8efc5c44fc-maas-api-xks-ca" pod "7f304f8d-a25b-4dfb-a750-3d8efc5c44fc" (UID: "7f304f8d-a25b-4dfb-a750-3d8efc5c44fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:31:22.763775 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:22.763747 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f304f8d-a25b-4dfb-a750-3d8efc5c44fc-kube-api-access-2tlqc" pod "7f304f8d-a25b-4dfb-a750-3d8efc5c44fc" (UID: "7f304f8d-a25b-4dfb-a750-3d8efc5c44fc"). InnerVolumeSpecName "kube-api-access-2tlqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:31:22.784286 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:22.784260 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-409364/test-model-409364-kserve-588bbd6c75-k96rk" Oct 30 14:31:22.859341 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:22.859311 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b106e9f4-a3ff-46b4-8880-5ac6b7c16372-tls-certs\" (UniqueName: \"kubernetes.io/secret/b106e9f4-a3ff-46b4-8880-5ac6b7c16372-tls-certs\") pod \"b106e9f4-a3ff-46b4-8880-5ac6b7c16372\" (UID: \"b106e9f4-a3ff-46b4-8880-5ac6b7c16372\") " Oct 30 14:31:22.859514 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:22.859501 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b106e9f4-a3ff-46b4-8880-5ac6b7c16372-home\" (UniqueName: \"kubernetes.io/empty-dir/b106e9f4-a3ff-46b4-8880-5ac6b7c16372-home\") pod \"b106e9f4-a3ff-46b4-8880-5ac6b7c16372\" (UID: \"b106e9f4-a3ff-46b4-8880-5ac6b7c16372\") " Oct 30 14:31:22.859582 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:22.859538 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b106e9f4-a3ff-46b4-8880-5ac6b7c16372-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b106e9f4-a3ff-46b4-8880-5ac6b7c16372-dshm\") pod \"b106e9f4-a3ff-46b4-8880-5ac6b7c16372\" (UID: \"b106e9f4-a3ff-46b4-8880-5ac6b7c16372\") " Oct 30 14:31:22.859649 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:22.859612 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b106e9f4-a3ff-46b4-8880-5ac6b7c16372-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b106e9f4-a3ff-46b4-8880-5ac6b7c16372-tmp-dir\") pod \"b106e9f4-a3ff-46b4-8880-5ac6b7c16372\" (UID: \"b106e9f4-a3ff-46b4-8880-5ac6b7c16372\") " Oct 30 14:31:22.859713 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:22.859691 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b106e9f4-a3ff-46b4-8880-5ac6b7c16372-kube-api-access-gd9hq\" (UniqueName: \"kubernetes.io/projected/b106e9f4-a3ff-46b4-8880-5ac6b7c16372-kube-api-access-gd9hq\") pod \"b106e9f4-a3ff-46b4-8880-5ac6b7c16372\" (UID: \"b106e9f4-a3ff-46b4-8880-5ac6b7c16372\") " Oct 30 14:31:22.859765 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:22.859722 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b106e9f4-a3ff-46b4-8880-5ac6b7c16372-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b106e9f4-a3ff-46b4-8880-5ac6b7c16372-model-cache\") pod \"b106e9f4-a3ff-46b4-8880-5ac6b7c16372\" (UID: \"b106e9f4-a3ff-46b4-8880-5ac6b7c16372\") " Oct 30 14:31:22.859846 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:22.859809 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b106e9f4-a3ff-46b4-8880-5ac6b7c16372-home" pod "b106e9f4-a3ff-46b4-8880-5ac6b7c16372" (UID: "b106e9f4-a3ff-46b4-8880-5ac6b7c16372"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:31:22.860115 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:22.859944 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b106e9f4-a3ff-46b4-8880-5ac6b7c16372-tmp-dir" pod "b106e9f4-a3ff-46b4-8880-5ac6b7c16372" (UID: "b106e9f4-a3ff-46b4-8880-5ac6b7c16372"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:31:22.860208 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:22.860148 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b106e9f4-a3ff-46b4-8880-5ac6b7c16372-model-cache" pod "b106e9f4-a3ff-46b4-8880-5ac6b7c16372" (UID: "b106e9f4-a3ff-46b4-8880-5ac6b7c16372"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:31:22.860433 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:22.860407 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f304f8d-a25b-4dfb-a750-3d8efc5c44fc-maas-api-xks-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:31:22.860518 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:22.860439 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tlqc\" (UniqueName: \"kubernetes.io/projected/7f304f8d-a25b-4dfb-a750-3d8efc5c44fc-kube-api-access-2tlqc\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:31:22.860518 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:22.860458 4982 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b106e9f4-a3ff-46b4-8880-5ac6b7c16372-home\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:31:22.860518 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:22.860473 4982 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b106e9f4-a3ff-46b4-8880-5ac6b7c16372-tmp-dir\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:31:22.860518 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:22.860489 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f304f8d-a25b-4dfb-a750-3d8efc5c44fc-maas-api-openshift-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:31:22.860518 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:22.860501 4982 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b106e9f4-a3ff-46b4-8880-5ac6b7c16372-model-cache\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:31:22.861990 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:22.861965 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b106e9f4-a3ff-46b4-8880-5ac6b7c16372-tls-certs" pod "b106e9f4-a3ff-46b4-8880-5ac6b7c16372" (UID: "b106e9f4-a3ff-46b4-8880-5ac6b7c16372"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:31:22.862145 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:22.862123 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b106e9f4-a3ff-46b4-8880-5ac6b7c16372-kube-api-access-gd9hq" pod "b106e9f4-a3ff-46b4-8880-5ac6b7c16372" (UID: "b106e9f4-a3ff-46b4-8880-5ac6b7c16372"). InnerVolumeSpecName "kube-api-access-gd9hq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:31:22.862207 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:22.862162 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b106e9f4-a3ff-46b4-8880-5ac6b7c16372-dshm" pod "b106e9f4-a3ff-46b4-8880-5ac6b7c16372" (UID: "b106e9f4-a3ff-46b4-8880-5ac6b7c16372"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:31:22.871523 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:22.871499 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-34bcc5/test-model-34bcc5-kserve-788cf96895-psz95"] Oct 30 14:31:22.877393 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:22.877372 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-34bcc5/test-model-34bcc5-kserve-788cf96895-psz95"] Oct 30 14:31:22.962128 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:22.962062 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd9hq\" (UniqueName: \"kubernetes.io/projected/b106e9f4-a3ff-46b4-8880-5ac6b7c16372-kube-api-access-gd9hq\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:31:22.962128 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:22.962087 4982 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b106e9f4-a3ff-46b4-8880-5ac6b7c16372-tls-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:31:22.962128 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:22.962097 4982 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b106e9f4-a3ff-46b4-8880-5ac6b7c16372-dshm\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:31:23.258089 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:23.258016 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:31:23.527752 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:23.527628 4982 generic.go:356] "Generic (PLEG): container finished" podID="b106e9f4-a3ff-46b4-8880-5ac6b7c16372" containerID="45da45ebe86de633560c645409aff8dec28d47a488b6eb059bfbdde3fbe66034" exitCode=0 Oct 30 14:31:23.527752 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:23.527696 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-409364/test-model-409364-kserve-588bbd6c75-k96rk" Oct 30 14:31:23.527752 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:23.527708 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-409364/test-model-409364-kserve-588bbd6c75-k96rk" event={"ID":"b106e9f4-a3ff-46b4-8880-5ac6b7c16372","Type":"ContainerDied","Data":"45da45ebe86de633560c645409aff8dec28d47a488b6eb059bfbdde3fbe66034"} Oct 30 14:31:23.527752 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:23.527750 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-409364/test-model-409364-kserve-588bbd6c75-k96rk" event={"ID":"b106e9f4-a3ff-46b4-8880-5ac6b7c16372","Type":"ContainerDied","Data":"994710c3c4bcf635670964f71b435c3b34f11d3d3c86550e525de224233dc146"} Oct 30 14:31:23.528375 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:23.527772 4982 scope.go:122] "RemoveContainer" containerID="45da45ebe86de633560c645409aff8dec28d47a488b6eb059bfbdde3fbe66034" Oct 30 14:31:23.529887 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:23.529868 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-34bcc5-fbwcx" event={"ID":"7f304f8d-a25b-4dfb-a750-3d8efc5c44fc","Type":"ContainerDied","Data":"66ddcddb3d0bda246c799bb7a7f911b3228dc2f4fd4af3f05a6dc4fd8776080e"} Oct 30 14:31:23.529951 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:23.529898 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-34bcc5-fbwcx" Oct 30 14:31:23.529994 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:23.529905 4982 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66ddcddb3d0bda246c799bb7a7f911b3228dc2f4fd4af3f05a6dc4fd8776080e" Oct 30 14:31:23.544743 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:23.544718 4982 scope.go:122] "RemoveContainer" containerID="45da45ebe86de633560c645409aff8dec28d47a488b6eb059bfbdde3fbe66034" Oct 30 14:31:23.545118 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:31:23.545097 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45da45ebe86de633560c645409aff8dec28d47a488b6eb059bfbdde3fbe66034\": container with ID starting with 45da45ebe86de633560c645409aff8dec28d47a488b6eb059bfbdde3fbe66034 not found: ID does not exist" containerID="45da45ebe86de633560c645409aff8dec28d47a488b6eb059bfbdde3fbe66034" Oct 30 14:31:23.545189 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:23.545123 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45da45ebe86de633560c645409aff8dec28d47a488b6eb059bfbdde3fbe66034"} err="failed to get container status \"45da45ebe86de633560c645409aff8dec28d47a488b6eb059bfbdde3fbe66034\": rpc error: code = NotFound desc = could not find container \"45da45ebe86de633560c645409aff8dec28d47a488b6eb059bfbdde3fbe66034\": container with ID starting with 45da45ebe86de633560c645409aff8dec28d47a488b6eb059bfbdde3fbe66034 not found: ID does not exist" Oct 30 14:31:23.605987 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:23.605956 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-409364/test-model-409364-kserve-588bbd6c75-k96rk"] Oct 30 14:31:23.610531 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:23.610503 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-409364/test-model-409364-kserve-588bbd6c75-k96rk"] Oct 30 14:31:24.404649 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:24.404592 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42c0836c-fae4-4eff-a0f9-298aafae924b" path="/var/lib/kubelet/pods/42c0836c-fae4-4eff-a0f9-298aafae924b/volumes" Oct 30 14:31:24.405288 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:24.405263 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b106e9f4-a3ff-46b4-8880-5ac6b7c16372" path="/var/lib/kubelet/pods/b106e9f4-a3ff-46b4-8880-5ac6b7c16372/volumes" Oct 30 14:31:25.155420 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:25.155384 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:31:25.317454 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:25.317415 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:31:26.342530 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:26.342494 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:31:26.488811 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:26.488782 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:31:26.763257 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:26.763150 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:31:27.201957 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:27.201921 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-93549e-kxtrl"] Oct 30 14:31:27.202561 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:27.202547 4982 cpu_manager.go:414] "containerMap: removing container" podUID="b106e9f4-a3ff-46b4-8880-5ac6b7c16372" containerName="main" Oct 30 14:31:27.202608 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:27.202563 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b106e9f4-a3ff-46b4-8880-5ac6b7c16372" containerName="main" Oct 30 14:31:27.202608 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:27.202583 4982 cpu_manager.go:414] "containerMap: removing container" podUID="7f304f8d-a25b-4dfb-a750-3d8efc5c44fc" containerName="revoke-keys" Oct 30 14:31:27.202608 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:27.202588 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f304f8d-a25b-4dfb-a750-3d8efc5c44fc" containerName="revoke-keys" Oct 30 14:31:27.202608 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:27.202602 4982 cpu_manager.go:414] "containerMap: removing container" podUID="42c0836c-fae4-4eff-a0f9-298aafae924b" containerName="main" Oct 30 14:31:27.202608 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:27.202607 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42c0836c-fae4-4eff-a0f9-298aafae924b" containerName="main" Oct 30 14:31:27.202772 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:27.202730 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f304f8d-a25b-4dfb-a750-3d8efc5c44fc" containerName="revoke-keys" Oct 30 14:31:27.202772 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:27.202740 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="42c0836c-fae4-4eff-a0f9-298aafae924b" containerName="main" Oct 30 14:31:27.202772 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:27.202754 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="b106e9f4-a3ff-46b4-8880-5ac6b7c16372" containerName="main" Oct 30 14:31:27.207910 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:27.207888 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-93549e-kxtrl" Oct 30 14:31:27.211855 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:27.211831 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:31:27.216284 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:27.215378 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-93549e-kxtrl"] Oct 30 14:31:27.240256 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:27.240220 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:31:27.240527 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:27.240225 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:31:27.335973 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:27.335940 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe899e87-8056-4fcf-bad4-1cc8665b3706-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-93549e-kxtrl\" (UID: \"fe899e87-8056-4fcf-bad4-1cc8665b3706\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-93549e-kxtrl" Oct 30 14:31:27.336152 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:27.336127 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr2zh\" (UniqueName: \"kubernetes.io/projected/fe899e87-8056-4fcf-bad4-1cc8665b3706-kube-api-access-wr2zh\") pod \"maas-api-revoke-keys-e2e-inf-b-93549e-kxtrl\" (UID: \"fe899e87-8056-4fcf-bad4-1cc8665b3706\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-93549e-kxtrl" Oct 30 14:31:27.336249 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:27.336231 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe899e87-8056-4fcf-bad4-1cc8665b3706-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-93549e-kxtrl\" (UID: \"fe899e87-8056-4fcf-bad4-1cc8665b3706\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-93549e-kxtrl" Oct 30 14:31:27.437075 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:27.437046 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe899e87-8056-4fcf-bad4-1cc8665b3706-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-93549e-kxtrl\" (UID: \"fe899e87-8056-4fcf-bad4-1cc8665b3706\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-93549e-kxtrl" Oct 30 14:31:27.437459 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:27.437136 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr2zh\" (UniqueName: \"kubernetes.io/projected/fe899e87-8056-4fcf-bad4-1cc8665b3706-kube-api-access-wr2zh\") pod \"maas-api-revoke-keys-e2e-inf-b-93549e-kxtrl\" (UID: \"fe899e87-8056-4fcf-bad4-1cc8665b3706\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-93549e-kxtrl" Oct 30 14:31:27.437459 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:27.437174 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe899e87-8056-4fcf-bad4-1cc8665b3706-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-93549e-kxtrl\" (UID: \"fe899e87-8056-4fcf-bad4-1cc8665b3706\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-93549e-kxtrl" Oct 30 14:31:27.437894 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:27.437874 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe899e87-8056-4fcf-bad4-1cc8665b3706-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-93549e-kxtrl\" (UID: \"fe899e87-8056-4fcf-bad4-1cc8665b3706\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-93549e-kxtrl" Oct 30 14:31:27.439432 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:27.439403 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe899e87-8056-4fcf-bad4-1cc8665b3706-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-93549e-kxtrl\" (UID: \"fe899e87-8056-4fcf-bad4-1cc8665b3706\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-93549e-kxtrl" Oct 30 14:31:27.463031 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:27.462972 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr2zh\" (UniqueName: \"kubernetes.io/projected/fe899e87-8056-4fcf-bad4-1cc8665b3706-kube-api-access-wr2zh\") pod \"maas-api-revoke-keys-e2e-inf-b-93549e-kxtrl\" (UID: \"fe899e87-8056-4fcf-bad4-1cc8665b3706\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-93549e-kxtrl" Oct 30 14:31:27.554210 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:27.554183 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-93549e-kxtrl" Oct 30 14:31:27.877084 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:27.877052 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-93549e-kxtrl"] Oct 30 14:31:27.877222 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:31:27.877065 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe899e87_8056_4fcf_bad4_1cc8665b3706.slice/crio-f344db84213bb426db4189c735d79329d602ec0fb6a08e05576f00b29c18fe4c WatchSource:0}: Error finding container f344db84213bb426db4189c735d79329d602ec0fb6a08e05576f00b29c18fe4c: Status 404 returned error can't find the container with id f344db84213bb426db4189c735d79329d602ec0fb6a08e05576f00b29c18fe4c Oct 30 14:31:28.014442 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:31:28.014396 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:31:28.567659 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:28.567608 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-93549e-kxtrl" event={"ID":"fe899e87-8056-4fcf-bad4-1cc8665b3706","Type":"ContainerStarted","Data":"f5fc1d0829dd04a8322f20dc60838701fc37d7e2049329febf466a994eb8f141"} Oct 30 14:31:28.568106 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:28.567696 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-93549e-kxtrl" event={"ID":"fe899e87-8056-4fcf-bad4-1cc8665b3706","Type":"ContainerStarted","Data":"f344db84213bb426db4189c735d79329d602ec0fb6a08e05576f00b29c18fe4c"} Oct 30 14:31:28.615692 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:28.615635 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-93549e-kxtrl" podStartSLOduration=1.615622511 podStartE2EDuration="1.615622511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:31:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:31:28.612050263 +0000 UTC m=+2558.356184303" watchObservedRunningTime="2026-10-30 14:31:28.615622511 +0000 UTC m=+2558.359756551" Oct 30 14:31:29.153732 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:29.153658 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:31:29.580446 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:29.580411 4982 generic.go:356] "Generic (PLEG): container finished" podID="fe899e87-8056-4fcf-bad4-1cc8665b3706" containerID="f5fc1d0829dd04a8322f20dc60838701fc37d7e2049329febf466a994eb8f141" exitCode=0 Oct 30 14:31:29.580937 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:29.580487 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-93549e-kxtrl" event={"ID":"fe899e87-8056-4fcf-bad4-1cc8665b3706","Type":"ContainerDied","Data":"f5fc1d0829dd04a8322f20dc60838701fc37d7e2049329febf466a994eb8f141"} Oct 30 14:31:29.832149 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:29.832059 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:31:29.973620 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:29.973577 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:31:30.456109 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:30.456075 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-409364-58bb5dbb9b-rdzd5"] Oct 30 14:31:30.501539 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:30.501509 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-396d70-86964fb4d6-zmxfm" Oct 30 14:31:30.501946 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:30.501924 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-396d70-56874f4975-w4bw5" Oct 30 14:31:30.627594 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:30.627499 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:31:30.774343 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:30.774312 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:31:30.803319 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:30.803293 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-93549e-kxtrl" Oct 30 14:31:30.891674 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:30.891604 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe899e87-8056-4fcf-bad4-1cc8665b3706-kube-api-access-wr2zh\" (UniqueName: \"kubernetes.io/projected/fe899e87-8056-4fcf-bad4-1cc8665b3706-kube-api-access-wr2zh\") pod \"fe899e87-8056-4fcf-bad4-1cc8665b3706\" (UID: \"fe899e87-8056-4fcf-bad4-1cc8665b3706\") " Oct 30 14:31:30.891874 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:30.891687 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe899e87-8056-4fcf-bad4-1cc8665b3706-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe899e87-8056-4fcf-bad4-1cc8665b3706-maas-api-openshift-ca\") pod \"fe899e87-8056-4fcf-bad4-1cc8665b3706\" (UID: \"fe899e87-8056-4fcf-bad4-1cc8665b3706\") " Oct 30 14:31:30.891955 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:30.891941 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe899e87-8056-4fcf-bad4-1cc8665b3706-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe899e87-8056-4fcf-bad4-1cc8665b3706-maas-api-xks-ca\") pod \"fe899e87-8056-4fcf-bad4-1cc8665b3706\" (UID: \"fe899e87-8056-4fcf-bad4-1cc8665b3706\") " Oct 30 14:31:30.892098 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:30.892070 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe899e87-8056-4fcf-bad4-1cc8665b3706-maas-api-openshift-ca" pod "fe899e87-8056-4fcf-bad4-1cc8665b3706" (UID: "fe899e87-8056-4fcf-bad4-1cc8665b3706"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:31:30.892496 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:30.892472 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe899e87-8056-4fcf-bad4-1cc8665b3706-maas-api-openshift-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:31:30.894034 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:30.894012 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe899e87-8056-4fcf-bad4-1cc8665b3706-maas-api-xks-ca" pod "fe899e87-8056-4fcf-bad4-1cc8665b3706" (UID: "fe899e87-8056-4fcf-bad4-1cc8665b3706"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:31:30.894286 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:30.894260 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe899e87-8056-4fcf-bad4-1cc8665b3706-kube-api-access-wr2zh" pod "fe899e87-8056-4fcf-bad4-1cc8665b3706" (UID: "fe899e87-8056-4fcf-bad4-1cc8665b3706"). InnerVolumeSpecName "kube-api-access-wr2zh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:31:30.993601 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:30.993571 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe899e87-8056-4fcf-bad4-1cc8665b3706-maas-api-xks-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:31:30.993601 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:30.993597 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr2zh\" (UniqueName: \"kubernetes.io/projected/fe899e87-8056-4fcf-bad4-1cc8665b3706-kube-api-access-wr2zh\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:31:31.249318 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:31.249262 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:31:31.601666 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:31.601606 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-93549e-kxtrl" Oct 30 14:31:31.601666 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:31.601634 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-93549e-kxtrl" event={"ID":"fe899e87-8056-4fcf-bad4-1cc8665b3706","Type":"ContainerDied","Data":"f344db84213bb426db4189c735d79329d602ec0fb6a08e05576f00b29c18fe4c"} Oct 30 14:31:31.601871 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:31.601699 4982 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f344db84213bb426db4189c735d79329d602ec0fb6a08e05576f00b29c18fe4c" Oct 30 14:31:33.269539 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:33.269454 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:31:34.528853 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:34.528812 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-b9c4b4/test-model-b9c4b4-kserve-f588646c5-52gfq" podUID="34670e92-5762-42a0-9899-863d0a92a6bf" containerName="main" containerID="cri-o://a67aa1ecab5f2e5b7ccb7e2a5b94a730817f8873f69aa1b68334409ae5447b59" gracePeriod=45 Oct 30 14:31:34.646870 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:34.646831 4982 generic.go:356] "Generic (PLEG): container finished" podID="34670e92-5762-42a0-9899-863d0a92a6bf" containerID="a67aa1ecab5f2e5b7ccb7e2a5b94a730817f8873f69aa1b68334409ae5447b59" exitCode=0 Oct 30 14:31:34.646936 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:34.646899 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b9c4b4/test-model-b9c4b4-kserve-f588646c5-52gfq" event={"ID":"34670e92-5762-42a0-9899-863d0a92a6bf","Type":"ContainerDied","Data":"a67aa1ecab5f2e5b7ccb7e2a5b94a730817f8873f69aa1b68334409ae5447b59"} Oct 30 14:31:34.810849 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:34.810824 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b9c4b4/test-model-b9c4b4-kserve-f588646c5-52gfq" Oct 30 14:31:34.867914 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:34.867883 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34670e92-5762-42a0-9899-863d0a92a6bf-kube-api-access-ttwkz\" (UniqueName: \"kubernetes.io/projected/34670e92-5762-42a0-9899-863d0a92a6bf-kube-api-access-ttwkz\") pod \"34670e92-5762-42a0-9899-863d0a92a6bf\" (UID: \"34670e92-5762-42a0-9899-863d0a92a6bf\") " Oct 30 14:31:34.868081 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:34.867926 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34670e92-5762-42a0-9899-863d0a92a6bf-tls-certs\" (UniqueName: \"kubernetes.io/secret/34670e92-5762-42a0-9899-863d0a92a6bf-tls-certs\") pod \"34670e92-5762-42a0-9899-863d0a92a6bf\" (UID: \"34670e92-5762-42a0-9899-863d0a92a6bf\") " Oct 30 14:31:34.868161 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:34.868139 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34670e92-5762-42a0-9899-863d0a92a6bf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34670e92-5762-42a0-9899-863d0a92a6bf-tmp-dir\") pod \"34670e92-5762-42a0-9899-863d0a92a6bf\" (UID: \"34670e92-5762-42a0-9899-863d0a92a6bf\") " Oct 30 14:31:34.868267 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:34.868247 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34670e92-5762-42a0-9899-863d0a92a6bf-home\" (UniqueName: \"kubernetes.io/empty-dir/34670e92-5762-42a0-9899-863d0a92a6bf-home\") pod \"34670e92-5762-42a0-9899-863d0a92a6bf\" (UID: \"34670e92-5762-42a0-9899-863d0a92a6bf\") " Oct 30 14:31:34.868343 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:34.868301 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34670e92-5762-42a0-9899-863d0a92a6bf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34670e92-5762-42a0-9899-863d0a92a6bf-model-cache\") pod \"34670e92-5762-42a0-9899-863d0a92a6bf\" (UID: \"34670e92-5762-42a0-9899-863d0a92a6bf\") " Oct 30 14:31:34.868583 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:34.868406 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34670e92-5762-42a0-9899-863d0a92a6bf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/34670e92-5762-42a0-9899-863d0a92a6bf-dshm\") pod \"34670e92-5762-42a0-9899-863d0a92a6bf\" (UID: \"34670e92-5762-42a0-9899-863d0a92a6bf\") " Oct 30 14:31:34.868832 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:34.868754 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34670e92-5762-42a0-9899-863d0a92a6bf-home" pod "34670e92-5762-42a0-9899-863d0a92a6bf" (UID: "34670e92-5762-42a0-9899-863d0a92a6bf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:31:34.868832 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:34.868766 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34670e92-5762-42a0-9899-863d0a92a6bf-model-cache" pod "34670e92-5762-42a0-9899-863d0a92a6bf" (UID: "34670e92-5762-42a0-9899-863d0a92a6bf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:31:34.868989 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:34.868866 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34670e92-5762-42a0-9899-863d0a92a6bf-tmp-dir" pod "34670e92-5762-42a0-9899-863d0a92a6bf" (UID: "34670e92-5762-42a0-9899-863d0a92a6bf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:31:34.869347 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:34.869280 4982 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34670e92-5762-42a0-9899-863d0a92a6bf-tmp-dir\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:31:34.869347 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:34.869313 4982 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34670e92-5762-42a0-9899-863d0a92a6bf-home\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:31:34.869347 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:34.869327 4982 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34670e92-5762-42a0-9899-863d0a92a6bf-model-cache\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:31:34.870598 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:34.870573 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34670e92-5762-42a0-9899-863d0a92a6bf-tls-certs" pod "34670e92-5762-42a0-9899-863d0a92a6bf" (UID: "34670e92-5762-42a0-9899-863d0a92a6bf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:31:34.870780 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:34.870758 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34670e92-5762-42a0-9899-863d0a92a6bf-dshm" pod "34670e92-5762-42a0-9899-863d0a92a6bf" (UID: "34670e92-5762-42a0-9899-863d0a92a6bf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:31:34.870878 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:34.870858 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34670e92-5762-42a0-9899-863d0a92a6bf-kube-api-access-ttwkz" pod "34670e92-5762-42a0-9899-863d0a92a6bf" (UID: "34670e92-5762-42a0-9899-863d0a92a6bf"). InnerVolumeSpecName "kube-api-access-ttwkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:31:34.979385 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:34.975457 4982 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34670e92-5762-42a0-9899-863d0a92a6bf-dshm\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:31:34.979385 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:34.975497 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttwkz\" (UniqueName: \"kubernetes.io/projected/34670e92-5762-42a0-9899-863d0a92a6bf-kube-api-access-ttwkz\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:31:34.979385 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:34.975523 4982 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34670e92-5762-42a0-9899-863d0a92a6bf-tls-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:31:35.069615 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:35.069585 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:31:35.456611 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:35.456515 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-409364-58bb5dbb9b-rdzd5" podUID="ef5aa554-9820-4315-9d99-e7143e52f56a" containerName="payload-processing" containerID="cri-o://e657bc211163ef3bb11b85680d1341f9ceb55a0712ed499718b59d0c75767cae" gracePeriod=25 Oct 30 14:31:35.664466 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:35.664426 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b9c4b4/test-model-b9c4b4-kserve-f588646c5-52gfq" event={"ID":"34670e92-5762-42a0-9899-863d0a92a6bf","Type":"ContainerDied","Data":"79b3145ef2b374a5cfefa6fe1390c0ca6c0400f029b71e07f5626734bd09e199"} Oct 30 14:31:35.665011 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:35.664716 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b9c4b4/test-model-b9c4b4-kserve-f588646c5-52gfq" Oct 30 14:31:35.665125 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:35.665104 4982 scope.go:122] "RemoveContainer" containerID="a67aa1ecab5f2e5b7ccb7e2a5b94a730817f8873f69aa1b68334409ae5447b59" Oct 30 14:31:35.689248 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:35.684021 4982 generic.go:356] "Generic (PLEG): container finished" podID="ef5aa554-9820-4315-9d99-e7143e52f56a" containerID="e657bc211163ef3bb11b85680d1341f9ceb55a0712ed499718b59d0c75767cae" exitCode=0 Oct 30 14:31:35.689248 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:35.684098 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-409364-58bb5dbb9b-rdzd5" event={"ID":"ef5aa554-9820-4315-9d99-e7143e52f56a","Type":"ContainerDied","Data":"e657bc211163ef3bb11b85680d1341f9ceb55a0712ed499718b59d0c75767cae"} Oct 30 14:31:35.739066 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:35.739040 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-b9c4b4/test-model-b9c4b4-kserve-f588646c5-52gfq"] Oct 30 14:31:35.753475 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:35.753447 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-b9c4b4/test-model-b9c4b4-kserve-f588646c5-52gfq"] Oct 30 14:31:35.756341 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:35.756315 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-409364-58bb5dbb9b-rdzd5" Oct 30 14:31:35.887534 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:35.887499 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef5aa554-9820-4315-9d99-e7143e52f56a-kube-api-access-ft5ds\" (UniqueName: \"kubernetes.io/projected/ef5aa554-9820-4315-9d99-e7143e52f56a-kube-api-access-ft5ds\") pod \"ef5aa554-9820-4315-9d99-e7143e52f56a\" (UID: \"ef5aa554-9820-4315-9d99-e7143e52f56a\") " Oct 30 14:31:35.887747 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:35.887656 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef5aa554-9820-4315-9d99-e7143e52f56a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef5aa554-9820-4315-9d99-e7143e52f56a-plugins-config-volume\") pod \"ef5aa554-9820-4315-9d99-e7143e52f56a\" (UID: \"ef5aa554-9820-4315-9d99-e7143e52f56a\") " Oct 30 14:31:35.888016 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:35.887973 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef5aa554-9820-4315-9d99-e7143e52f56a-plugins-config-volume" pod "ef5aa554-9820-4315-9d99-e7143e52f56a" (UID: "ef5aa554-9820-4315-9d99-e7143e52f56a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:31:35.888192 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:35.888177 4982 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef5aa554-9820-4315-9d99-e7143e52f56a-plugins-config-volume\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:31:35.889830 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:35.889804 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef5aa554-9820-4315-9d99-e7143e52f56a-kube-api-access-ft5ds" pod "ef5aa554-9820-4315-9d99-e7143e52f56a" (UID: "ef5aa554-9820-4315-9d99-e7143e52f56a"). InnerVolumeSpecName "kube-api-access-ft5ds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:31:35.989790 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:35.989725 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ft5ds\" (UniqueName: \"kubernetes.io/projected/ef5aa554-9820-4315-9d99-e7143e52f56a-kube-api-access-ft5ds\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:31:36.048542 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:36.048504 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00aba928-8v68d"] Oct 30 14:31:36.049519 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:36.049499 4982 cpu_manager.go:414] "containerMap: removing container" podUID="34670e92-5762-42a0-9899-863d0a92a6bf" containerName="main" Oct 30 14:31:36.049565 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:36.049524 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34670e92-5762-42a0-9899-863d0a92a6bf" containerName="main" Oct 30 14:31:36.049597 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:36.049574 4982 cpu_manager.go:414] "containerMap: removing container" podUID="ef5aa554-9820-4315-9d99-e7143e52f56a" containerName="payload-processing" Oct 30 14:31:36.049597 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:36.049582 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef5aa554-9820-4315-9d99-e7143e52f56a" containerName="payload-processing" Oct 30 14:31:36.049689 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:36.049601 4982 cpu_manager.go:414] "containerMap: removing container" podUID="fe899e87-8056-4fcf-bad4-1cc8665b3706" containerName="revoke-keys" Oct 30 14:31:36.049689 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:36.049609 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe899e87-8056-4fcf-bad4-1cc8665b3706" containerName="revoke-keys" Oct 30 14:31:36.049782 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:36.049770 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef5aa554-9820-4315-9d99-e7143e52f56a" containerName="payload-processing" Oct 30 14:31:36.049814 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:36.049788 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe899e87-8056-4fcf-bad4-1cc8665b3706" containerName="revoke-keys" Oct 30 14:31:36.049814 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:36.049805 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="34670e92-5762-42a0-9899-863d0a92a6bf" containerName="main" Oct 30 14:31:36.054161 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:36.054139 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00aba928-8v68d" Oct 30 14:31:36.074100 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:36.074075 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00aba928-8v68d"] Oct 30 14:31:36.075153 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:36.075123 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:31:36.075280 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:36.075182 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:31:36.192278 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:36.192244 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/659a82d6-b679-4925-91d7-d8064ed22314-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00aba928-8v68d\" (UID: \"659a82d6-b679-4925-91d7-d8064ed22314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00aba928-8v68d" Oct 30 14:31:36.192278 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:36.192287 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/659a82d6-b679-4925-91d7-d8064ed22314-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00aba928-8v68d\" (UID: \"659a82d6-b679-4925-91d7-d8064ed22314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00aba928-8v68d" Oct 30 14:31:36.192496 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:36.192408 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2828\" (UniqueName: \"kubernetes.io/projected/659a82d6-b679-4925-91d7-d8064ed22314-kube-api-access-m2828\") pod \"maas-api-revoke-sub-00aba928-8v68d\" (UID: \"659a82d6-b679-4925-91d7-d8064ed22314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00aba928-8v68d" Oct 30 14:31:36.293405 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:36.293313 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2828\" (UniqueName: \"kubernetes.io/projected/659a82d6-b679-4925-91d7-d8064ed22314-kube-api-access-m2828\") pod \"maas-api-revoke-sub-00aba928-8v68d\" (UID: \"659a82d6-b679-4925-91d7-d8064ed22314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00aba928-8v68d" Oct 30 14:31:36.293567 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:36.293488 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/659a82d6-b679-4925-91d7-d8064ed22314-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00aba928-8v68d\" (UID: \"659a82d6-b679-4925-91d7-d8064ed22314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00aba928-8v68d" Oct 30 14:31:36.293567 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:36.293513 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/659a82d6-b679-4925-91d7-d8064ed22314-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00aba928-8v68d\" (UID: \"659a82d6-b679-4925-91d7-d8064ed22314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00aba928-8v68d" Oct 30 14:31:36.294128 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:36.294103 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/659a82d6-b679-4925-91d7-d8064ed22314-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00aba928-8v68d\" (UID: \"659a82d6-b679-4925-91d7-d8064ed22314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00aba928-8v68d" Oct 30 14:31:36.295890 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:36.295864 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/659a82d6-b679-4925-91d7-d8064ed22314-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00aba928-8v68d\" (UID: \"659a82d6-b679-4925-91d7-d8064ed22314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00aba928-8v68d" Oct 30 14:31:36.319926 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:36.319900 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2828\" (UniqueName: \"kubernetes.io/projected/659a82d6-b679-4925-91d7-d8064ed22314-kube-api-access-m2828\") pod \"maas-api-revoke-sub-00aba928-8v68d\" (UID: \"659a82d6-b679-4925-91d7-d8064ed22314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00aba928-8v68d" Oct 30 14:31:36.368443 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:36.368419 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00aba928-8v68d" Oct 30 14:31:36.400916 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:36.400890 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34670e92-5762-42a0-9899-863d0a92a6bf" path="/var/lib/kubelet/pods/34670e92-5762-42a0-9899-863d0a92a6bf/volumes" Oct 30 14:31:36.677871 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:31:36.677813 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod659a82d6_b679_4925_91d7_d8064ed22314.slice/crio-344693f1741e219963088df536babe772fc86443d77241f9755ae4764ce7ebed WatchSource:0}: Error finding container 344693f1741e219963088df536babe772fc86443d77241f9755ae4764ce7ebed: Status 404 returned error can't find the container with id 344693f1741e219963088df536babe772fc86443d77241f9755ae4764ce7ebed Oct 30 14:31:36.679211 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:36.679188 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00aba928-8v68d"] Oct 30 14:31:36.694793 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:36.694760 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00aba928-8v68d" event={"ID":"659a82d6-b679-4925-91d7-d8064ed22314","Type":"ContainerStarted","Data":"344693f1741e219963088df536babe772fc86443d77241f9755ae4764ce7ebed"} Oct 30 14:31:36.696241 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:36.696209 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-409364-58bb5dbb9b-rdzd5" Oct 30 14:31:36.696370 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:36.696240 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-409364-58bb5dbb9b-rdzd5" event={"ID":"ef5aa554-9820-4315-9d99-e7143e52f56a","Type":"ContainerDied","Data":"1e973af9327d74155d6298f21262800e701e23d945596604c459feee71ee58d3"} Oct 30 14:31:36.696370 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:36.696306 4982 scope.go:122] "RemoveContainer" containerID="e657bc211163ef3bb11b85680d1341f9ceb55a0712ed499718b59d0c75767cae" Oct 30 14:31:36.755312 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:36.755275 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-409364-58bb5dbb9b-rdzd5"] Oct 30 14:31:36.768150 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:36.768111 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-409364-58bb5dbb9b-rdzd5"] Oct 30 14:31:36.897209 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:36.897180 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:31:37.621941 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:37.621909 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-93549e-56ff978bf6-qh2c5"] Oct 30 14:31:37.722685 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:37.716040 4982 generic.go:356] "Generic (PLEG): container finished" podID="659a82d6-b679-4925-91d7-d8064ed22314" containerID="9ee010f1c7d06772fd92c3ae361f625711546b32cc3cecc41faf851b2fd9aac2" exitCode=0 Oct 30 14:31:37.722685 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:37.716122 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00aba928-8v68d" event={"ID":"659a82d6-b679-4925-91d7-d8064ed22314","Type":"ContainerDied","Data":"9ee010f1c7d06772fd92c3ae361f625711546b32cc3cecc41faf851b2fd9aac2"} Oct 30 14:31:37.780622 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:37.780569 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:31:37.921248 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:37.921169 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:31:38.402880 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:38.402842 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef5aa554-9820-4315-9d99-e7143e52f56a" path="/var/lib/kubelet/pods/ef5aa554-9820-4315-9d99-e7143e52f56a/volumes" Oct 30 14:31:38.752835 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:38.752788 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:31:38.910254 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:38.910225 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00aba928-8v68d" Oct 30 14:31:39.031784 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:39.031707 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/659a82d6-b679-4925-91d7-d8064ed22314-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/659a82d6-b679-4925-91d7-d8064ed22314-maas-api-xks-ca\") pod \"659a82d6-b679-4925-91d7-d8064ed22314\" (UID: \"659a82d6-b679-4925-91d7-d8064ed22314\") " Oct 30 14:31:39.031784 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:39.031746 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/659a82d6-b679-4925-91d7-d8064ed22314-kube-api-access-m2828\" (UniqueName: \"kubernetes.io/projected/659a82d6-b679-4925-91d7-d8064ed22314-kube-api-access-m2828\") pod \"659a82d6-b679-4925-91d7-d8064ed22314\" (UID: \"659a82d6-b679-4925-91d7-d8064ed22314\") " Oct 30 14:31:39.031784 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:39.031777 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/659a82d6-b679-4925-91d7-d8064ed22314-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/659a82d6-b679-4925-91d7-d8064ed22314-maas-api-openshift-ca\") pod \"659a82d6-b679-4925-91d7-d8064ed22314\" (UID: \"659a82d6-b679-4925-91d7-d8064ed22314\") " Oct 30 14:31:39.032173 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:39.032148 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/659a82d6-b679-4925-91d7-d8064ed22314-maas-api-openshift-ca" pod "659a82d6-b679-4925-91d7-d8064ed22314" (UID: "659a82d6-b679-4925-91d7-d8064ed22314"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:31:39.032389 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:39.032359 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/659a82d6-b679-4925-91d7-d8064ed22314-maas-api-openshift-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:31:39.034037 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:39.034010 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/659a82d6-b679-4925-91d7-d8064ed22314-maas-api-xks-ca" pod "659a82d6-b679-4925-91d7-d8064ed22314" (UID: "659a82d6-b679-4925-91d7-d8064ed22314"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:31:39.034285 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:39.034268 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/659a82d6-b679-4925-91d7-d8064ed22314-kube-api-access-m2828" pod "659a82d6-b679-4925-91d7-d8064ed22314" (UID: "659a82d6-b679-4925-91d7-d8064ed22314"). InnerVolumeSpecName "kube-api-access-m2828". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:31:39.133803 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:39.133770 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/659a82d6-b679-4925-91d7-d8064ed22314-maas-api-xks-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:31:39.133803 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:39.133800 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2828\" (UniqueName: \"kubernetes.io/projected/659a82d6-b679-4925-91d7-d8064ed22314-kube-api-access-m2828\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:31:39.740918 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:39.740887 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00aba928-8v68d" Oct 30 14:31:39.741099 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:39.740896 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00aba928-8v68d" event={"ID":"659a82d6-b679-4925-91d7-d8064ed22314","Type":"ContainerDied","Data":"344693f1741e219963088df536babe772fc86443d77241f9755ae4764ce7ebed"} Oct 30 14:31:39.741099 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:39.741011 4982 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="344693f1741e219963088df536babe772fc86443d77241f9755ae4764ce7ebed" Oct 30 14:31:40.736556 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:40.736515 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:31:42.613899 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:42.613855 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:31:42.623695 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:42.623629 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-93549e-56ff978bf6-qh2c5" podUID="8d73bb58-560d-4c40-aa66-1ad183d82e20" containerName="payload-processing" containerID="cri-o://c512940230e6f18ca851a3cde9666937dc400546657ad985cbdba37e1eb42399" gracePeriod=25 Oct 30 14:31:42.769907 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:42.769877 4982 generic.go:356] "Generic (PLEG): container finished" podID="8d73bb58-560d-4c40-aa66-1ad183d82e20" containerID="c512940230e6f18ca851a3cde9666937dc400546657ad985cbdba37e1eb42399" exitCode=0 Oct 30 14:31:42.770042 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:42.769908 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-93549e-56ff978bf6-qh2c5" event={"ID":"8d73bb58-560d-4c40-aa66-1ad183d82e20","Type":"ContainerDied","Data":"c512940230e6f18ca851a3cde9666937dc400546657ad985cbdba37e1eb42399"} Oct 30 14:31:42.919712 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:42.919685 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-93549e-56ff978bf6-qh2c5" Oct 30 14:31:42.984268 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:42.984025 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d73bb58-560d-4c40-aa66-1ad183d82e20-kube-api-access-6wb7p\" (UniqueName: \"kubernetes.io/projected/8d73bb58-560d-4c40-aa66-1ad183d82e20-kube-api-access-6wb7p\") pod \"8d73bb58-560d-4c40-aa66-1ad183d82e20\" (UID: \"8d73bb58-560d-4c40-aa66-1ad183d82e20\") " Oct 30 14:31:42.984268 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:42.984085 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d73bb58-560d-4c40-aa66-1ad183d82e20-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d73bb58-560d-4c40-aa66-1ad183d82e20-plugins-config-volume\") pod \"8d73bb58-560d-4c40-aa66-1ad183d82e20\" (UID: \"8d73bb58-560d-4c40-aa66-1ad183d82e20\") " Oct 30 14:31:42.984548 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:42.984522 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d73bb58-560d-4c40-aa66-1ad183d82e20-plugins-config-volume" pod "8d73bb58-560d-4c40-aa66-1ad183d82e20" (UID: "8d73bb58-560d-4c40-aa66-1ad183d82e20"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:31:42.985470 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:42.984870 4982 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d73bb58-560d-4c40-aa66-1ad183d82e20-plugins-config-volume\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:31:42.987747 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:42.987719 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d73bb58-560d-4c40-aa66-1ad183d82e20-kube-api-access-6wb7p" pod "8d73bb58-560d-4c40-aa66-1ad183d82e20" (UID: "8d73bb58-560d-4c40-aa66-1ad183d82e20"). InnerVolumeSpecName "kube-api-access-6wb7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:31:43.086230 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:43.086195 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wb7p\" (UniqueName: \"kubernetes.io/projected/8d73bb58-560d-4c40-aa66-1ad183d82e20-kube-api-access-6wb7p\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:31:43.778620 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:43.778588 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-93549e-56ff978bf6-qh2c5" Oct 30 14:31:43.778620 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:43.778601 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-93549e-56ff978bf6-qh2c5" event={"ID":"8d73bb58-560d-4c40-aa66-1ad183d82e20","Type":"ContainerDied","Data":"2787b1ab5c3f605612f100f588355664682b499b1580ae737b1468c6cca91c59"} Oct 30 14:31:43.779157 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:43.778668 4982 scope.go:122] "RemoveContainer" containerID="c512940230e6f18ca851a3cde9666937dc400546657ad985cbdba37e1eb42399" Oct 30 14:31:43.842162 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:43.842127 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-93549e-56ff978bf6-qh2c5"] Oct 30 14:31:43.855194 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:43.855166 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-93549e-56ff978bf6-qh2c5"] Oct 30 14:31:44.057004 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:44.056974 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-93549e-65ffc5485c-wqqd4"] Oct 30 14:31:44.057229 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:44.057203 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-93549e-65ffc5485c-wqqd4" podUID="f3ef6149-023a-44f7-a1d7-ac0acd7de745" containerName="maas-api" containerID="cri-o://a5b64e11026c8677183555d580dd1cbe7180e04ddb14a0cb8d7034df3880fb69" gracePeriod=30 Oct 30 14:31:44.157937 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:44.157896 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:31:44.370204 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:44.370176 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-93549e-65ffc5485c-wqqd4" Oct 30 14:31:44.373320 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:44.373283 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:31:44.402654 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:44.402617 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d73bb58-560d-4c40-aa66-1ad183d82e20" path="/var/lib/kubelet/pods/8d73bb58-560d-4c40-aa66-1ad183d82e20/volumes" Oct 30 14:31:44.516835 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:44.516783 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3ef6149-023a-44f7-a1d7-ac0acd7de745-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3ef6149-023a-44f7-a1d7-ac0acd7de745-metrics-certs\") pod \"f3ef6149-023a-44f7-a1d7-ac0acd7de745\" (UID: \"f3ef6149-023a-44f7-a1d7-ac0acd7de745\") " Oct 30 14:31:44.517048 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:44.516918 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3ef6149-023a-44f7-a1d7-ac0acd7de745-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3ef6149-023a-44f7-a1d7-ac0acd7de745-maas-api-tls\") pod \"f3ef6149-023a-44f7-a1d7-ac0acd7de745\" (UID: \"f3ef6149-023a-44f7-a1d7-ac0acd7de745\") " Oct 30 14:31:44.517048 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:44.516957 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3ef6149-023a-44f7-a1d7-ac0acd7de745-kube-api-access-wzc5h\" (UniqueName: \"kubernetes.io/projected/f3ef6149-023a-44f7-a1d7-ac0acd7de745-kube-api-access-wzc5h\") pod \"f3ef6149-023a-44f7-a1d7-ac0acd7de745\" (UID: \"f3ef6149-023a-44f7-a1d7-ac0acd7de745\") " Oct 30 14:31:44.517048 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:44.517013 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3ef6149-023a-44f7-a1d7-ac0acd7de745-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3ef6149-023a-44f7-a1d7-ac0acd7de745-openshift-service-ca\") pod \"f3ef6149-023a-44f7-a1d7-ac0acd7de745\" (UID: \"f3ef6149-023a-44f7-a1d7-ac0acd7de745\") " Oct 30 14:31:44.518078 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:44.518049 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3ef6149-023a-44f7-a1d7-ac0acd7de745-openshift-service-ca" pod "f3ef6149-023a-44f7-a1d7-ac0acd7de745" (UID: "f3ef6149-023a-44f7-a1d7-ac0acd7de745"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:31:44.520707 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:44.519608 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3ef6149-023a-44f7-a1d7-ac0acd7de745-maas-api-tls" pod "f3ef6149-023a-44f7-a1d7-ac0acd7de745" (UID: "f3ef6149-023a-44f7-a1d7-ac0acd7de745"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:31:44.520707 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:44.519953 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3ef6149-023a-44f7-a1d7-ac0acd7de745-metrics-certs" pod "f3ef6149-023a-44f7-a1d7-ac0acd7de745" (UID: "f3ef6149-023a-44f7-a1d7-ac0acd7de745"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:31:44.520707 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:44.520254 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3ef6149-023a-44f7-a1d7-ac0acd7de745-kube-api-access-wzc5h" pod "f3ef6149-023a-44f7-a1d7-ac0acd7de745" (UID: "f3ef6149-023a-44f7-a1d7-ac0acd7de745"). InnerVolumeSpecName "kube-api-access-wzc5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:31:44.623862 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:44.623714 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3ef6149-023a-44f7-a1d7-ac0acd7de745-maas-api-tls\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:31:44.623862 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:44.623747 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzc5h\" (UniqueName: \"kubernetes.io/projected/f3ef6149-023a-44f7-a1d7-ac0acd7de745-kube-api-access-wzc5h\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:31:44.623862 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:44.623763 4982 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3ef6149-023a-44f7-a1d7-ac0acd7de745-openshift-service-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:31:44.623862 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:44.623777 4982 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3ef6149-023a-44f7-a1d7-ac0acd7de745-metrics-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:31:44.713673 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:44.713425 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:31:44.794445 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:44.794380 4982 generic.go:356] "Generic (PLEG): container finished" podID="f3ef6149-023a-44f7-a1d7-ac0acd7de745" containerID="a5b64e11026c8677183555d580dd1cbe7180e04ddb14a0cb8d7034df3880fb69" exitCode=0 Oct 30 14:31:44.795204 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:44.795178 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-93549e-65ffc5485c-wqqd4" event={"ID":"f3ef6149-023a-44f7-a1d7-ac0acd7de745","Type":"ContainerDied","Data":"a5b64e11026c8677183555d580dd1cbe7180e04ddb14a0cb8d7034df3880fb69"} Oct 30 14:31:44.795364 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:44.795348 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-93549e-65ffc5485c-wqqd4" event={"ID":"f3ef6149-023a-44f7-a1d7-ac0acd7de745","Type":"ContainerDied","Data":"2900a84fe2192570e18275d72a75957c015f8623709bb2ee87cea26147544497"} Oct 30 14:31:44.795470 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:44.795457 4982 scope.go:122] "RemoveContainer" containerID="a5b64e11026c8677183555d580dd1cbe7180e04ddb14a0cb8d7034df3880fb69" Oct 30 14:31:44.795715 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:44.795697 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-93549e-65ffc5485c-wqqd4" Oct 30 14:31:44.818696 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:44.818626 4982 scope.go:122] "RemoveContainer" containerID="a5b64e11026c8677183555d580dd1cbe7180e04ddb14a0cb8d7034df3880fb69" Oct 30 14:31:44.818997 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:31:44.818964 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5b64e11026c8677183555d580dd1cbe7180e04ddb14a0cb8d7034df3880fb69\": container with ID starting with a5b64e11026c8677183555d580dd1cbe7180e04ddb14a0cb8d7034df3880fb69 not found: ID does not exist" containerID="a5b64e11026c8677183555d580dd1cbe7180e04ddb14a0cb8d7034df3880fb69" Oct 30 14:31:44.819115 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:44.819005 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5b64e11026c8677183555d580dd1cbe7180e04ddb14a0cb8d7034df3880fb69"} err="failed to get container status \"a5b64e11026c8677183555d580dd1cbe7180e04ddb14a0cb8d7034df3880fb69\": rpc error: code = NotFound desc = could not find container \"a5b64e11026c8677183555d580dd1cbe7180e04ddb14a0cb8d7034df3880fb69\": container with ID starting with a5b64e11026c8677183555d580dd1cbe7180e04ddb14a0cb8d7034df3880fb69 not found: ID does not exist" Oct 30 14:31:44.852727 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:44.852699 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-93549e-kxtrl"] Oct 30 14:31:44.856504 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:44.856373 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-93549e-kxtrl"] Oct 30 14:31:44.866657 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:44.864434 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-93549e-65ffc5485c-wqqd4"] Oct 30 14:31:44.867222 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:44.867132 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:31:44.869120 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:44.869097 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-93549e-65ffc5485c-wqqd4"] Oct 30 14:31:45.376169 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:45.376138 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:31:45.525081 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:45.525042 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:31:46.402245 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:46.402212 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3ef6149-023a-44f7-a1d7-ac0acd7de745" path="/var/lib/kubelet/pods/f3ef6149-023a-44f7-a1d7-ac0acd7de745/volumes" Oct 30 14:31:46.402763 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:46.402665 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe899e87-8056-4fcf-bad4-1cc8665b3706" path="/var/lib/kubelet/pods/fe899e87-8056-4fcf-bad4-1cc8665b3706/volumes" Oct 30 14:31:46.788501 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:46.788413 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:31:48.635257 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:48.635221 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:31:50.432516 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:50.432479 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:31:51.447789 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:51.447750 4982 scope.go:122] "RemoveContainer" containerID="d07ee4ced56696441a46a377d411b21ff950ca3710aeea3786850c25463b0d7f" Oct 30 14:31:51.458072 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:51.458041 4982 scope.go:122] "RemoveContainer" containerID="c2c12278bc9458bfed71997cac45483f53606815bd4729e933eedec7e73994ac" Oct 30 14:31:52.411855 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:52.411814 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:31:52.560608 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:52.560573 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:31:52.773292 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:52.773187 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:31:54.200759 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:54.200717 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:31:54.344626 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:54.344589 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:31:54.705036 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:54.705000 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd6b9bd9-xljkb"] Oct 30 14:31:54.705714 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:54.705692 4982 cpu_manager.go:414] "containerMap: removing container" podUID="8d73bb58-560d-4c40-aa66-1ad183d82e20" containerName="payload-processing" Oct 30 14:31:54.705714 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:54.705713 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d73bb58-560d-4c40-aa66-1ad183d82e20" containerName="payload-processing" Oct 30 14:31:54.705893 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:54.705732 4982 cpu_manager.go:414] "containerMap: removing container" podUID="f3ef6149-023a-44f7-a1d7-ac0acd7de745" containerName="maas-api" Oct 30 14:31:54.705893 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:54.705739 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3ef6149-023a-44f7-a1d7-ac0acd7de745" containerName="maas-api" Oct 30 14:31:54.705893 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:54.705801 4982 cpu_manager.go:414] "containerMap: removing container" podUID="659a82d6-b679-4925-91d7-d8064ed22314" containerName="revoke-keys" Oct 30 14:31:54.705893 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:54.705809 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="659a82d6-b679-4925-91d7-d8064ed22314" containerName="revoke-keys" Oct 30 14:31:54.706089 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:54.705987 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3ef6149-023a-44f7-a1d7-ac0acd7de745" containerName="maas-api" Oct 30 14:31:54.706089 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:54.706012 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="659a82d6-b679-4925-91d7-d8064ed22314" containerName="revoke-keys" Oct 30 14:31:54.706089 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:54.706057 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d73bb58-560d-4c40-aa66-1ad183d82e20" containerName="payload-processing" Oct 30 14:31:54.711478 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:54.711459 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6b9bd9-xljkb" Oct 30 14:31:54.730165 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:54.730141 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd6b9bd9-xljkb"] Oct 30 14:31:54.738702 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:54.738678 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:31:54.738848 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:54.738735 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:31:54.817367 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:54.817333 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64fe4a3b-0e15-47e6-a017-7f43db151a0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd6b9bd9-xljkb\" (UID: \"64fe4a3b-0e15-47e6-a017-7f43db151a0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6b9bd9-xljkb" Oct 30 14:31:54.817512 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:54.817488 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxrkx\" (UniqueName: \"kubernetes.io/projected/64fe4a3b-0e15-47e6-a017-7f43db151a0d-kube-api-access-pxrkx\") pod \"maas-api-revoke-sub-cd6b9bd9-xljkb\" (UID: \"64fe4a3b-0e15-47e6-a017-7f43db151a0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6b9bd9-xljkb" Oct 30 14:31:54.817556 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:54.817536 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64fe4a3b-0e15-47e6-a017-7f43db151a0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd6b9bd9-xljkb\" (UID: \"64fe4a3b-0e15-47e6-a017-7f43db151a0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6b9bd9-xljkb" Oct 30 14:31:54.919142 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:54.919089 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxrkx\" (UniqueName: \"kubernetes.io/projected/64fe4a3b-0e15-47e6-a017-7f43db151a0d-kube-api-access-pxrkx\") pod \"maas-api-revoke-sub-cd6b9bd9-xljkb\" (UID: \"64fe4a3b-0e15-47e6-a017-7f43db151a0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6b9bd9-xljkb" Oct 30 14:31:54.919337 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:54.919177 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64fe4a3b-0e15-47e6-a017-7f43db151a0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd6b9bd9-xljkb\" (UID: \"64fe4a3b-0e15-47e6-a017-7f43db151a0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6b9bd9-xljkb" Oct 30 14:31:54.919413 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:54.919380 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64fe4a3b-0e15-47e6-a017-7f43db151a0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd6b9bd9-xljkb\" (UID: \"64fe4a3b-0e15-47e6-a017-7f43db151a0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6b9bd9-xljkb" Oct 30 14:31:54.920030 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:54.920005 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64fe4a3b-0e15-47e6-a017-7f43db151a0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd6b9bd9-xljkb\" (UID: \"64fe4a3b-0e15-47e6-a017-7f43db151a0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6b9bd9-xljkb" Oct 30 14:31:54.921624 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:54.921595 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64fe4a3b-0e15-47e6-a017-7f43db151a0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd6b9bd9-xljkb\" (UID: \"64fe4a3b-0e15-47e6-a017-7f43db151a0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6b9bd9-xljkb" Oct 30 14:31:54.945298 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:54.945271 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxrkx\" (UniqueName: \"kubernetes.io/projected/64fe4a3b-0e15-47e6-a017-7f43db151a0d-kube-api-access-pxrkx\") pod \"maas-api-revoke-sub-cd6b9bd9-xljkb\" (UID: \"64fe4a3b-0e15-47e6-a017-7f43db151a0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6b9bd9-xljkb" Oct 30 14:31:55.026216 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:55.026135 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6b9bd9-xljkb" Oct 30 14:31:55.348792 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:31:55.348755 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64fe4a3b_0e15_47e6_a017_7f43db151a0d.slice/crio-31c67f0b60b2fb2f1ddc7e9b64e3b6d8f97602d0cb242e3d4b97ced5a7fa9177 WatchSource:0}: Error finding container 31c67f0b60b2fb2f1ddc7e9b64e3b6d8f97602d0cb242e3d4b97ced5a7fa9177: Status 404 returned error can't find the container with id 31c67f0b60b2fb2f1ddc7e9b64e3b6d8f97602d0cb242e3d4b97ced5a7fa9177 Oct 30 14:31:55.349444 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:55.349415 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd6b9bd9-xljkb"] Oct 30 14:31:55.947004 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:55.946925 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6b9bd9-xljkb" event={"ID":"64fe4a3b-0e15-47e6-a017-7f43db151a0d","Type":"ContainerStarted","Data":"51231bc9fc393839d061100ca4affb0e6f7e10185949ad657a05b441cda32c83"} Oct 30 14:31:55.947004 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:55.946993 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6b9bd9-xljkb" event={"ID":"64fe4a3b-0e15-47e6-a017-7f43db151a0d","Type":"ContainerStarted","Data":"31c67f0b60b2fb2f1ddc7e9b64e3b6d8f97602d0cb242e3d4b97ced5a7fa9177"} Oct 30 14:31:56.005160 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:56.005105 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6b9bd9-xljkb" podStartSLOduration=2.00509196 podStartE2EDuration="2.00509196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:31:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:31:55.992224747 +0000 UTC m=+2585.736358788" watchObservedRunningTime="2026-10-30 14:31:56.00509196 +0000 UTC m=+2585.749226081" Oct 30 14:31:56.067961 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:56.067928 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:31:56.956261 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:56.956228 4982 generic.go:356] "Generic (PLEG): container finished" podID="64fe4a3b-0e15-47e6-a017-7f43db151a0d" containerID="51231bc9fc393839d061100ca4affb0e6f7e10185949ad657a05b441cda32c83" exitCode=0 Oct 30 14:31:56.956711 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:56.956298 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6b9bd9-xljkb" event={"ID":"64fe4a3b-0e15-47e6-a017-7f43db151a0d","Type":"ContainerDied","Data":"51231bc9fc393839d061100ca4affb0e6f7e10185949ad657a05b441cda32c83"} Oct 30 14:31:57.920676 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:57.920324 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-e371c3/auth-test-model-e371c3-kserve-866c468ff7-2bf7k"] Oct 30 14:31:57.942027 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:57.942001 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-e371c3/auth-test-model-e371c3-kserve-866c468ff7-2bf7k" Oct 30 14:31:57.961924 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:57.961890 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-e371c3\"/\"default-dockercfg-gwfbq\"" Oct 30 14:31:57.980694 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:57.975252 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-e371c3\"/\"auth-test-model-e371c3-kserve-self-signed-certs\"" Oct 30 14:31:57.980694 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:57.978779 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-e371c3\"/\"openshift-service-ca.crt\"" Oct 30 14:31:57.980694 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:57.979395 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-e371c3\"/\"kube-root-ca.crt\"" Oct 30 14:31:57.986579 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:57.986555 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-e371c3/auth-test-model-e371c3-kserve-866c468ff7-2bf7k"] Oct 30 14:31:58.031959 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:58.031879 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:31:58.047240 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:31:58.047162 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:31:58.073748 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:58.073711 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3770ada4-af76-45ff-8782-f84ea8f5cc26-tmp-dir\") pod \"auth-test-model-e371c3-kserve-866c468ff7-2bf7k\" (UID: \"3770ada4-af76-45ff-8782-f84ea8f5cc26\") " pod="ai-tenant-e2e-shared-a-w4-e371c3/auth-test-model-e371c3-kserve-866c468ff7-2bf7k" Oct 30 14:31:58.074008 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:58.073788 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3770ada4-af76-45ff-8782-f84ea8f5cc26-model-cache\") pod \"auth-test-model-e371c3-kserve-866c468ff7-2bf7k\" (UID: \"3770ada4-af76-45ff-8782-f84ea8f5cc26\") " pod="ai-tenant-e2e-shared-a-w4-e371c3/auth-test-model-e371c3-kserve-866c468ff7-2bf7k" Oct 30 14:31:58.074008 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:58.073824 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3770ada4-af76-45ff-8782-f84ea8f5cc26-home\") pod \"auth-test-model-e371c3-kserve-866c468ff7-2bf7k\" (UID: \"3770ada4-af76-45ff-8782-f84ea8f5cc26\") " pod="ai-tenant-e2e-shared-a-w4-e371c3/auth-test-model-e371c3-kserve-866c468ff7-2bf7k" Oct 30 14:31:58.074008 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:58.073965 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjml6\" (UniqueName: \"kubernetes.io/projected/3770ada4-af76-45ff-8782-f84ea8f5cc26-kube-api-access-gjml6\") pod \"auth-test-model-e371c3-kserve-866c468ff7-2bf7k\" (UID: \"3770ada4-af76-45ff-8782-f84ea8f5cc26\") " pod="ai-tenant-e2e-shared-a-w4-e371c3/auth-test-model-e371c3-kserve-866c468ff7-2bf7k" Oct 30 14:31:58.074008 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:58.074011 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3770ada4-af76-45ff-8782-f84ea8f5cc26-tls-certs\") pod \"auth-test-model-e371c3-kserve-866c468ff7-2bf7k\" (UID: \"3770ada4-af76-45ff-8782-f84ea8f5cc26\") " pod="ai-tenant-e2e-shared-a-w4-e371c3/auth-test-model-e371c3-kserve-866c468ff7-2bf7k" Oct 30 14:31:58.074392 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:58.074048 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3770ada4-af76-45ff-8782-f84ea8f5cc26-dshm\") pod \"auth-test-model-e371c3-kserve-866c468ff7-2bf7k\" (UID: \"3770ada4-af76-45ff-8782-f84ea8f5cc26\") " pod="ai-tenant-e2e-shared-a-w4-e371c3/auth-test-model-e371c3-kserve-866c468ff7-2bf7k" Oct 30 14:31:58.175684 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:58.175571 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3770ada4-af76-45ff-8782-f84ea8f5cc26-model-cache\") pod \"auth-test-model-e371c3-kserve-866c468ff7-2bf7k\" (UID: \"3770ada4-af76-45ff-8782-f84ea8f5cc26\") " pod="ai-tenant-e2e-shared-a-w4-e371c3/auth-test-model-e371c3-kserve-866c468ff7-2bf7k" Oct 30 14:31:58.175947 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:58.175922 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3770ada4-af76-45ff-8782-f84ea8f5cc26-home\") pod \"auth-test-model-e371c3-kserve-866c468ff7-2bf7k\" (UID: \"3770ada4-af76-45ff-8782-f84ea8f5cc26\") " pod="ai-tenant-e2e-shared-a-w4-e371c3/auth-test-model-e371c3-kserve-866c468ff7-2bf7k" Oct 30 14:31:58.176207 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:58.176188 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjml6\" (UniqueName: \"kubernetes.io/projected/3770ada4-af76-45ff-8782-f84ea8f5cc26-kube-api-access-gjml6\") pod \"auth-test-model-e371c3-kserve-866c468ff7-2bf7k\" (UID: \"3770ada4-af76-45ff-8782-f84ea8f5cc26\") " pod="ai-tenant-e2e-shared-a-w4-e371c3/auth-test-model-e371c3-kserve-866c468ff7-2bf7k" Oct 30 14:31:58.176350 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:58.176326 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3770ada4-af76-45ff-8782-f84ea8f5cc26-tls-certs\") pod \"auth-test-model-e371c3-kserve-866c468ff7-2bf7k\" (UID: \"3770ada4-af76-45ff-8782-f84ea8f5cc26\") " pod="ai-tenant-e2e-shared-a-w4-e371c3/auth-test-model-e371c3-kserve-866c468ff7-2bf7k" Oct 30 14:31:58.176470 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:58.176457 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3770ada4-af76-45ff-8782-f84ea8f5cc26-dshm\") pod \"auth-test-model-e371c3-kserve-866c468ff7-2bf7k\" (UID: \"3770ada4-af76-45ff-8782-f84ea8f5cc26\") " pod="ai-tenant-e2e-shared-a-w4-e371c3/auth-test-model-e371c3-kserve-866c468ff7-2bf7k" Oct 30 14:31:58.176731 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:58.176711 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3770ada4-af76-45ff-8782-f84ea8f5cc26-tmp-dir\") pod \"auth-test-model-e371c3-kserve-866c468ff7-2bf7k\" (UID: \"3770ada4-af76-45ff-8782-f84ea8f5cc26\") " pod="ai-tenant-e2e-shared-a-w4-e371c3/auth-test-model-e371c3-kserve-866c468ff7-2bf7k" Oct 30 14:31:58.178660 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:58.177262 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3770ada4-af76-45ff-8782-f84ea8f5cc26-tmp-dir\") pod \"auth-test-model-e371c3-kserve-866c468ff7-2bf7k\" (UID: \"3770ada4-af76-45ff-8782-f84ea8f5cc26\") " pod="ai-tenant-e2e-shared-a-w4-e371c3/auth-test-model-e371c3-kserve-866c468ff7-2bf7k" Oct 30 14:31:58.188710 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:58.179339 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3770ada4-af76-45ff-8782-f84ea8f5cc26-model-cache\") pod \"auth-test-model-e371c3-kserve-866c468ff7-2bf7k\" (UID: \"3770ada4-af76-45ff-8782-f84ea8f5cc26\") " pod="ai-tenant-e2e-shared-a-w4-e371c3/auth-test-model-e371c3-kserve-866c468ff7-2bf7k" Oct 30 14:31:58.188710 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:58.179552 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3770ada4-af76-45ff-8782-f84ea8f5cc26-home\") pod \"auth-test-model-e371c3-kserve-866c468ff7-2bf7k\" (UID: \"3770ada4-af76-45ff-8782-f84ea8f5cc26\") " pod="ai-tenant-e2e-shared-a-w4-e371c3/auth-test-model-e371c3-kserve-866c468ff7-2bf7k" Oct 30 14:31:58.226915 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:58.226875 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:31:58.230831 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:58.230776 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjml6\" (UniqueName: \"kubernetes.io/projected/3770ada4-af76-45ff-8782-f84ea8f5cc26-kube-api-access-gjml6\") pod \"auth-test-model-e371c3-kserve-866c468ff7-2bf7k\" (UID: \"3770ada4-af76-45ff-8782-f84ea8f5cc26\") " pod="ai-tenant-e2e-shared-a-w4-e371c3/auth-test-model-e371c3-kserve-866c468ff7-2bf7k" Oct 30 14:31:58.296509 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:58.296480 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3770ada4-af76-45ff-8782-f84ea8f5cc26-dshm\") pod \"auth-test-model-e371c3-kserve-866c468ff7-2bf7k\" (UID: \"3770ada4-af76-45ff-8782-f84ea8f5cc26\") " pod="ai-tenant-e2e-shared-a-w4-e371c3/auth-test-model-e371c3-kserve-866c468ff7-2bf7k" Oct 30 14:31:58.298968 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:58.298944 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3770ada4-af76-45ff-8782-f84ea8f5cc26-tls-certs\") pod \"auth-test-model-e371c3-kserve-866c468ff7-2bf7k\" (UID: \"3770ada4-af76-45ff-8782-f84ea8f5cc26\") " pod="ai-tenant-e2e-shared-a-w4-e371c3/auth-test-model-e371c3-kserve-866c468ff7-2bf7k" Oct 30 14:31:58.336930 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:58.336904 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6b9bd9-xljkb" Oct 30 14:31:58.490259 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:58.484968 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64fe4a3b-0e15-47e6-a017-7f43db151a0d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64fe4a3b-0e15-47e6-a017-7f43db151a0d-maas-api-xks-ca\") pod \"64fe4a3b-0e15-47e6-a017-7f43db151a0d\" (UID: \"64fe4a3b-0e15-47e6-a017-7f43db151a0d\") " Oct 30 14:31:58.490259 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:58.485049 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64fe4a3b-0e15-47e6-a017-7f43db151a0d-kube-api-access-pxrkx\" (UniqueName: \"kubernetes.io/projected/64fe4a3b-0e15-47e6-a017-7f43db151a0d-kube-api-access-pxrkx\") pod \"64fe4a3b-0e15-47e6-a017-7f43db151a0d\" (UID: \"64fe4a3b-0e15-47e6-a017-7f43db151a0d\") " Oct 30 14:31:58.490259 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:58.485215 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64fe4a3b-0e15-47e6-a017-7f43db151a0d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64fe4a3b-0e15-47e6-a017-7f43db151a0d-maas-api-openshift-ca\") pod \"64fe4a3b-0e15-47e6-a017-7f43db151a0d\" (UID: \"64fe4a3b-0e15-47e6-a017-7f43db151a0d\") " Oct 30 14:31:58.490259 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:58.485846 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64fe4a3b-0e15-47e6-a017-7f43db151a0d-maas-api-openshift-ca" pod "64fe4a3b-0e15-47e6-a017-7f43db151a0d" (UID: "64fe4a3b-0e15-47e6-a017-7f43db151a0d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:31:58.490259 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:58.488446 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64fe4a3b-0e15-47e6-a017-7f43db151a0d-maas-api-xks-ca" pod "64fe4a3b-0e15-47e6-a017-7f43db151a0d" (UID: "64fe4a3b-0e15-47e6-a017-7f43db151a0d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:31:58.490259 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:58.488581 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64fe4a3b-0e15-47e6-a017-7f43db151a0d-kube-api-access-pxrkx" pod "64fe4a3b-0e15-47e6-a017-7f43db151a0d" (UID: "64fe4a3b-0e15-47e6-a017-7f43db151a0d"). InnerVolumeSpecName "kube-api-access-pxrkx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:31:58.559399 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:58.559362 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-e371c3/auth-test-model-e371c3-kserve-866c468ff7-2bf7k" Oct 30 14:31:58.589861 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:58.589837 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64fe4a3b-0e15-47e6-a017-7f43db151a0d-maas-api-xks-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:31:58.590020 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:58.589866 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxrkx\" (UniqueName: \"kubernetes.io/projected/64fe4a3b-0e15-47e6-a017-7f43db151a0d-kube-api-access-pxrkx\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:31:58.590020 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:58.589882 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64fe4a3b-0e15-47e6-a017-7f43db151a0d-maas-api-openshift-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:31:58.876289 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:31:58.876234 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3770ada4_af76_45ff_8782_f84ea8f5cc26.slice/crio-ba617501a24ac1c56067439dac3f97b377ffb58c580984fde866ab8cfdfb5aa1 WatchSource:0}: Error finding container ba617501a24ac1c56067439dac3f97b377ffb58c580984fde866ab8cfdfb5aa1: Status 404 returned error can't find the container with id ba617501a24ac1c56067439dac3f97b377ffb58c580984fde866ab8cfdfb5aa1 Oct 30 14:31:58.877499 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:58.877467 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-e371c3/auth-test-model-e371c3-kserve-866c468ff7-2bf7k"] Oct 30 14:31:58.972936 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:58.972909 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6b9bd9-xljkb" Oct 30 14:31:58.973319 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:58.972932 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6b9bd9-xljkb" event={"ID":"64fe4a3b-0e15-47e6-a017-7f43db151a0d","Type":"ContainerDied","Data":"31c67f0b60b2fb2f1ddc7e9b64e3b6d8f97602d0cb242e3d4b97ced5a7fa9177"} Oct 30 14:31:58.973319 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:58.972994 4982 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31c67f0b60b2fb2f1ddc7e9b64e3b6d8f97602d0cb242e3d4b97ced5a7fa9177" Oct 30 14:31:58.974911 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:58.974885 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-e371c3/auth-test-model-e371c3-kserve-866c468ff7-2bf7k" event={"ID":"3770ada4-af76-45ff-8782-f84ea8f5cc26","Type":"ContainerStarted","Data":"69d64e4971db8430a6e9cfd63f7dbaeb401d86bbc14c5214a25eba9900a92089"} Oct 30 14:31:58.975030 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:58.974918 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-e371c3/auth-test-model-e371c3-kserve-866c468ff7-2bf7k" event={"ID":"3770ada4-af76-45ff-8782-f84ea8f5cc26","Type":"ContainerStarted","Data":"ba617501a24ac1c56067439dac3f97b377ffb58c580984fde866ab8cfdfb5aa1"} Oct 30 14:31:59.023073 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:59.023022 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-e371c3/auth-test-model-e371c3-kserve-866c468ff7-2bf7k" podStartSLOduration=2.023009425 podStartE2EDuration="2.023009425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:31:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:31:59.020566231 +0000 UTC m=+2588.764700273" watchObservedRunningTime="2026-10-30 14:31:59.023009425 +0000 UTC m=+2588.767143465" Oct 30 14:31:59.356193 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:59.356160 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z"] Oct 30 14:31:59.356890 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:59.356867 4982 cpu_manager.go:414] "containerMap: removing container" podUID="64fe4a3b-0e15-47e6-a017-7f43db151a0d" containerName="revoke-keys" Oct 30 14:31:59.356890 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:59.356887 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64fe4a3b-0e15-47e6-a017-7f43db151a0d" containerName="revoke-keys" Oct 30 14:31:59.357082 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:59.357006 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="64fe4a3b-0e15-47e6-a017-7f43db151a0d" containerName="revoke-keys" Oct 30 14:31:59.360684 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:59.360663 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z" Oct 30 14:31:59.385175 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:59.385150 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-1d3696-openshift-default-dockercfg-5g6bv\"" Oct 30 14:31:59.385686 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:59.385607 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z"] Oct 30 14:31:59.416227 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:59.416189 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33aa1ab8-e5ee-416a-beec-abd4714629e8-istio-envoy\") pod \"e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z\" (UID: \"33aa1ab8-e5ee-416a-beec-abd4714629e8\") " pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z" Oct 30 14:31:59.416397 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:59.416298 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33aa1ab8-e5ee-416a-beec-abd4714629e8-istio-data\") pod \"e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z\" (UID: \"33aa1ab8-e5ee-416a-beec-abd4714629e8\") " pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z" Oct 30 14:31:59.416397 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:59.416345 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33aa1ab8-e5ee-416a-beec-abd4714629e8-workload-socket\") pod \"e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z\" (UID: \"33aa1ab8-e5ee-416a-beec-abd4714629e8\") " pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z" Oct 30 14:31:59.416397 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:59.416376 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33aa1ab8-e5ee-416a-beec-abd4714629e8-workload-certs\") pod \"e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z\" (UID: \"33aa1ab8-e5ee-416a-beec-abd4714629e8\") " pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z" Oct 30 14:31:59.416571 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:59.416495 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33aa1ab8-e5ee-416a-beec-abd4714629e8-istio-token\") pod \"e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z\" (UID: \"33aa1ab8-e5ee-416a-beec-abd4714629e8\") " pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z" Oct 30 14:31:59.416680 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:59.416567 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33aa1ab8-e5ee-416a-beec-abd4714629e8-credential-socket\") pod \"e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z\" (UID: \"33aa1ab8-e5ee-416a-beec-abd4714629e8\") " pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z" Oct 30 14:31:59.416680 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:59.416601 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33aa1ab8-e5ee-416a-beec-abd4714629e8-istiod-ca-cert\") pod \"e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z\" (UID: \"33aa1ab8-e5ee-416a-beec-abd4714629e8\") " pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z" Oct 30 14:31:59.416680 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:59.416653 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsddv\" (UniqueName: \"kubernetes.io/projected/33aa1ab8-e5ee-416a-beec-abd4714629e8-kube-api-access-hsddv\") pod \"e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z\" (UID: \"33aa1ab8-e5ee-416a-beec-abd4714629e8\") " pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z" Oct 30 14:31:59.416841 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:59.416777 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33aa1ab8-e5ee-416a-beec-abd4714629e8-istio-podinfo\") pod \"e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z\" (UID: \"33aa1ab8-e5ee-416a-beec-abd4714629e8\") " pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z" Oct 30 14:31:59.518242 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:59.518199 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33aa1ab8-e5ee-416a-beec-abd4714629e8-istio-podinfo\") pod \"e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z\" (UID: \"33aa1ab8-e5ee-416a-beec-abd4714629e8\") " pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z" Oct 30 14:31:59.518430 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:59.518366 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33aa1ab8-e5ee-416a-beec-abd4714629e8-istio-envoy\") pod \"e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z\" (UID: \"33aa1ab8-e5ee-416a-beec-abd4714629e8\") " pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z" Oct 30 14:31:59.518891 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:59.518840 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33aa1ab8-e5ee-416a-beec-abd4714629e8-istio-data\") pod \"e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z\" (UID: \"33aa1ab8-e5ee-416a-beec-abd4714629e8\") " pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z" Oct 30 14:31:59.519094 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:59.518472 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33aa1ab8-e5ee-416a-beec-abd4714629e8-istio-data\") pod \"e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z\" (UID: \"33aa1ab8-e5ee-416a-beec-abd4714629e8\") " pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z" Oct 30 14:31:59.519245 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:59.519226 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33aa1ab8-e5ee-416a-beec-abd4714629e8-workload-socket\") pod \"e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z\" (UID: \"33aa1ab8-e5ee-416a-beec-abd4714629e8\") " pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z" Oct 30 14:31:59.519384 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:59.519367 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33aa1ab8-e5ee-416a-beec-abd4714629e8-workload-certs\") pod \"e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z\" (UID: \"33aa1ab8-e5ee-416a-beec-abd4714629e8\") " pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z" Oct 30 14:31:59.519578 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:59.519553 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33aa1ab8-e5ee-416a-beec-abd4714629e8-workload-socket\") pod \"e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z\" (UID: \"33aa1ab8-e5ee-416a-beec-abd4714629e8\") " pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z" Oct 30 14:31:59.519717 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:59.519573 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33aa1ab8-e5ee-416a-beec-abd4714629e8-istio-token\") pod \"e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z\" (UID: \"33aa1ab8-e5ee-416a-beec-abd4714629e8\") " pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z" Oct 30 14:31:59.519866 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:59.519842 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33aa1ab8-e5ee-416a-beec-abd4714629e8-workload-certs\") pod \"e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z\" (UID: \"33aa1ab8-e5ee-416a-beec-abd4714629e8\") " pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z" Oct 30 14:31:59.519955 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:59.519858 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33aa1ab8-e5ee-416a-beec-abd4714629e8-credential-socket\") pod \"e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z\" (UID: \"33aa1ab8-e5ee-416a-beec-abd4714629e8\") " pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z" Oct 30 14:31:59.520070 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:59.520054 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33aa1ab8-e5ee-416a-beec-abd4714629e8-istiod-ca-cert\") pod \"e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z\" (UID: \"33aa1ab8-e5ee-416a-beec-abd4714629e8\") " pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z" Oct 30 14:31:59.520205 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:59.520189 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsddv\" (UniqueName: \"kubernetes.io/projected/33aa1ab8-e5ee-416a-beec-abd4714629e8-kube-api-access-hsddv\") pod \"e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z\" (UID: \"33aa1ab8-e5ee-416a-beec-abd4714629e8\") " pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z" Oct 30 14:31:59.520446 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:59.520202 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33aa1ab8-e5ee-416a-beec-abd4714629e8-credential-socket\") pod \"e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z\" (UID: \"33aa1ab8-e5ee-416a-beec-abd4714629e8\") " pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z" Oct 30 14:31:59.521554 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:59.520851 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33aa1ab8-e5ee-416a-beec-abd4714629e8-istiod-ca-cert\") pod \"e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z\" (UID: \"33aa1ab8-e5ee-416a-beec-abd4714629e8\") " pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z" Oct 30 14:31:59.521554 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:59.521285 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33aa1ab8-e5ee-416a-beec-abd4714629e8-istio-podinfo\") pod \"e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z\" (UID: \"33aa1ab8-e5ee-416a-beec-abd4714629e8\") " pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z" Oct 30 14:31:59.522037 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:59.522006 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33aa1ab8-e5ee-416a-beec-abd4714629e8-istio-envoy\") pod \"e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z\" (UID: \"33aa1ab8-e5ee-416a-beec-abd4714629e8\") " pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z" Oct 30 14:31:59.545477 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:59.545448 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33aa1ab8-e5ee-416a-beec-abd4714629e8-istio-token\") pod \"e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z\" (UID: \"33aa1ab8-e5ee-416a-beec-abd4714629e8\") " pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z" Oct 30 14:31:59.546177 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:59.546155 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsddv\" (UniqueName: \"kubernetes.io/projected/33aa1ab8-e5ee-416a-beec-abd4714629e8-kube-api-access-hsddv\") pod \"e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z\" (UID: \"33aa1ab8-e5ee-416a-beec-abd4714629e8\") " pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z" Oct 30 14:31:59.559709 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:59.559691 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-e371c3/auth-test-model-e371c3-kserve-866c468ff7-2bf7k" Oct 30 14:31:59.678212 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:59.678149 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z" Oct 30 14:31:59.957602 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:31:59.957530 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:32:00.008540 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:32:00.008510 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33aa1ab8_e5ee_416a_beec_abd4714629e8.slice/crio-f9bc5fbcb9ce0a9cd1cb0d9f60cc20e9b30956166afeed65fafc2bb3c6e0738c WatchSource:0}: Error finding container f9bc5fbcb9ce0a9cd1cb0d9f60cc20e9b30956166afeed65fafc2bb3c6e0738c: Status 404 returned error can't find the container with id f9bc5fbcb9ce0a9cd1cb0d9f60cc20e9b30956166afeed65fafc2bb3c6e0738c Oct 30 14:32:00.009078 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:00.009054 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z"] Oct 30 14:32:00.011389 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:00.011360 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:32:00.011475 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:00.011422 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:32:00.011475 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:00.011453 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:32:01.015609 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:01.015569 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z" event={"ID":"33aa1ab8-e5ee-416a-beec-abd4714629e8","Type":"ContainerStarted","Data":"6df1a589e0dbc8ee3ebd0663fd0c30e518afd48d062e115e05d332aa24387046"} Oct 30 14:32:01.016045 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:01.015635 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z" event={"ID":"33aa1ab8-e5ee-416a-beec-abd4714629e8","Type":"ContainerStarted","Data":"f9bc5fbcb9ce0a9cd1cb0d9f60cc20e9b30956166afeed65fafc2bb3c6e0738c"} Oct 30 14:32:01.083665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:01.080947 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z" podStartSLOduration=2.080927641 podStartE2EDuration="2.080927641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:31:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:32:01.077139152 +0000 UTC m=+2590.821273192" watchObservedRunningTime="2026-10-30 14:32:01.080927641 +0000 UTC m=+2590.825061684" Oct 30 14:32:01.678442 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:01.678411 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z" Oct 30 14:32:01.713200 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:01.713162 4982 patch_prober.go:29] interesting pod/e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Oct 30 14:32:01.713548 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:01.713513 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z" podUID="33aa1ab8-e5ee-416a-beec-abd4714629e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Oct 30 14:32:01.722410 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:01.722377 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:32:01.872575 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:01.872530 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:32:02.678670 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:02.678613 4982 patch_prober.go:29] interesting pod/e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Oct 30 14:32:02.679157 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:02.678695 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z" podUID="33aa1ab8-e5ee-416a-beec-abd4714629e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Oct 30 14:32:03.679392 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:03.679356 4982 patch_prober.go:29] interesting pod/e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Oct 30 14:32:03.679900 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:03.679413 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z" podUID="33aa1ab8-e5ee-416a-beec-abd4714629e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Oct 30 14:32:03.835985 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:03.835940 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:32:04.678878 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:04.678788 4982 patch_prober.go:29] interesting pod/e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Oct 30 14:32:04.678878 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:04.678842 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z" podUID="33aa1ab8-e5ee-416a-beec-abd4714629e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Oct 30 14:32:05.417086 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:05.417048 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:32:05.682624 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:05.682052 4982 patch_prober.go:29] interesting pod/e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Oct 30 14:32:05.682624 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:05.682129 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z" podUID="33aa1ab8-e5ee-416a-beec-abd4714629e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Oct 30 14:32:05.951459 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:05.951012 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:32:06.103110 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:06.103071 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:32:06.592100 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:06.592062 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:32:06.684226 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:06.684191 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z" Oct 30 14:32:06.684670 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:06.684651 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z" Oct 30 14:32:06.685414 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:06.685397 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z" Oct 30 14:32:08.091208 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:08.091170 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:32:08.559634 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:08.559607 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-e371c3/auth-test-model-e371c3-kserve-866c468ff7-2bf7k" Oct 30 14:32:08.572001 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:08.571977 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-e371c3/auth-test-model-e371c3-kserve-866c468ff7-2bf7k" Oct 30 14:32:09.166156 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:09.166130 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-e371c3/auth-test-model-e371c3-kserve-866c468ff7-2bf7k" Oct 30 14:32:09.868027 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:09.867992 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:32:11.820700 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:11.820633 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:32:13.156105 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:13.156068 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:32:13.304745 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:13.304703 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:32:13.623509 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:13.623476 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:32:15.200737 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:15.200697 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:32:15.755486 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:15.754930 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:32:15.883779 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:15.883746 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-34bcc5-fbwcx"] Oct 30 14:32:15.921304 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:15.921226 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:32:15.928697 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:15.925731 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-34bcc5-fbwcx"] Oct 30 14:32:16.386143 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:16.386097 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:32:16.402461 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:16.402419 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f304f8d-a25b-4dfb-a750-3d8efc5c44fc" path="/var/lib/kubelet/pods/7f304f8d-a25b-4dfb-a750-3d8efc5c44fc/volumes" Oct 30 14:32:17.647940 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:17.647907 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:32:17.813797 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:17.813763 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f3c580-l4kxb"] Oct 30 14:32:17.822936 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:17.822910 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f3c580-l4kxb" Oct 30 14:32:17.827438 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:17.827414 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f3c580-l4kxb"] Oct 30 14:32:17.840363 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:17.840341 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:32:17.852691 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:17.852666 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:32:17.859685 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:17.859632 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:32:17.966065 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:17.965972 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a58b83db-877e-47c3-9108-cfa4bbb67028-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f3c580-l4kxb\" (UID: \"a58b83db-877e-47c3-9108-cfa4bbb67028\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f3c580-l4kxb" Oct 30 14:32:17.966236 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:17.966182 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfvcb\" (UniqueName: \"kubernetes.io/projected/a58b83db-877e-47c3-9108-cfa4bbb67028-kube-api-access-qfvcb\") pod \"maas-api-revoke-keys-e2e-inf-a-f3c580-l4kxb\" (UID: \"a58b83db-877e-47c3-9108-cfa4bbb67028\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f3c580-l4kxb" Oct 30 14:32:17.966362 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:17.966345 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a58b83db-877e-47c3-9108-cfa4bbb67028-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f3c580-l4kxb\" (UID: \"a58b83db-877e-47c3-9108-cfa4bbb67028\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f3c580-l4kxb" Oct 30 14:32:18.067844 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:18.067809 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfvcb\" (UniqueName: \"kubernetes.io/projected/a58b83db-877e-47c3-9108-cfa4bbb67028-kube-api-access-qfvcb\") pod \"maas-api-revoke-keys-e2e-inf-a-f3c580-l4kxb\" (UID: \"a58b83db-877e-47c3-9108-cfa4bbb67028\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f3c580-l4kxb" Oct 30 14:32:18.068063 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:18.067897 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a58b83db-877e-47c3-9108-cfa4bbb67028-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f3c580-l4kxb\" (UID: \"a58b83db-877e-47c3-9108-cfa4bbb67028\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f3c580-l4kxb" Oct 30 14:32:18.068063 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:18.067976 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a58b83db-877e-47c3-9108-cfa4bbb67028-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f3c580-l4kxb\" (UID: \"a58b83db-877e-47c3-9108-cfa4bbb67028\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f3c580-l4kxb" Oct 30 14:32:18.068562 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:18.068528 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a58b83db-877e-47c3-9108-cfa4bbb67028-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f3c580-l4kxb\" (UID: \"a58b83db-877e-47c3-9108-cfa4bbb67028\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f3c580-l4kxb" Oct 30 14:32:18.070474 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:18.070451 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a58b83db-877e-47c3-9108-cfa4bbb67028-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f3c580-l4kxb\" (UID: \"a58b83db-877e-47c3-9108-cfa4bbb67028\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f3c580-l4kxb" Oct 30 14:32:18.094415 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:18.094384 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfvcb\" (UniqueName: \"kubernetes.io/projected/a58b83db-877e-47c3-9108-cfa4bbb67028-kube-api-access-qfvcb\") pod \"maas-api-revoke-keys-e2e-inf-a-f3c580-l4kxb\" (UID: \"a58b83db-877e-47c3-9108-cfa4bbb67028\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f3c580-l4kxb" Oct 30 14:32:18.146838 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:18.146809 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f3c580-l4kxb" Oct 30 14:32:18.465898 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:32:18.465865 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda58b83db_877e_47c3_9108_cfa4bbb67028.slice/crio-800c073b632053104dfebb2afe074d8df7948104c7bb2af04dcc9db8aff7ad45 WatchSource:0}: Error finding container 800c073b632053104dfebb2afe074d8df7948104c7bb2af04dcc9db8aff7ad45: Status 404 returned error can't find the container with id 800c073b632053104dfebb2afe074d8df7948104c7bb2af04dcc9db8aff7ad45 Oct 30 14:32:18.466291 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:18.466239 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f3c580-l4kxb"] Oct 30 14:32:19.261691 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:19.261571 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f3c580-l4kxb" event={"ID":"a58b83db-877e-47c3-9108-cfa4bbb67028","Type":"ContainerStarted","Data":"885d3f04c7e01e2465afbcab73aca890bdda620a71669063acd87a64fcd8693a"} Oct 30 14:32:19.261691 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:19.261624 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f3c580-l4kxb" event={"ID":"a58b83db-877e-47c3-9108-cfa4bbb67028","Type":"ContainerStarted","Data":"800c073b632053104dfebb2afe074d8df7948104c7bb2af04dcc9db8aff7ad45"} Oct 30 14:32:19.307276 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:19.307230 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f3c580-l4kxb" podStartSLOduration=2.307216666 podStartE2EDuration="2.307216666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:32:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:32:19.304873835 +0000 UTC m=+2609.049007877" watchObservedRunningTime="2026-10-30 14:32:19.307216666 +0000 UTC m=+2609.051350706" Oct 30 14:32:19.765385 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:19.765309 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:32:20.270122 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:20.270092 4982 generic.go:356] "Generic (PLEG): container finished" podID="a58b83db-877e-47c3-9108-cfa4bbb67028" containerID="885d3f04c7e01e2465afbcab73aca890bdda620a71669063acd87a64fcd8693a" exitCode=0 Oct 30 14:32:20.270517 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:20.270176 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f3c580-l4kxb" event={"ID":"a58b83db-877e-47c3-9108-cfa4bbb67028","Type":"ContainerDied","Data":"885d3f04c7e01e2465afbcab73aca890bdda620a71669063acd87a64fcd8693a"} Oct 30 14:32:21.444849 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:21.444823 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f3c580-l4kxb" Oct 30 14:32:21.516228 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:21.515131 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a58b83db-877e-47c3-9108-cfa4bbb67028-kube-api-access-qfvcb\" (UniqueName: \"kubernetes.io/projected/a58b83db-877e-47c3-9108-cfa4bbb67028-kube-api-access-qfvcb\") pod \"a58b83db-877e-47c3-9108-cfa4bbb67028\" (UID: \"a58b83db-877e-47c3-9108-cfa4bbb67028\") " Oct 30 14:32:21.516228 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:21.515341 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a58b83db-877e-47c3-9108-cfa4bbb67028-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a58b83db-877e-47c3-9108-cfa4bbb67028-maas-api-xks-ca\") pod \"a58b83db-877e-47c3-9108-cfa4bbb67028\" (UID: \"a58b83db-877e-47c3-9108-cfa4bbb67028\") " Oct 30 14:32:21.516228 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:21.515430 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a58b83db-877e-47c3-9108-cfa4bbb67028-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a58b83db-877e-47c3-9108-cfa4bbb67028-maas-api-openshift-ca\") pod \"a58b83db-877e-47c3-9108-cfa4bbb67028\" (UID: \"a58b83db-877e-47c3-9108-cfa4bbb67028\") " Oct 30 14:32:21.516803 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:21.516761 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a58b83db-877e-47c3-9108-cfa4bbb67028-maas-api-openshift-ca" pod "a58b83db-877e-47c3-9108-cfa4bbb67028" (UID: "a58b83db-877e-47c3-9108-cfa4bbb67028"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:32:21.519159 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:21.519124 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a58b83db-877e-47c3-9108-cfa4bbb67028-kube-api-access-qfvcb" pod "a58b83db-877e-47c3-9108-cfa4bbb67028" (UID: "a58b83db-877e-47c3-9108-cfa4bbb67028"). InnerVolumeSpecName "kube-api-access-qfvcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:32:21.519892 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:21.519866 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a58b83db-877e-47c3-9108-cfa4bbb67028-maas-api-xks-ca" pod "a58b83db-877e-47c3-9108-cfa4bbb67028" (UID: "a58b83db-877e-47c3-9108-cfa4bbb67028"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:32:21.617757 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:21.617723 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a58b83db-877e-47c3-9108-cfa4bbb67028-maas-api-xks-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:32:21.617757 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:21.617750 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a58b83db-877e-47c3-9108-cfa4bbb67028-maas-api-openshift-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:32:21.617757 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:21.617759 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfvcb\" (UniqueName: \"kubernetes.io/projected/a58b83db-877e-47c3-9108-cfa4bbb67028-kube-api-access-qfvcb\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:32:21.639080 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:21.639052 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:32:22.286130 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:22.286099 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f3c580-l4kxb" Oct 30 14:32:22.286326 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:22.286134 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f3c580-l4kxb" event={"ID":"a58b83db-877e-47c3-9108-cfa4bbb67028","Type":"ContainerDied","Data":"800c073b632053104dfebb2afe074d8df7948104c7bb2af04dcc9db8aff7ad45"} Oct 30 14:32:22.286326 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:22.286182 4982 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="800c073b632053104dfebb2afe074d8df7948104c7bb2af04dcc9db8aff7ad45" Oct 30 14:32:23.279273 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:23.279239 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:32:23.427561 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:23.427523 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:32:23.567139 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:23.567103 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:32:24.636249 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:24.636207 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:32:25.117472 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:25.117432 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fde75d-9hjg8"] Oct 30 14:32:25.118110 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:25.118091 4982 cpu_manager.go:414] "containerMap: removing container" podUID="a58b83db-877e-47c3-9108-cfa4bbb67028" containerName="revoke-keys" Oct 30 14:32:25.118193 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:25.118110 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a58b83db-877e-47c3-9108-cfa4bbb67028" containerName="revoke-keys" Oct 30 14:32:25.118250 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:25.118208 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="a58b83db-877e-47c3-9108-cfa4bbb67028" containerName="revoke-keys" Oct 30 14:32:25.122019 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:25.121998 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fde75d-9hjg8" Oct 30 14:32:25.143737 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:25.143710 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fde75d-9hjg8"] Oct 30 14:32:25.151172 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:25.151146 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:32:25.151366 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:25.151175 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:32:25.169377 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:25.169344 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx6vj\" (UniqueName: \"kubernetes.io/projected/2b7670b5-5942-49ab-b6ae-7200698f575a-kube-api-access-kx6vj\") pod \"maas-api-revoke-keys-e2e-inf-b-fde75d-9hjg8\" (UID: \"2b7670b5-5942-49ab-b6ae-7200698f575a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fde75d-9hjg8" Oct 30 14:32:25.169692 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:25.169617 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b7670b5-5942-49ab-b6ae-7200698f575a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-fde75d-9hjg8\" (UID: \"2b7670b5-5942-49ab-b6ae-7200698f575a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fde75d-9hjg8" Oct 30 14:32:25.169859 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:25.169837 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b7670b5-5942-49ab-b6ae-7200698f575a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-fde75d-9hjg8\" (UID: \"2b7670b5-5942-49ab-b6ae-7200698f575a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fde75d-9hjg8" Oct 30 14:32:25.271956 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:25.271917 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b7670b5-5942-49ab-b6ae-7200698f575a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-fde75d-9hjg8\" (UID: \"2b7670b5-5942-49ab-b6ae-7200698f575a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fde75d-9hjg8" Oct 30 14:32:25.273158 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:25.273132 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx6vj\" (UniqueName: \"kubernetes.io/projected/2b7670b5-5942-49ab-b6ae-7200698f575a-kube-api-access-kx6vj\") pod \"maas-api-revoke-keys-e2e-inf-b-fde75d-9hjg8\" (UID: \"2b7670b5-5942-49ab-b6ae-7200698f575a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fde75d-9hjg8" Oct 30 14:32:25.273732 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:25.273706 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b7670b5-5942-49ab-b6ae-7200698f575a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-fde75d-9hjg8\" (UID: \"2b7670b5-5942-49ab-b6ae-7200698f575a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fde75d-9hjg8" Oct 30 14:32:25.283946 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:25.272954 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b7670b5-5942-49ab-b6ae-7200698f575a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-fde75d-9hjg8\" (UID: \"2b7670b5-5942-49ab-b6ae-7200698f575a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fde75d-9hjg8" Oct 30 14:32:25.299922 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:25.299895 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx6vj\" (UniqueName: \"kubernetes.io/projected/2b7670b5-5942-49ab-b6ae-7200698f575a-kube-api-access-kx6vj\") pod \"maas-api-revoke-keys-e2e-inf-b-fde75d-9hjg8\" (UID: \"2b7670b5-5942-49ab-b6ae-7200698f575a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fde75d-9hjg8" Oct 30 14:32:25.312458 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:25.312433 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b7670b5-5942-49ab-b6ae-7200698f575a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-fde75d-9hjg8\" (UID: \"2b7670b5-5942-49ab-b6ae-7200698f575a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fde75d-9hjg8" Oct 30 14:32:25.438122 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:25.438037 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fde75d-9hjg8" Oct 30 14:32:25.491487 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:25.491446 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:32:25.650199 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:25.650158 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:32:25.773469 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:32:25.773427 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b7670b5_5942_49ab_b6ae_7200698f575a.slice/crio-01e43a8622f6227bc09a64c5d51d74a7f27dc70958daac13e540d6f0d0df7187 WatchSource:0}: Error finding container 01e43a8622f6227bc09a64c5d51d74a7f27dc70958daac13e540d6f0d0df7187: Status 404 returned error can't find the container with id 01e43a8622f6227bc09a64c5d51d74a7f27dc70958daac13e540d6f0d0df7187 Oct 30 14:32:25.774419 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:25.774393 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fde75d-9hjg8"] Oct 30 14:32:26.375714 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:26.374505 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fde75d-9hjg8" event={"ID":"2b7670b5-5942-49ab-b6ae-7200698f575a","Type":"ContainerStarted","Data":"38f6732e14a1b4e5c8aee94df9a6cf7581b0153baee176dd320c2a411b9a96bd"} Oct 30 14:32:26.375714 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:26.374562 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fde75d-9hjg8" event={"ID":"2b7670b5-5942-49ab-b6ae-7200698f575a","Type":"ContainerStarted","Data":"01e43a8622f6227bc09a64c5d51d74a7f27dc70958daac13e540d6f0d0df7187"} Oct 30 14:32:27.432623 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:27.432583 4982 generic.go:356] "Generic (PLEG): container finished" podID="2b7670b5-5942-49ab-b6ae-7200698f575a" containerID="38f6732e14a1b4e5c8aee94df9a6cf7581b0153baee176dd320c2a411b9a96bd" exitCode=0 Oct 30 14:32:27.433147 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:27.432737 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fde75d-9hjg8" event={"ID":"2b7670b5-5942-49ab-b6ae-7200698f575a","Type":"ContainerDied","Data":"38f6732e14a1b4e5c8aee94df9a6cf7581b0153baee176dd320c2a411b9a96bd"} Oct 30 14:32:27.739980 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:27.739856 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:32:28.048880 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:32:28.048792 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:32:28.222583 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:28.222547 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f3c580-6d587b47f8-zthxw"] Oct 30 14:32:28.258433 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:28.258395 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f3c580-7bff98f799-c6d7q"] Oct 30 14:32:28.430044 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:28.429980 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:32:28.621100 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:28.621040 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:32:28.820142 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:28.820114 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fde75d-9hjg8" Oct 30 14:32:28.969664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:28.966394 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b7670b5-5942-49ab-b6ae-7200698f575a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b7670b5-5942-49ab-b6ae-7200698f575a-maas-api-xks-ca\") pod \"2b7670b5-5942-49ab-b6ae-7200698f575a\" (UID: \"2b7670b5-5942-49ab-b6ae-7200698f575a\") " Oct 30 14:32:28.969664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:28.966589 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b7670b5-5942-49ab-b6ae-7200698f575a-kube-api-access-kx6vj\" (UniqueName: \"kubernetes.io/projected/2b7670b5-5942-49ab-b6ae-7200698f575a-kube-api-access-kx6vj\") pod \"2b7670b5-5942-49ab-b6ae-7200698f575a\" (UID: \"2b7670b5-5942-49ab-b6ae-7200698f575a\") " Oct 30 14:32:28.969664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:28.966674 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b7670b5-5942-49ab-b6ae-7200698f575a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b7670b5-5942-49ab-b6ae-7200698f575a-maas-api-openshift-ca\") pod \"2b7670b5-5942-49ab-b6ae-7200698f575a\" (UID: \"2b7670b5-5942-49ab-b6ae-7200698f575a\") " Oct 30 14:32:28.969664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:28.967605 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b7670b5-5942-49ab-b6ae-7200698f575a-maas-api-openshift-ca" pod "2b7670b5-5942-49ab-b6ae-7200698f575a" (UID: "2b7670b5-5942-49ab-b6ae-7200698f575a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:32:29.014905 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:29.014794 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b7670b5-5942-49ab-b6ae-7200698f575a-kube-api-access-kx6vj" pod "2b7670b5-5942-49ab-b6ae-7200698f575a" (UID: "2b7670b5-5942-49ab-b6ae-7200698f575a"). InnerVolumeSpecName "kube-api-access-kx6vj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:32:29.045004 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:29.044946 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b7670b5-5942-49ab-b6ae-7200698f575a-maas-api-xks-ca" pod "2b7670b5-5942-49ab-b6ae-7200698f575a" (UID: "2b7670b5-5942-49ab-b6ae-7200698f575a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:32:29.076754 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:29.076545 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b7670b5-5942-49ab-b6ae-7200698f575a-maas-api-xks-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:32:29.077026 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:29.077002 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kx6vj\" (UniqueName: \"kubernetes.io/projected/2b7670b5-5942-49ab-b6ae-7200698f575a-kube-api-access-kx6vj\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:32:29.077139 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:29.077125 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b7670b5-5942-49ab-b6ae-7200698f575a-maas-api-openshift-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:32:29.470795 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:29.470764 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fde75d-9hjg8" Oct 30 14:32:29.471003 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:29.470799 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fde75d-9hjg8" event={"ID":"2b7670b5-5942-49ab-b6ae-7200698f575a","Type":"ContainerDied","Data":"01e43a8622f6227bc09a64c5d51d74a7f27dc70958daac13e540d6f0d0df7187"} Oct 30 14:32:29.471003 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:29.470853 4982 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01e43a8622f6227bc09a64c5d51d74a7f27dc70958daac13e540d6f0d0df7187" Oct 30 14:32:29.624228 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:29.624187 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:32:31.438833 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:31.438801 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:32:33.223966 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:33.223923 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-f3c580-6d587b47f8-zthxw" podUID="522b3312-a38d-4253-998d-226d3bb2ae62" containerName="payload-processing" containerID="cri-o://55cacd2b97162e09df600d8d9bd9fe12e9403dab34763ede166dfbec82c36827" gracePeriod=25 Oct 30 14:32:33.259103 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:33.259047 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3c580-7bff98f799-c6d7q" podUID="a2fc2354-7034-40dd-9086-ef67d615ec96" containerName="payload-pre-processing" containerID="cri-o://d6aa943ac56d42cbd55cfa730e0da33f617931fcf5a68a1d5a8f69ea68526731" gracePeriod=25 Oct 30 14:32:33.330033 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:33.329995 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:32:33.506915 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:33.506832 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:32:34.021163 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:34.021125 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b9c4b4-qn6wb"] Oct 30 14:32:34.021794 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:34.021770 4982 cpu_manager.go:414] "containerMap: removing container" podUID="2b7670b5-5942-49ab-b6ae-7200698f575a" containerName="revoke-keys" Oct 30 14:32:34.021794 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:34.021785 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b7670b5-5942-49ab-b6ae-7200698f575a" containerName="revoke-keys" Oct 30 14:32:34.021979 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:34.021917 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b7670b5-5942-49ab-b6ae-7200698f575a" containerName="revoke-keys" Oct 30 14:32:34.026835 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:34.026809 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b9c4b4-qn6wb" Oct 30 14:32:34.046664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:34.046616 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b9c4b4-qn6wb"] Oct 30 14:32:34.055395 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:34.055368 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:32:34.055761 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:34.055700 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:32:34.106288 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:34.106237 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f68fg\" (UniqueName: \"kubernetes.io/projected/9d729832-e94c-4dc1-95cb-b67bbe754469-kube-api-access-f68fg\") pod \"maas-api-revoke-keys-e2e-inf-b-b9c4b4-qn6wb\" (UID: \"9d729832-e94c-4dc1-95cb-b67bbe754469\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b9c4b4-qn6wb" Oct 30 14:32:34.106462 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:34.106381 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d729832-e94c-4dc1-95cb-b67bbe754469-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b9c4b4-qn6wb\" (UID: \"9d729832-e94c-4dc1-95cb-b67bbe754469\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b9c4b4-qn6wb" Oct 30 14:32:34.106539 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:34.106497 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d729832-e94c-4dc1-95cb-b67bbe754469-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b9c4b4-qn6wb\" (UID: \"9d729832-e94c-4dc1-95cb-b67bbe754469\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b9c4b4-qn6wb" Oct 30 14:32:34.165001 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:34.164957 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3c580-7bff98f799-c6d7q" podUID="a2fc2354-7034-40dd-9086-ef67d615ec96" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:32:34.165174 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:34.165028 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-f3c580-6d587b47f8-zthxw" podUID="522b3312-a38d-4253-998d-226d3bb2ae62" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:32:34.207925 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:34.207888 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f68fg\" (UniqueName: \"kubernetes.io/projected/9d729832-e94c-4dc1-95cb-b67bbe754469-kube-api-access-f68fg\") pod \"maas-api-revoke-keys-e2e-inf-b-b9c4b4-qn6wb\" (UID: \"9d729832-e94c-4dc1-95cb-b67bbe754469\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b9c4b4-qn6wb" Oct 30 14:32:34.208103 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:34.207963 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d729832-e94c-4dc1-95cb-b67bbe754469-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b9c4b4-qn6wb\" (UID: \"9d729832-e94c-4dc1-95cb-b67bbe754469\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b9c4b4-qn6wb" Oct 30 14:32:34.208103 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:34.208047 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d729832-e94c-4dc1-95cb-b67bbe754469-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b9c4b4-qn6wb\" (UID: \"9d729832-e94c-4dc1-95cb-b67bbe754469\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b9c4b4-qn6wb" Oct 30 14:32:34.208800 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:34.208765 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d729832-e94c-4dc1-95cb-b67bbe754469-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b9c4b4-qn6wb\" (UID: \"9d729832-e94c-4dc1-95cb-b67bbe754469\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b9c4b4-qn6wb" Oct 30 14:32:34.210483 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:34.210456 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d729832-e94c-4dc1-95cb-b67bbe754469-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b9c4b4-qn6wb\" (UID: \"9d729832-e94c-4dc1-95cb-b67bbe754469\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b9c4b4-qn6wb" Oct 30 14:32:34.233720 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:34.233686 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f68fg\" (UniqueName: \"kubernetes.io/projected/9d729832-e94c-4dc1-95cb-b67bbe754469-kube-api-access-f68fg\") pod \"maas-api-revoke-keys-e2e-inf-b-b9c4b4-qn6wb\" (UID: \"9d729832-e94c-4dc1-95cb-b67bbe754469\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b9c4b4-qn6wb" Oct 30 14:32:34.302923 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:34.302793 4982 patch_prober.go:29] interesting pod/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:32:34.302923 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:34.302871 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj" podUID="2de9f651-0083-42cd-89b2-6d28d35bb449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:32:34.342772 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:34.342738 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b9c4b4-qn6wb" Oct 30 14:32:34.659415 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:32:34.659379 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d729832_e94c_4dc1_95cb_b67bbe754469.slice/crio-26f0223cc947e8088967683d7c32e3d7b74fc1fec5195cf361fa24690c30b0fd WatchSource:0}: Error finding container 26f0223cc947e8088967683d7c32e3d7b74fc1fec5195cf361fa24690c30b0fd: Status 404 returned error can't find the container with id 26f0223cc947e8088967683d7c32e3d7b74fc1fec5195cf361fa24690c30b0fd Oct 30 14:32:34.662606 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:34.662572 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b9c4b4-qn6wb"] Oct 30 14:32:35.396166 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:35.396084 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:32:35.607622 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:35.607582 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fde75d-7d8d99ffcb-79hw8"] Oct 30 14:32:35.619782 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:35.619738 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b9c4b4-qn6wb" event={"ID":"9d729832-e94c-4dc1-95cb-b67bbe754469","Type":"ContainerStarted","Data":"9d53a51b7e0da7bd7e9529f05caaecebad2daed53dd87b10e5b0be9178678b6a"} Oct 30 14:32:35.619966 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:35.619794 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b9c4b4-qn6wb" event={"ID":"9d729832-e94c-4dc1-95cb-b67bbe754469","Type":"ContainerStarted","Data":"26f0223cc947e8088967683d7c32e3d7b74fc1fec5195cf361fa24690c30b0fd"} Oct 30 14:32:35.680448 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:35.680360 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fde75d-75cd788695-5n8hq"] Oct 30 14:32:35.796980 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:35.796919 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b9c4b4-qn6wb" podStartSLOduration=2.79690289 podStartE2EDuration="2.79690289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:32:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:32:35.791720398 +0000 UTC m=+2625.535854443" watchObservedRunningTime="2026-10-30 14:32:35.79690289 +0000 UTC m=+2625.541036982" Oct 30 14:32:35.834608 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:35.834556 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:32:35.991520 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:35.991414 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:32:36.631736 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:36.631702 4982 generic.go:356] "Generic (PLEG): container finished" podID="9d729832-e94c-4dc1-95cb-b67bbe754469" containerID="9d53a51b7e0da7bd7e9529f05caaecebad2daed53dd87b10e5b0be9178678b6a" exitCode=0 Oct 30 14:32:36.632112 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:36.631784 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b9c4b4-qn6wb" event={"ID":"9d729832-e94c-4dc1-95cb-b67bbe754469","Type":"ContainerDied","Data":"9d53a51b7e0da7bd7e9529f05caaecebad2daed53dd87b10e5b0be9178678b6a"} Oct 30 14:32:37.508448 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:37.508406 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:32:37.640620 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:37.640587 4982 generic.go:356] "Generic (PLEG): container finished" podID="2de9f651-0083-42cd-89b2-6d28d35bb449" containerID="e5559289b82187bb9c7a4148b85b8a8b2bbe15ed976c90af906c8283cf10f3c4" exitCode=0 Oct 30 14:32:37.640995 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:37.640663 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj" event={"ID":"2de9f651-0083-42cd-89b2-6d28d35bb449","Type":"ContainerDied","Data":"e5559289b82187bb9c7a4148b85b8a8b2bbe15ed976c90af906c8283cf10f3c4"} Oct 30 14:32:37.641350 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:37.641166 4982 scope.go:122] "RemoveContainer" containerID="e5559289b82187bb9c7a4148b85b8a8b2bbe15ed976c90af906c8283cf10f3c4" Oct 30 14:32:37.642896 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:37.642865 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:32:37.642996 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:37.642947 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:32:37.642996 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:37.642988 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:32:37.895826 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:37.895799 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b9c4b4-qn6wb" Oct 30 14:32:38.001246 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:38.001209 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d729832-e94c-4dc1-95cb-b67bbe754469-kube-api-access-f68fg\" (UniqueName: \"kubernetes.io/projected/9d729832-e94c-4dc1-95cb-b67bbe754469-kube-api-access-f68fg\") pod \"9d729832-e94c-4dc1-95cb-b67bbe754469\" (UID: \"9d729832-e94c-4dc1-95cb-b67bbe754469\") " Oct 30 14:32:38.001490 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:38.001470 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d729832-e94c-4dc1-95cb-b67bbe754469-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d729832-e94c-4dc1-95cb-b67bbe754469-maas-api-xks-ca\") pod \"9d729832-e94c-4dc1-95cb-b67bbe754469\" (UID: \"9d729832-e94c-4dc1-95cb-b67bbe754469\") " Oct 30 14:32:38.001713 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:38.001691 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d729832-e94c-4dc1-95cb-b67bbe754469-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d729832-e94c-4dc1-95cb-b67bbe754469-maas-api-openshift-ca\") pod \"9d729832-e94c-4dc1-95cb-b67bbe754469\" (UID: \"9d729832-e94c-4dc1-95cb-b67bbe754469\") " Oct 30 14:32:38.002207 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:38.002175 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d729832-e94c-4dc1-95cb-b67bbe754469-maas-api-openshift-ca" pod "9d729832-e94c-4dc1-95cb-b67bbe754469" (UID: "9d729832-e94c-4dc1-95cb-b67bbe754469"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:32:38.002335 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:38.002315 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d729832-e94c-4dc1-95cb-b67bbe754469-maas-api-openshift-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:32:38.004058 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:38.004033 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d729832-e94c-4dc1-95cb-b67bbe754469-maas-api-xks-ca" pod "9d729832-e94c-4dc1-95cb-b67bbe754469" (UID: "9d729832-e94c-4dc1-95cb-b67bbe754469"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:32:38.005130 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:38.005104 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d729832-e94c-4dc1-95cb-b67bbe754469-kube-api-access-f68fg" pod "9d729832-e94c-4dc1-95cb-b67bbe754469" (UID: "9d729832-e94c-4dc1-95cb-b67bbe754469"). InnerVolumeSpecName "kube-api-access-f68fg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:32:38.103686 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:38.103628 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f68fg\" (UniqueName: \"kubernetes.io/projected/9d729832-e94c-4dc1-95cb-b67bbe754469-kube-api-access-f68fg\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:32:38.103686 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:38.103680 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d729832-e94c-4dc1-95cb-b67bbe754469-maas-api-xks-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:32:38.649781 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:38.649742 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b9c4b4-qn6wb" Oct 30 14:32:38.649781 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:38.649771 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b9c4b4-qn6wb" event={"ID":"9d729832-e94c-4dc1-95cb-b67bbe754469","Type":"ContainerDied","Data":"26f0223cc947e8088967683d7c32e3d7b74fc1fec5195cf361fa24690c30b0fd"} Oct 30 14:32:38.650243 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:38.649822 4982 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26f0223cc947e8088967683d7c32e3d7b74fc1fec5195cf361fa24690c30b0fd" Oct 30 14:32:38.651931 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:38.651909 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj" event={"ID":"2de9f651-0083-42cd-89b2-6d28d35bb449","Type":"ContainerStarted","Data":"e2e8377d208c59d979d3c59cf2d0733e9dfd509fa7f957e13d571980796e509a"} Oct 30 14:32:39.065806 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:39.065766 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj" Oct 30 14:32:39.066734 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:39.066702 4982 patch_prober.go:29] interesting pod/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 14:32:39.066877 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:39.066756 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj" podUID="2de9f651-0083-42cd-89b2-6d28d35bb449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 14:32:39.317243 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:39.317151 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:32:40.066528 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:40.066474 4982 patch_prober.go:29] interesting pod/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 14:32:40.066938 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:40.066569 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj" podUID="2de9f651-0083-42cd-89b2-6d28d35bb449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 14:32:40.609664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:40.608725 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-fde75d-7d8d99ffcb-79hw8" podUID="b473ec20-3b31-418d-87ac-bc1269c41744" containerName="payload-processing" containerID="cri-o://75bd369161fed518765352e20b7d7514b67e39becfd19fb6bd46ca7de3a10bc6" gracePeriod=25 Oct 30 14:32:40.689748 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:40.683720 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fde75d-75cd788695-5n8hq" podUID="a7b9a748-bba1-45e2-9612-0fd47cc5f8a6" containerName="payload-pre-processing" containerID="cri-o://c6944f8c7ad6bb968408d0640d78188f7d98c042c5806245ac4c254bcad24437" gracePeriod=25 Oct 30 14:32:41.065769 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:41.065735 4982 patch_prober.go:29] interesting pod/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 14:32:41.065926 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:41.065801 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj" podUID="2de9f651-0083-42cd-89b2-6d28d35bb449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 14:32:41.196663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:41.195202 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:32:42.065690 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:42.065609 4982 patch_prober.go:29] interesting pod/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 14:32:42.065895 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:42.065714 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj" podUID="2de9f651-0083-42cd-89b2-6d28d35bb449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 14:32:42.256813 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:42.256782 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754"] Oct 30 14:32:42.257432 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:42.257418 4982 cpu_manager.go:414] "containerMap: removing container" podUID="9d729832-e94c-4dc1-95cb-b67bbe754469" containerName="revoke-keys" Oct 30 14:32:42.257481 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:42.257434 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d729832-e94c-4dc1-95cb-b67bbe754469" containerName="revoke-keys" Oct 30 14:32:42.257549 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:42.257540 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d729832-e94c-4dc1-95cb-b67bbe754469" containerName="revoke-keys" Oct 30 14:32:42.261449 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:42.261427 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754" Oct 30 14:32:42.278599 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:42.278577 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-acefbb-openshift-default-dockercfg-6v2bk\"" Oct 30 14:32:42.327026 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:42.326803 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754"] Oct 30 14:32:42.368966 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:42.368938 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd0cec81-8246-425e-8b19-8a3f2937dabc-istiod-ca-cert\") pod \"e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754\" (UID: \"bd0cec81-8246-425e-8b19-8a3f2937dabc\") " pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754" Oct 30 14:32:42.369119 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:42.368972 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb679\" (UniqueName: \"kubernetes.io/projected/bd0cec81-8246-425e-8b19-8a3f2937dabc-kube-api-access-cb679\") pod \"e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754\" (UID: \"bd0cec81-8246-425e-8b19-8a3f2937dabc\") " pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754" Oct 30 14:32:42.369119 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:42.369004 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd0cec81-8246-425e-8b19-8a3f2937dabc-istio-envoy\") pod \"e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754\" (UID: \"bd0cec81-8246-425e-8b19-8a3f2937dabc\") " pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754" Oct 30 14:32:42.369119 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:42.369037 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd0cec81-8246-425e-8b19-8a3f2937dabc-workload-socket\") pod \"e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754\" (UID: \"bd0cec81-8246-425e-8b19-8a3f2937dabc\") " pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754" Oct 30 14:32:42.369119 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:42.369076 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd0cec81-8246-425e-8b19-8a3f2937dabc-credential-socket\") pod \"e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754\" (UID: \"bd0cec81-8246-425e-8b19-8a3f2937dabc\") " pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754" Oct 30 14:32:42.369119 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:42.369101 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd0cec81-8246-425e-8b19-8a3f2937dabc-istio-podinfo\") pod \"e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754\" (UID: \"bd0cec81-8246-425e-8b19-8a3f2937dabc\") " pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754" Oct 30 14:32:42.369288 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:42.369184 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd0cec81-8246-425e-8b19-8a3f2937dabc-istio-data\") pod \"e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754\" (UID: \"bd0cec81-8246-425e-8b19-8a3f2937dabc\") " pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754" Oct 30 14:32:42.369288 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:42.369226 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd0cec81-8246-425e-8b19-8a3f2937dabc-workload-certs\") pod \"e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754\" (UID: \"bd0cec81-8246-425e-8b19-8a3f2937dabc\") " pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754" Oct 30 14:32:42.369288 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:42.369244 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd0cec81-8246-425e-8b19-8a3f2937dabc-istio-token\") pod \"e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754\" (UID: \"bd0cec81-8246-425e-8b19-8a3f2937dabc\") " pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754" Oct 30 14:32:42.415697 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:42.415660 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fde75d-75cd788695-5n8hq" podUID="a7b9a748-bba1-45e2-9612-0fd47cc5f8a6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:32:42.471040 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:42.471012 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd0cec81-8246-425e-8b19-8a3f2937dabc-istiod-ca-cert\") pod \"e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754\" (UID: \"bd0cec81-8246-425e-8b19-8a3f2937dabc\") " pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754" Oct 30 14:32:42.471185 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:42.471047 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb679\" (UniqueName: \"kubernetes.io/projected/bd0cec81-8246-425e-8b19-8a3f2937dabc-kube-api-access-cb679\") pod \"e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754\" (UID: \"bd0cec81-8246-425e-8b19-8a3f2937dabc\") " pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754" Oct 30 14:32:42.471185 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:42.471081 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd0cec81-8246-425e-8b19-8a3f2937dabc-istio-envoy\") pod \"e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754\" (UID: \"bd0cec81-8246-425e-8b19-8a3f2937dabc\") " pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754" Oct 30 14:32:42.471185 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:42.471118 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd0cec81-8246-425e-8b19-8a3f2937dabc-workload-socket\") pod \"e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754\" (UID: \"bd0cec81-8246-425e-8b19-8a3f2937dabc\") " pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754" Oct 30 14:32:42.471185 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:42.471159 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd0cec81-8246-425e-8b19-8a3f2937dabc-credential-socket\") pod \"e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754\" (UID: \"bd0cec81-8246-425e-8b19-8a3f2937dabc\") " pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754" Oct 30 14:32:42.471381 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:42.471194 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd0cec81-8246-425e-8b19-8a3f2937dabc-istio-podinfo\") pod \"e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754\" (UID: \"bd0cec81-8246-425e-8b19-8a3f2937dabc\") " pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754" Oct 30 14:32:42.471381 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:42.471302 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd0cec81-8246-425e-8b19-8a3f2937dabc-istio-data\") pod \"e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754\" (UID: \"bd0cec81-8246-425e-8b19-8a3f2937dabc\") " pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754" Oct 30 14:32:42.471381 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:42.471361 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd0cec81-8246-425e-8b19-8a3f2937dabc-workload-certs\") pod \"e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754\" (UID: \"bd0cec81-8246-425e-8b19-8a3f2937dabc\") " pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754" Oct 30 14:32:42.471541 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:42.471389 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd0cec81-8246-425e-8b19-8a3f2937dabc-istio-token\") pod \"e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754\" (UID: \"bd0cec81-8246-425e-8b19-8a3f2937dabc\") " pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754" Oct 30 14:32:42.471608 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:42.471583 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd0cec81-8246-425e-8b19-8a3f2937dabc-workload-socket\") pod \"e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754\" (UID: \"bd0cec81-8246-425e-8b19-8a3f2937dabc\") " pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754" Oct 30 14:32:42.471703 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:42.471676 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd0cec81-8246-425e-8b19-8a3f2937dabc-credential-socket\") pod \"e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754\" (UID: \"bd0cec81-8246-425e-8b19-8a3f2937dabc\") " pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754" Oct 30 14:32:42.471771 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:42.471756 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd0cec81-8246-425e-8b19-8a3f2937dabc-workload-certs\") pod \"e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754\" (UID: \"bd0cec81-8246-425e-8b19-8a3f2937dabc\") " pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754" Oct 30 14:32:42.471925 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:42.471897 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd0cec81-8246-425e-8b19-8a3f2937dabc-istiod-ca-cert\") pod \"e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754\" (UID: \"bd0cec81-8246-425e-8b19-8a3f2937dabc\") " pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754" Oct 30 14:32:42.471925 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:42.471893 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd0cec81-8246-425e-8b19-8a3f2937dabc-istio-data\") pod \"e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754\" (UID: \"bd0cec81-8246-425e-8b19-8a3f2937dabc\") " pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754" Oct 30 14:32:42.473594 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:42.473575 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd0cec81-8246-425e-8b19-8a3f2937dabc-istio-envoy\") pod \"e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754\" (UID: \"bd0cec81-8246-425e-8b19-8a3f2937dabc\") " pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754" Oct 30 14:32:42.473846 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:42.473829 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd0cec81-8246-425e-8b19-8a3f2937dabc-istio-podinfo\") pod \"e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754\" (UID: \"bd0cec81-8246-425e-8b19-8a3f2937dabc\") " pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754" Oct 30 14:32:42.512662 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:42.510290 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb679\" (UniqueName: \"kubernetes.io/projected/bd0cec81-8246-425e-8b19-8a3f2937dabc-kube-api-access-cb679\") pod \"e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754\" (UID: \"bd0cec81-8246-425e-8b19-8a3f2937dabc\") " pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754" Oct 30 14:32:42.517660 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:42.512983 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd0cec81-8246-425e-8b19-8a3f2937dabc-istio-token\") pod \"e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754\" (UID: \"bd0cec81-8246-425e-8b19-8a3f2937dabc\") " pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754" Oct 30 14:32:42.575594 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:42.575566 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754" Oct 30 14:32:42.940503 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:42.940465 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754"] Oct 30 14:32:42.940811 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:32:42.940785 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd0cec81_8246_425e_8b19_8a3f2937dabc.slice/crio-f21207fd6ac85850b7e5ea846c2426e124c2ab734973b370b5fa4dd7f7354f6c WatchSource:0}: Error finding container f21207fd6ac85850b7e5ea846c2426e124c2ab734973b370b5fa4dd7f7354f6c: Status 404 returned error can't find the container with id f21207fd6ac85850b7e5ea846c2426e124c2ab734973b370b5fa4dd7f7354f6c Oct 30 14:32:42.943243 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:42.943208 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:32:42.943320 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:42.943278 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:32:42.943320 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:42.943307 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:32:43.068677 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:43.066475 4982 patch_prober.go:29] interesting pod/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 14:32:43.068677 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:43.066516 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj" podUID="2de9f651-0083-42cd-89b2-6d28d35bb449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 14:32:43.137755 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:43.137704 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:32:43.429687 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:43.429584 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-fde75d-7d8d99ffcb-79hw8" podUID="b473ec20-3b31-418d-87ac-bc1269c41744" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:32:43.728656 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:43.728525 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754" event={"ID":"bd0cec81-8246-425e-8b19-8a3f2937dabc","Type":"ContainerStarted","Data":"242b6205c94f85c84c6820c6e46e7a4f482fbc439aed3e6d47f6a3873a0daf66"} Oct 30 14:32:43.728656 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:43.728581 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754" event={"ID":"bd0cec81-8246-425e-8b19-8a3f2937dabc","Type":"ContainerStarted","Data":"f21207fd6ac85850b7e5ea846c2426e124c2ab734973b370b5fa4dd7f7354f6c"} Oct 30 14:32:43.790999 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:43.790943 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754" podStartSLOduration=1.790928088 podStartE2EDuration="1.790928088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:32:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:32:43.789161142 +0000 UTC m=+2633.533295195" watchObservedRunningTime="2026-10-30 14:32:43.790928088 +0000 UTC m=+2633.535062204" Oct 30 14:32:44.071080 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:44.071049 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj" Oct 30 14:32:44.071562 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:44.071540 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj" Oct 30 14:32:44.072448 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:44.072424 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj" Oct 30 14:32:44.164920 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:44.164876 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-f3c580-6d587b47f8-zthxw" podUID="522b3312-a38d-4253-998d-226d3bb2ae62" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:32:44.165103 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:44.164931 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3c580-7bff98f799-c6d7q" podUID="a2fc2354-7034-40dd-9086-ef67d615ec96" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:32:44.460363 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:44.460276 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b9c4b4-76948497d-7vhrd"] Oct 30 14:32:44.517905 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:44.517866 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b9c4b4-6498846c67-q52pq"] Oct 30 14:32:44.576140 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:44.576054 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754" Oct 30 14:32:44.577798 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:44.577772 4982 patch_prober.go:29] interesting pod/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Oct 30 14:32:44.577867 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:44.577827 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754" podUID="bd0cec81-8246-425e-8b19-8a3f2937dabc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Oct 30 14:32:44.644267 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:44.644212 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:32:44.789716 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:44.787115 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:32:45.248663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:45.248495 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:32:45.413416 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:45.413149 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:32:45.419766 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:45.419695 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f3c580-l4kxb"] Oct 30 14:32:45.464765 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:45.464719 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f3c580-l4kxb"] Oct 30 14:32:45.576392 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:45.576356 4982 patch_prober.go:29] interesting pod/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Oct 30 14:32:45.576542 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:45.576415 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754" podUID="bd0cec81-8246-425e-8b19-8a3f2937dabc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Oct 30 14:32:45.641726 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:45.641689 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:32:45.874723 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:45.874612 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:32:46.420087 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:46.420046 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a58b83db-877e-47c3-9108-cfa4bbb67028" path="/var/lib/kubelet/pods/a58b83db-877e-47c3-9108-cfa4bbb67028/volumes" Oct 30 14:32:46.576136 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:46.576103 4982 patch_prober.go:29] interesting pod/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Oct 30 14:32:46.576552 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:46.576165 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754" podUID="bd0cec81-8246-425e-8b19-8a3f2937dabc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Oct 30 14:32:47.454616 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:47.454578 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:32:47.576087 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:47.576040 4982 patch_prober.go:29] interesting pod/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Oct 30 14:32:47.576293 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:47.576116 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754" podUID="bd0cec81-8246-425e-8b19-8a3f2937dabc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Oct 30 14:32:48.575944 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:48.575902 4982 patch_prober.go:29] interesting pod/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Oct 30 14:32:48.576151 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:48.575976 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754" podUID="bd0cec81-8246-425e-8b19-8a3f2937dabc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Oct 30 14:32:49.447026 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:49.446984 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:32:49.460994 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:49.460939 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-b9c4b4-76948497d-7vhrd" podUID="68c95c68-83ae-4ea1-8ef2-b90f99c077e3" containerName="payload-processing" containerID="cri-o://fe7c21f76782ed21ccc3b1d710ec55d95e4a06820290bf20109672b4a006e11e" gracePeriod=25 Oct 30 14:32:49.518743 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:49.518702 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b9c4b4-6498846c67-q52pq" podUID="b87e2746-abcd-4e8a-bda1-cf9badd05b04" containerName="payload-pre-processing" containerID="cri-o://7daa8053c072645197f3bec6f19ffed89e19e22e92a3eacdf53da4fb3788d231" gracePeriod=25 Oct 30 14:32:49.575958 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:49.575927 4982 patch_prober.go:29] interesting pod/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Oct 30 14:32:49.576117 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:49.575991 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754" podUID="bd0cec81-8246-425e-8b19-8a3f2937dabc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Oct 30 14:32:49.779331 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:49.779296 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b9c4b4-76948497d-7vhrd" Oct 30 14:32:49.782151 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:49.782124 4982 generic.go:356] "Generic (PLEG): container finished" podID="68c95c68-83ae-4ea1-8ef2-b90f99c077e3" containerID="fe7c21f76782ed21ccc3b1d710ec55d95e4a06820290bf20109672b4a006e11e" exitCode=0 Oct 30 14:32:49.782280 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:49.782167 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b9c4b4-76948497d-7vhrd" Oct 30 14:32:49.782280 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:49.782203 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b9c4b4-76948497d-7vhrd" event={"ID":"68c95c68-83ae-4ea1-8ef2-b90f99c077e3","Type":"ContainerDied","Data":"fe7c21f76782ed21ccc3b1d710ec55d95e4a06820290bf20109672b4a006e11e"} Oct 30 14:32:49.782280 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:49.782256 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b9c4b4-76948497d-7vhrd" event={"ID":"68c95c68-83ae-4ea1-8ef2-b90f99c077e3","Type":"ContainerDied","Data":"c40ff919cbef033f9acda46ba4ed48c33ec75b757f4b1e544ef5ba5344e195c1"} Oct 30 14:32:49.782280 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:49.782279 4982 scope.go:122] "RemoveContainer" containerID="fe7c21f76782ed21ccc3b1d710ec55d95e4a06820290bf20109672b4a006e11e" Oct 30 14:32:49.795426 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:49.795397 4982 scope.go:122] "RemoveContainer" containerID="fe7c21f76782ed21ccc3b1d710ec55d95e4a06820290bf20109672b4a006e11e" Oct 30 14:32:49.795762 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:32:49.795737 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe7c21f76782ed21ccc3b1d710ec55d95e4a06820290bf20109672b4a006e11e\": container with ID starting with fe7c21f76782ed21ccc3b1d710ec55d95e4a06820290bf20109672b4a006e11e not found: ID does not exist" containerID="fe7c21f76782ed21ccc3b1d710ec55d95e4a06820290bf20109672b4a006e11e" Oct 30 14:32:49.795884 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:49.795769 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe7c21f76782ed21ccc3b1d710ec55d95e4a06820290bf20109672b4a006e11e"} err="failed to get container status \"fe7c21f76782ed21ccc3b1d710ec55d95e4a06820290bf20109672b4a006e11e\": rpc error: code = NotFound desc = could not find container \"fe7c21f76782ed21ccc3b1d710ec55d95e4a06820290bf20109672b4a006e11e\": container with ID starting with fe7c21f76782ed21ccc3b1d710ec55d95e4a06820290bf20109672b4a006e11e not found: ID does not exist" Oct 30 14:32:49.892216 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:49.892185 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68c95c68-83ae-4ea1-8ef2-b90f99c077e3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68c95c68-83ae-4ea1-8ef2-b90f99c077e3-plugins-config-volume\") pod \"68c95c68-83ae-4ea1-8ef2-b90f99c077e3\" (UID: \"68c95c68-83ae-4ea1-8ef2-b90f99c077e3\") " Oct 30 14:32:49.892431 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:49.892408 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68c95c68-83ae-4ea1-8ef2-b90f99c077e3-kube-api-access-bjclk\" (UniqueName: \"kubernetes.io/projected/68c95c68-83ae-4ea1-8ef2-b90f99c077e3-kube-api-access-bjclk\") pod \"68c95c68-83ae-4ea1-8ef2-b90f99c077e3\" (UID: \"68c95c68-83ae-4ea1-8ef2-b90f99c077e3\") " Oct 30 14:32:49.892615 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:49.892590 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68c95c68-83ae-4ea1-8ef2-b90f99c077e3-plugins-config-volume" pod "68c95c68-83ae-4ea1-8ef2-b90f99c077e3" (UID: "68c95c68-83ae-4ea1-8ef2-b90f99c077e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:32:49.892995 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:49.892977 4982 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68c95c68-83ae-4ea1-8ef2-b90f99c077e3-plugins-config-volume\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:32:49.895206 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:49.895178 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68c95c68-83ae-4ea1-8ef2-b90f99c077e3-kube-api-access-bjclk" pod "68c95c68-83ae-4ea1-8ef2-b90f99c077e3" (UID: "68c95c68-83ae-4ea1-8ef2-b90f99c077e3"). InnerVolumeSpecName "kube-api-access-bjclk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:32:49.994478 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:49.994401 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjclk\" (UniqueName: \"kubernetes.io/projected/68c95c68-83ae-4ea1-8ef2-b90f99c077e3-kube-api-access-bjclk\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:32:50.134654 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:50.134604 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b9c4b4-76948497d-7vhrd"] Oct 30 14:32:50.140985 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:50.140956 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b9c4b4-76948497d-7vhrd"] Oct 30 14:32:50.410524 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:50.407188 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68c95c68-83ae-4ea1-8ef2-b90f99c077e3" path="/var/lib/kubelet/pods/68c95c68-83ae-4ea1-8ef2-b90f99c077e3/volumes" Oct 30 14:32:50.581010 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:50.580978 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754" Oct 30 14:32:50.581467 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:50.581433 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754" Oct 30 14:32:50.582251 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:50.582229 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754" Oct 30 14:32:51.457741 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:51.457707 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:32:51.525100 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:51.525067 4982 scope.go:122] "RemoveContainer" containerID="961d8f130b9ada713e709c06b8e5f07b7a67cdb374b7b5233dafce862afcc431" Oct 30 14:32:51.713365 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:51.713306 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:32:51.886211 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:51.886169 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:32:52.163154 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:52.163111 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:32:52.408211 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:52.408174 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fde75d-75cd788695-5n8hq" podUID="a7b9a748-bba1-45e2-9612-0fd47cc5f8a6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:32:52.655785 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:52.655745 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:32:52.897670 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:52.895016 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5842c"] Oct 30 14:32:52.897670 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:52.895263 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5842c" podUID="f56cfd10-4ad6-4c14-b614-ae3b48b87dbe" containerName="istio-proxy" containerID="cri-o://dffe6a02a7a9b8f26bd1e813d4a04b50f1970cb1dbe6cc1cdb7998e5cfc012d4" gracePeriod=30 Oct 30 14:32:53.104156 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:53.104124 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:32:53.458664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:53.457817 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-fde75d-7d8d99ffcb-79hw8" podUID="b473ec20-3b31-418d-87ac-bc1269c41744" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:32:53.642138 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:53.642057 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:32:53.781814 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:53.778761 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fde75d-9hjg8"] Oct 30 14:32:53.802660 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:53.790488 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:32:53.802660 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:53.791373 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fde75d-9hjg8"] Oct 30 14:32:53.882865 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:53.882783 4982 generic.go:356] "Generic (PLEG): container finished" podID="522b3312-a38d-4253-998d-226d3bb2ae62" containerID="55cacd2b97162e09df600d8d9bd9fe12e9403dab34763ede166dfbec82c36827" exitCode=0 Oct 30 14:32:53.883008 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:53.882863 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f3c580-6d587b47f8-zthxw" event={"ID":"522b3312-a38d-4253-998d-226d3bb2ae62","Type":"ContainerDied","Data":"55cacd2b97162e09df600d8d9bd9fe12e9403dab34763ede166dfbec82c36827"} Oct 30 14:32:53.884596 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:53.884574 4982 generic.go:356] "Generic (PLEG): container finished" podID="a2fc2354-7034-40dd-9086-ef67d615ec96" containerID="d6aa943ac56d42cbd55cfa730e0da33f617931fcf5a68a1d5a8f69ea68526731" exitCode=0 Oct 30 14:32:53.884730 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:53.884649 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3c580-7bff98f799-c6d7q" event={"ID":"a2fc2354-7034-40dd-9086-ef67d615ec96","Type":"ContainerDied","Data":"d6aa943ac56d42cbd55cfa730e0da33f617931fcf5a68a1d5a8f69ea68526731"} Oct 30 14:32:53.929286 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:53.929257 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f3c580-6d587b47f8-zthxw" Oct 30 14:32:53.932853 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:53.932830 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3c580-7bff98f799-c6d7q" Oct 30 14:32:54.009416 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:54.009376 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/522b3312-a38d-4253-998d-226d3bb2ae62-kube-api-access-8wrfq\" (UniqueName: \"kubernetes.io/projected/522b3312-a38d-4253-998d-226d3bb2ae62-kube-api-access-8wrfq\") pod \"522b3312-a38d-4253-998d-226d3bb2ae62\" (UID: \"522b3312-a38d-4253-998d-226d3bb2ae62\") " Oct 30 14:32:54.009609 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:54.009429 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/522b3312-a38d-4253-998d-226d3bb2ae62-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/522b3312-a38d-4253-998d-226d3bb2ae62-plugins-config-volume\") pod \"522b3312-a38d-4253-998d-226d3bb2ae62\" (UID: \"522b3312-a38d-4253-998d-226d3bb2ae62\") " Oct 30 14:32:54.009609 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:54.009453 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2fc2354-7034-40dd-9086-ef67d615ec96-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2fc2354-7034-40dd-9086-ef67d615ec96-plugins-config-volume\") pod \"a2fc2354-7034-40dd-9086-ef67d615ec96\" (UID: \"a2fc2354-7034-40dd-9086-ef67d615ec96\") " Oct 30 14:32:54.009609 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:54.009523 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2fc2354-7034-40dd-9086-ef67d615ec96-kube-api-access-zz7b7\" (UniqueName: \"kubernetes.io/projected/a2fc2354-7034-40dd-9086-ef67d615ec96-kube-api-access-zz7b7\") pod \"a2fc2354-7034-40dd-9086-ef67d615ec96\" (UID: \"a2fc2354-7034-40dd-9086-ef67d615ec96\") " Oct 30 14:32:54.009933 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:54.009905 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2fc2354-7034-40dd-9086-ef67d615ec96-plugins-config-volume" pod "a2fc2354-7034-40dd-9086-ef67d615ec96" (UID: "a2fc2354-7034-40dd-9086-ef67d615ec96"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:32:54.010004 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:54.009928 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/522b3312-a38d-4253-998d-226d3bb2ae62-plugins-config-volume" pod "522b3312-a38d-4253-998d-226d3bb2ae62" (UID: "522b3312-a38d-4253-998d-226d3bb2ae62"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:32:54.012728 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:54.012684 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2fc2354-7034-40dd-9086-ef67d615ec96-kube-api-access-zz7b7" pod "a2fc2354-7034-40dd-9086-ef67d615ec96" (UID: "a2fc2354-7034-40dd-9086-ef67d615ec96"). InnerVolumeSpecName "kube-api-access-zz7b7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:32:54.032144 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:54.029010 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/522b3312-a38d-4253-998d-226d3bb2ae62-kube-api-access-8wrfq" pod "522b3312-a38d-4253-998d-226d3bb2ae62" (UID: "522b3312-a38d-4253-998d-226d3bb2ae62"). InnerVolumeSpecName "kube-api-access-8wrfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:32:54.047882 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:54.047842 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:32:54.049490 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:54.049460 4982 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/522b3312-a38d-4253-998d-226d3bb2ae62-plugins-config-volume\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:32:54.049628 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:54.049497 4982 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2fc2354-7034-40dd-9086-ef67d615ec96-plugins-config-volume\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:32:54.150982 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:54.150894 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wrfq\" (UniqueName: \"kubernetes.io/projected/522b3312-a38d-4253-998d-226d3bb2ae62-kube-api-access-8wrfq\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:32:54.150982 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:54.150925 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zz7b7\" (UniqueName: \"kubernetes.io/projected/a2fc2354-7034-40dd-9086-ef67d615ec96-kube-api-access-zz7b7\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:32:54.196319 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:54.196284 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:32:54.363513 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:54.363474 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:32:54.452774 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:54.452695 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b7670b5-5942-49ab-b6ae-7200698f575a" path="/var/lib/kubelet/pods/2b7670b5-5942-49ab-b6ae-7200698f575a/volumes" Oct 30 14:32:54.514409 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:54.513129 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:32:54.822954 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:54.822920 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0b1c15-t58mk"] Oct 30 14:32:54.823937 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:54.823906 4982 cpu_manager.go:414] "containerMap: removing container" podUID="522b3312-a38d-4253-998d-226d3bb2ae62" containerName="payload-processing" Oct 30 14:32:54.823937 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:54.823934 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="522b3312-a38d-4253-998d-226d3bb2ae62" containerName="payload-processing" Oct 30 14:32:54.824118 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:54.823978 4982 cpu_manager.go:414] "containerMap: removing container" podUID="68c95c68-83ae-4ea1-8ef2-b90f99c077e3" containerName="payload-processing" Oct 30 14:32:54.824118 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:54.823985 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68c95c68-83ae-4ea1-8ef2-b90f99c077e3" containerName="payload-processing" Oct 30 14:32:54.824118 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:54.824015 4982 cpu_manager.go:414] "containerMap: removing container" podUID="a2fc2354-7034-40dd-9086-ef67d615ec96" containerName="payload-pre-processing" Oct 30 14:32:54.824118 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:54.824023 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2fc2354-7034-40dd-9086-ef67d615ec96" containerName="payload-pre-processing" Oct 30 14:32:54.824257 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:54.824176 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="68c95c68-83ae-4ea1-8ef2-b90f99c077e3" containerName="payload-processing" Oct 30 14:32:54.824257 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:54.824191 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2fc2354-7034-40dd-9086-ef67d615ec96" containerName="payload-pre-processing" Oct 30 14:32:54.824257 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:54.824208 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="522b3312-a38d-4253-998d-226d3bb2ae62" containerName="payload-processing" Oct 30 14:32:54.829045 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:54.829024 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0b1c15-t58mk" Oct 30 14:32:54.846301 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:54.846275 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0b1c15-t58mk"] Oct 30 14:32:54.854672 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:54.854629 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:32:54.854925 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:54.854906 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:32:54.874858 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:54.874827 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a35dab5-bed1-4d54-95d0-370ccf127c4b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0b1c15-t58mk\" (UID: \"7a35dab5-bed1-4d54-95d0-370ccf127c4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0b1c15-t58mk" Oct 30 14:32:54.875137 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:54.875114 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfcd9\" (UniqueName: \"kubernetes.io/projected/7a35dab5-bed1-4d54-95d0-370ccf127c4b-kube-api-access-bfcd9\") pod \"maas-api-revoke-keys-e2e-inf-b-0b1c15-t58mk\" (UID: \"7a35dab5-bed1-4d54-95d0-370ccf127c4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0b1c15-t58mk" Oct 30 14:32:54.875207 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:54.875184 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a35dab5-bed1-4d54-95d0-370ccf127c4b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0b1c15-t58mk\" (UID: \"7a35dab5-bed1-4d54-95d0-370ccf127c4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0b1c15-t58mk" Oct 30 14:32:54.897955 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:54.897923 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f3c580-6d587b47f8-zthxw" event={"ID":"522b3312-a38d-4253-998d-226d3bb2ae62","Type":"ContainerDied","Data":"f0814caeb11905efd1df4526886f890f609424276ae5367c3821db361b4464a2"} Oct 30 14:32:54.898099 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:54.897991 4982 scope.go:122] "RemoveContainer" containerID="55cacd2b97162e09df600d8d9bd9fe12e9403dab34763ede166dfbec82c36827" Oct 30 14:32:54.898178 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:54.898164 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f3c580-6d587b47f8-zthxw" Oct 30 14:32:54.902518 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:54.901853 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3c580-7bff98f799-c6d7q" event={"ID":"a2fc2354-7034-40dd-9086-ef67d615ec96","Type":"ContainerDied","Data":"25d17f35fee99ed05cd0f1038dc2aebcb02702f1129dce2771b04979ad4fcd52"} Oct 30 14:32:54.902518 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:54.901997 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3c580-7bff98f799-c6d7q" Oct 30 14:32:54.915599 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:54.915581 4982 scope.go:122] "RemoveContainer" containerID="d6aa943ac56d42cbd55cfa730e0da33f617931fcf5a68a1d5a8f69ea68526731" Oct 30 14:32:54.958528 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:54.958506 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f3c580-6d587b47f8-zthxw"] Oct 30 14:32:54.964928 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:54.964903 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f3c580-6d587b47f8-zthxw"] Oct 30 14:32:54.976026 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:54.976002 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfcd9\" (UniqueName: \"kubernetes.io/projected/7a35dab5-bed1-4d54-95d0-370ccf127c4b-kube-api-access-bfcd9\") pod \"maas-api-revoke-keys-e2e-inf-b-0b1c15-t58mk\" (UID: \"7a35dab5-bed1-4d54-95d0-370ccf127c4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0b1c15-t58mk" Oct 30 14:32:54.976163 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:54.976142 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a35dab5-bed1-4d54-95d0-370ccf127c4b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0b1c15-t58mk\" (UID: \"7a35dab5-bed1-4d54-95d0-370ccf127c4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0b1c15-t58mk" Oct 30 14:32:54.976260 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:54.976241 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a35dab5-bed1-4d54-95d0-370ccf127c4b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0b1c15-t58mk\" (UID: \"7a35dab5-bed1-4d54-95d0-370ccf127c4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0b1c15-t58mk" Oct 30 14:32:54.976862 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:54.976839 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a35dab5-bed1-4d54-95d0-370ccf127c4b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0b1c15-t58mk\" (UID: \"7a35dab5-bed1-4d54-95d0-370ccf127c4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0b1c15-t58mk" Oct 30 14:32:54.978702 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:54.978678 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a35dab5-bed1-4d54-95d0-370ccf127c4b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0b1c15-t58mk\" (UID: \"7a35dab5-bed1-4d54-95d0-370ccf127c4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0b1c15-t58mk" Oct 30 14:32:55.002061 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:55.002038 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f3c580-7bff98f799-c6d7q"] Oct 30 14:32:55.002185 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:55.002162 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfcd9\" (UniqueName: \"kubernetes.io/projected/7a35dab5-bed1-4d54-95d0-370ccf127c4b-kube-api-access-bfcd9\") pod \"maas-api-revoke-keys-e2e-inf-b-0b1c15-t58mk\" (UID: \"7a35dab5-bed1-4d54-95d0-370ccf127c4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0b1c15-t58mk" Oct 30 14:32:55.009498 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:55.009475 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f3c580-7bff98f799-c6d7q"] Oct 30 14:32:55.144406 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:55.144310 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0b1c15-t58mk" Oct 30 14:32:55.406128 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:55.406046 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80180acb-ww7mh"] Oct 30 14:32:55.410525 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:55.410503 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80180acb-ww7mh" Oct 30 14:32:55.440505 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:55.440470 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80180acb-ww7mh"] Oct 30 14:32:55.472333 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:55.472298 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0b1c15-t58mk"] Oct 30 14:32:55.473822 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:32:55.473766 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a35dab5_bed1_4d54_95d0_370ccf127c4b.slice/crio-5083626822c8d3ee7d1dec49f30c9616f26177e3fbd2f40346351c86fcda47b9 WatchSource:0}: Error finding container 5083626822c8d3ee7d1dec49f30c9616f26177e3fbd2f40346351c86fcda47b9: Status 404 returned error can't find the container with id 5083626822c8d3ee7d1dec49f30c9616f26177e3fbd2f40346351c86fcda47b9 Oct 30 14:32:55.494908 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:55.493606 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgq8m\" (UniqueName: \"kubernetes.io/projected/260f52cf-e8e0-4122-9818-f9e8a149cb55-kube-api-access-tgq8m\") pod \"maas-api-revoke-sub-80180acb-ww7mh\" (UID: \"260f52cf-e8e0-4122-9818-f9e8a149cb55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80180acb-ww7mh" Oct 30 14:32:55.496566 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:55.495305 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/260f52cf-e8e0-4122-9818-f9e8a149cb55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80180acb-ww7mh\" (UID: \"260f52cf-e8e0-4122-9818-f9e8a149cb55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80180acb-ww7mh" Oct 30 14:32:55.496566 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:55.495426 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/260f52cf-e8e0-4122-9818-f9e8a149cb55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80180acb-ww7mh\" (UID: \"260f52cf-e8e0-4122-9818-f9e8a149cb55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80180acb-ww7mh" Oct 30 14:32:55.540346 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:55.540308 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b9c4b4-6498846c67-q52pq" podUID="b87e2746-abcd-4e8a-bda1-cf9badd05b04" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:32:55.596613 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:55.596588 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgq8m\" (UniqueName: \"kubernetes.io/projected/260f52cf-e8e0-4122-9818-f9e8a149cb55-kube-api-access-tgq8m\") pod \"maas-api-revoke-sub-80180acb-ww7mh\" (UID: \"260f52cf-e8e0-4122-9818-f9e8a149cb55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80180acb-ww7mh" Oct 30 14:32:55.596769 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:55.596683 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/260f52cf-e8e0-4122-9818-f9e8a149cb55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80180acb-ww7mh\" (UID: \"260f52cf-e8e0-4122-9818-f9e8a149cb55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80180acb-ww7mh" Oct 30 14:32:55.596769 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:55.596734 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/260f52cf-e8e0-4122-9818-f9e8a149cb55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80180acb-ww7mh\" (UID: \"260f52cf-e8e0-4122-9818-f9e8a149cb55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80180acb-ww7mh" Oct 30 14:32:55.597412 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:55.597390 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/260f52cf-e8e0-4122-9818-f9e8a149cb55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80180acb-ww7mh\" (UID: \"260f52cf-e8e0-4122-9818-f9e8a149cb55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80180acb-ww7mh" Oct 30 14:32:55.599035 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:55.599017 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/260f52cf-e8e0-4122-9818-f9e8a149cb55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80180acb-ww7mh\" (UID: \"260f52cf-e8e0-4122-9818-f9e8a149cb55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80180acb-ww7mh" Oct 30 14:32:55.622342 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:55.622314 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgq8m\" (UniqueName: \"kubernetes.io/projected/260f52cf-e8e0-4122-9818-f9e8a149cb55-kube-api-access-tgq8m\") pod \"maas-api-revoke-sub-80180acb-ww7mh\" (UID: \"260f52cf-e8e0-4122-9818-f9e8a149cb55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80180acb-ww7mh" Oct 30 14:32:55.728147 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:55.728120 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80180acb-ww7mh" Oct 30 14:32:55.912705 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:55.912672 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0b1c15-t58mk" event={"ID":"7a35dab5-bed1-4d54-95d0-370ccf127c4b","Type":"ContainerStarted","Data":"9848c739c54ccfeadcb7297f715799b332288bbb605ca1a9c06d9fd719956df8"} Oct 30 14:32:55.912845 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:55.912728 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0b1c15-t58mk" event={"ID":"7a35dab5-bed1-4d54-95d0-370ccf127c4b","Type":"ContainerStarted","Data":"5083626822c8d3ee7d1dec49f30c9616f26177e3fbd2f40346351c86fcda47b9"} Oct 30 14:32:55.946051 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:55.945992 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:32:55.969662 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:55.966031 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0b1c15-t58mk" podStartSLOduration=1.96601088 podStartE2EDuration="1.96601088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:32:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:32:55.95611948 +0000 UTC m=+2645.700253519" watchObservedRunningTime="2026-10-30 14:32:55.96601088 +0000 UTC m=+2645.710144921" Oct 30 14:32:56.041475 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:32:56.041441 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod260f52cf_e8e0_4122_9818_f9e8a149cb55.slice/crio-300ce0cddcf4b5238b2e32e7e18e720216777be696d1381c6d8f589133d309e6 WatchSource:0}: Error finding container 300ce0cddcf4b5238b2e32e7e18e720216777be696d1381c6d8f589133d309e6: Status 404 returned error can't find the container with id 300ce0cddcf4b5238b2e32e7e18e720216777be696d1381c6d8f589133d309e6 Oct 30 14:32:56.042208 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:56.042181 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80180acb-ww7mh"] Oct 30 14:32:56.401764 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:56.401731 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="522b3312-a38d-4253-998d-226d3bb2ae62" path="/var/lib/kubelet/pods/522b3312-a38d-4253-998d-226d3bb2ae62/volumes" Oct 30 14:32:56.402272 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:56.402250 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2fc2354-7034-40dd-9086-ef67d615ec96" path="/var/lib/kubelet/pods/a2fc2354-7034-40dd-9086-ef67d615ec96/volumes" Oct 30 14:32:56.925262 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:56.925227 4982 generic.go:356] "Generic (PLEG): container finished" podID="7a35dab5-bed1-4d54-95d0-370ccf127c4b" containerID="9848c739c54ccfeadcb7297f715799b332288bbb605ca1a9c06d9fd719956df8" exitCode=0 Oct 30 14:32:56.925448 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:56.925301 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0b1c15-t58mk" event={"ID":"7a35dab5-bed1-4d54-95d0-370ccf127c4b","Type":"ContainerDied","Data":"9848c739c54ccfeadcb7297f715799b332288bbb605ca1a9c06d9fd719956df8"} Oct 30 14:32:56.927326 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:56.927293 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80180acb-ww7mh" event={"ID":"260f52cf-e8e0-4122-9818-f9e8a149cb55","Type":"ContainerStarted","Data":"67580ea999e193191948f43619c552c074b802a27b9a3003caf6e39389186969"} Oct 30 14:32:56.927498 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:56.927348 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80180acb-ww7mh" event={"ID":"260f52cf-e8e0-4122-9818-f9e8a149cb55","Type":"ContainerStarted","Data":"300ce0cddcf4b5238b2e32e7e18e720216777be696d1381c6d8f589133d309e6"} Oct 30 14:32:56.990660 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:56.989858 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:32:57.024654 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:57.024582 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80180acb-ww7mh" podStartSLOduration=2.024565424 podStartE2EDuration="2.024565424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:32:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:32:57.021998618 +0000 UTC m=+2646.766132662" watchObservedRunningTime="2026-10-30 14:32:57.024565424 +0000 UTC m=+2646.768699466" Oct 30 14:32:57.135965 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:57.135934 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:32:57.936004 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:57.935966 4982 generic.go:356] "Generic (PLEG): container finished" podID="260f52cf-e8e0-4122-9818-f9e8a149cb55" containerID="67580ea999e193191948f43619c552c074b802a27b9a3003caf6e39389186969" exitCode=0 Oct 30 14:32:57.936180 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:57.936047 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80180acb-ww7mh" event={"ID":"260f52cf-e8e0-4122-9818-f9e8a149cb55","Type":"ContainerDied","Data":"67580ea999e193191948f43619c552c074b802a27b9a3003caf6e39389186969"} Oct 30 14:32:57.945015 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:57.944987 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:32:57.952401 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:57.952368 4982 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-5842c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Oct 30 14:32:57.954292 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:57.954094 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5842c" podUID="f56cfd10-4ad6-4c14-b614-ae3b48b87dbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Oct 30 14:32:58.027358 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:32:58.027317 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:32:58.182793 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:58.182430 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0b1c15-t58mk" Oct 30 14:32:58.231520 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:58.231484 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a35dab5-bed1-4d54-95d0-370ccf127c4b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a35dab5-bed1-4d54-95d0-370ccf127c4b-maas-api-xks-ca\") pod \"7a35dab5-bed1-4d54-95d0-370ccf127c4b\" (UID: \"7a35dab5-bed1-4d54-95d0-370ccf127c4b\") " Oct 30 14:32:58.231709 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:58.231572 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a35dab5-bed1-4d54-95d0-370ccf127c4b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a35dab5-bed1-4d54-95d0-370ccf127c4b-maas-api-openshift-ca\") pod \"7a35dab5-bed1-4d54-95d0-370ccf127c4b\" (UID: \"7a35dab5-bed1-4d54-95d0-370ccf127c4b\") " Oct 30 14:32:58.231881 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:58.231842 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a35dab5-bed1-4d54-95d0-370ccf127c4b-kube-api-access-bfcd9\" (UniqueName: \"kubernetes.io/projected/7a35dab5-bed1-4d54-95d0-370ccf127c4b-kube-api-access-bfcd9\") pod \"7a35dab5-bed1-4d54-95d0-370ccf127c4b\" (UID: \"7a35dab5-bed1-4d54-95d0-370ccf127c4b\") " Oct 30 14:32:58.232395 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:58.232072 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a35dab5-bed1-4d54-95d0-370ccf127c4b-maas-api-openshift-ca" pod "7a35dab5-bed1-4d54-95d0-370ccf127c4b" (UID: "7a35dab5-bed1-4d54-95d0-370ccf127c4b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:32:58.232797 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:58.232778 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a35dab5-bed1-4d54-95d0-370ccf127c4b-maas-api-openshift-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:32:58.234401 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:58.234379 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a35dab5-bed1-4d54-95d0-370ccf127c4b-maas-api-xks-ca" pod "7a35dab5-bed1-4d54-95d0-370ccf127c4b" (UID: "7a35dab5-bed1-4d54-95d0-370ccf127c4b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:32:58.234599 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:58.234584 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a35dab5-bed1-4d54-95d0-370ccf127c4b-kube-api-access-bfcd9" pod "7a35dab5-bed1-4d54-95d0-370ccf127c4b" (UID: "7a35dab5-bed1-4d54-95d0-370ccf127c4b"). InnerVolumeSpecName "kube-api-access-bfcd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:32:58.246410 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:58.246388 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5842c" Oct 30 14:32:58.333805 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:58.333774 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f56cfd10-4ad6-4c14-b614-ae3b48b87dbe-istio-token\" (UniqueName: \"kubernetes.io/projected/f56cfd10-4ad6-4c14-b614-ae3b48b87dbe-istio-token\") pod \"f56cfd10-4ad6-4c14-b614-ae3b48b87dbe\" (UID: \"f56cfd10-4ad6-4c14-b614-ae3b48b87dbe\") " Oct 30 14:32:58.333987 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:58.333842 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f56cfd10-4ad6-4c14-b614-ae3b48b87dbe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f56cfd10-4ad6-4c14-b614-ae3b48b87dbe-credential-socket\") pod \"f56cfd10-4ad6-4c14-b614-ae3b48b87dbe\" (UID: \"f56cfd10-4ad6-4c14-b614-ae3b48b87dbe\") " Oct 30 14:32:58.333987 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:58.333883 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f56cfd10-4ad6-4c14-b614-ae3b48b87dbe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f56cfd10-4ad6-4c14-b614-ae3b48b87dbe-istiod-ca-cert\") pod \"f56cfd10-4ad6-4c14-b614-ae3b48b87dbe\" (UID: \"f56cfd10-4ad6-4c14-b614-ae3b48b87dbe\") " Oct 30 14:32:58.334101 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:58.334041 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f56cfd10-4ad6-4c14-b614-ae3b48b87dbe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f56cfd10-4ad6-4c14-b614-ae3b48b87dbe-istio-podinfo\") pod \"f56cfd10-4ad6-4c14-b614-ae3b48b87dbe\" (UID: \"f56cfd10-4ad6-4c14-b614-ae3b48b87dbe\") " Oct 30 14:32:58.334101 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:58.334078 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f56cfd10-4ad6-4c14-b614-ae3b48b87dbe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f56cfd10-4ad6-4c14-b614-ae3b48b87dbe-workload-certs\") pod \"f56cfd10-4ad6-4c14-b614-ae3b48b87dbe\" (UID: \"f56cfd10-4ad6-4c14-b614-ae3b48b87dbe\") " Oct 30 14:32:58.334202 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:58.334126 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f56cfd10-4ad6-4c14-b614-ae3b48b87dbe-kube-api-access-8ssg9\" (UniqueName: \"kubernetes.io/projected/f56cfd10-4ad6-4c14-b614-ae3b48b87dbe-kube-api-access-8ssg9\") pod \"f56cfd10-4ad6-4c14-b614-ae3b48b87dbe\" (UID: \"f56cfd10-4ad6-4c14-b614-ae3b48b87dbe\") " Oct 30 14:32:58.334255 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:58.334225 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f56cfd10-4ad6-4c14-b614-ae3b48b87dbe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f56cfd10-4ad6-4c14-b614-ae3b48b87dbe-workload-socket\") pod \"f56cfd10-4ad6-4c14-b614-ae3b48b87dbe\" (UID: \"f56cfd10-4ad6-4c14-b614-ae3b48b87dbe\") " Oct 30 14:32:58.334308 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:58.334268 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f56cfd10-4ad6-4c14-b614-ae3b48b87dbe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f56cfd10-4ad6-4c14-b614-ae3b48b87dbe-istio-envoy\") pod \"f56cfd10-4ad6-4c14-b614-ae3b48b87dbe\" (UID: \"f56cfd10-4ad6-4c14-b614-ae3b48b87dbe\") " Oct 30 14:32:58.334407 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:58.334390 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f56cfd10-4ad6-4c14-b614-ae3b48b87dbe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f56cfd10-4ad6-4c14-b614-ae3b48b87dbe-istio-data\") pod \"f56cfd10-4ad6-4c14-b614-ae3b48b87dbe\" (UID: \"f56cfd10-4ad6-4c14-b614-ae3b48b87dbe\") " Oct 30 14:32:58.334502 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:58.334475 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f56cfd10-4ad6-4c14-b614-ae3b48b87dbe-istiod-ca-cert" pod "f56cfd10-4ad6-4c14-b614-ae3b48b87dbe" (UID: "f56cfd10-4ad6-4c14-b614-ae3b48b87dbe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:32:58.334754 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:58.334733 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f56cfd10-4ad6-4c14-b614-ae3b48b87dbe-credential-socket" pod "f56cfd10-4ad6-4c14-b614-ae3b48b87dbe" (UID: "f56cfd10-4ad6-4c14-b614-ae3b48b87dbe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:32:58.334959 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:58.334940 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f56cfd10-4ad6-4c14-b614-ae3b48b87dbe-istio-data" pod "f56cfd10-4ad6-4c14-b614-ae3b48b87dbe" (UID: "f56cfd10-4ad6-4c14-b614-ae3b48b87dbe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:32:58.335039 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:58.335014 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a35dab5-bed1-4d54-95d0-370ccf127c4b-maas-api-xks-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:32:58.335039 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:58.335032 4982 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f56cfd10-4ad6-4c14-b614-ae3b48b87dbe-credential-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:32:58.335154 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:58.335046 4982 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f56cfd10-4ad6-4c14-b614-ae3b48b87dbe-istiod-ca-cert\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:32:58.335154 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:58.335060 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfcd9\" (UniqueName: \"kubernetes.io/projected/7a35dab5-bed1-4d54-95d0-370ccf127c4b-kube-api-access-bfcd9\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:32:58.335571 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:58.335536 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f56cfd10-4ad6-4c14-b614-ae3b48b87dbe-workload-certs" pod "f56cfd10-4ad6-4c14-b614-ae3b48b87dbe" (UID: "f56cfd10-4ad6-4c14-b614-ae3b48b87dbe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:32:58.335914 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:58.335887 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f56cfd10-4ad6-4c14-b614-ae3b48b87dbe-workload-socket" pod "f56cfd10-4ad6-4c14-b614-ae3b48b87dbe" (UID: "f56cfd10-4ad6-4c14-b614-ae3b48b87dbe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:32:58.337713 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:58.337669 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f56cfd10-4ad6-4c14-b614-ae3b48b87dbe-kube-api-access-8ssg9" pod "f56cfd10-4ad6-4c14-b614-ae3b48b87dbe" (UID: "f56cfd10-4ad6-4c14-b614-ae3b48b87dbe"). InnerVolumeSpecName "kube-api-access-8ssg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:32:58.337713 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:58.337688 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f56cfd10-4ad6-4c14-b614-ae3b48b87dbe-istio-podinfo" pod "f56cfd10-4ad6-4c14-b614-ae3b48b87dbe" (UID: "f56cfd10-4ad6-4c14-b614-ae3b48b87dbe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:32:58.337999 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:58.337974 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f56cfd10-4ad6-4c14-b614-ae3b48b87dbe-istio-token" pod "f56cfd10-4ad6-4c14-b614-ae3b48b87dbe" (UID: "f56cfd10-4ad6-4c14-b614-ae3b48b87dbe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:32:58.339578 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:58.339555 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f56cfd10-4ad6-4c14-b614-ae3b48b87dbe-istio-envoy" pod "f56cfd10-4ad6-4c14-b614-ae3b48b87dbe" (UID: "f56cfd10-4ad6-4c14-b614-ae3b48b87dbe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:32:58.435950 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:58.435924 4982 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f56cfd10-4ad6-4c14-b614-ae3b48b87dbe-istio-podinfo\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:32:58.436085 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:58.435953 4982 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f56cfd10-4ad6-4c14-b614-ae3b48b87dbe-workload-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:32:58.436085 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:58.435967 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ssg9\" (UniqueName: \"kubernetes.io/projected/f56cfd10-4ad6-4c14-b614-ae3b48b87dbe-kube-api-access-8ssg9\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:32:58.436085 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:58.435983 4982 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f56cfd10-4ad6-4c14-b614-ae3b48b87dbe-workload-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:32:58.436085 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:58.435995 4982 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f56cfd10-4ad6-4c14-b614-ae3b48b87dbe-istio-envoy\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:32:58.436085 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:58.436007 4982 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f56cfd10-4ad6-4c14-b614-ae3b48b87dbe-istio-data\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:32:58.436085 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:58.436018 4982 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f56cfd10-4ad6-4c14-b614-ae3b48b87dbe-istio-token\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:32:58.664575 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:58.664541 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:32:58.814980 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:58.814947 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:32:58.989984 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:58.989934 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0b1c15-t58mk" event={"ID":"7a35dab5-bed1-4d54-95d0-370ccf127c4b","Type":"ContainerDied","Data":"5083626822c8d3ee7d1dec49f30c9616f26177e3fbd2f40346351c86fcda47b9"} Oct 30 14:32:58.990256 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:58.990220 4982 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5083626822c8d3ee7d1dec49f30c9616f26177e3fbd2f40346351c86fcda47b9" Oct 30 14:32:58.990505 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:58.990488 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0b1c15-t58mk" Oct 30 14:32:59.004679 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:58.997331 4982 generic.go:356] "Generic (PLEG): container finished" podID="f56cfd10-4ad6-4c14-b614-ae3b48b87dbe" containerID="dffe6a02a7a9b8f26bd1e813d4a04b50f1970cb1dbe6cc1cdb7998e5cfc012d4" exitCode=0 Oct 30 14:32:59.004679 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:58.997615 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5842c" Oct 30 14:32:59.004679 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:58.998456 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5842c" event={"ID":"f56cfd10-4ad6-4c14-b614-ae3b48b87dbe","Type":"ContainerDied","Data":"dffe6a02a7a9b8f26bd1e813d4a04b50f1970cb1dbe6cc1cdb7998e5cfc012d4"} Oct 30 14:32:59.004679 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:58.998505 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5842c" event={"ID":"f56cfd10-4ad6-4c14-b614-ae3b48b87dbe","Type":"ContainerDied","Data":"3e0cb5a32365a330219ffa4727e432e72e114a1113514db6597d4fe554306eb8"} Oct 30 14:32:59.004679 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:58.998533 4982 scope.go:122] "RemoveContainer" containerID="dffe6a02a7a9b8f26bd1e813d4a04b50f1970cb1dbe6cc1cdb7998e5cfc012d4" Oct 30 14:32:59.069258 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:59.069231 4982 scope.go:122] "RemoveContainer" containerID="dffe6a02a7a9b8f26bd1e813d4a04b50f1970cb1dbe6cc1cdb7998e5cfc012d4" Oct 30 14:32:59.070127 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:32:59.070100 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dffe6a02a7a9b8f26bd1e813d4a04b50f1970cb1dbe6cc1cdb7998e5cfc012d4\": container with ID starting with dffe6a02a7a9b8f26bd1e813d4a04b50f1970cb1dbe6cc1cdb7998e5cfc012d4 not found: ID does not exist" containerID="dffe6a02a7a9b8f26bd1e813d4a04b50f1970cb1dbe6cc1cdb7998e5cfc012d4" Oct 30 14:32:59.070367 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:59.070343 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dffe6a02a7a9b8f26bd1e813d4a04b50f1970cb1dbe6cc1cdb7998e5cfc012d4"} err="failed to get container status \"dffe6a02a7a9b8f26bd1e813d4a04b50f1970cb1dbe6cc1cdb7998e5cfc012d4\": rpc error: code = NotFound desc = could not find container \"dffe6a02a7a9b8f26bd1e813d4a04b50f1970cb1dbe6cc1cdb7998e5cfc012d4\": container with ID starting with dffe6a02a7a9b8f26bd1e813d4a04b50f1970cb1dbe6cc1cdb7998e5cfc012d4 not found: ID does not exist" Oct 30 14:32:59.098369 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:59.098341 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5842c"] Oct 30 14:32:59.112470 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:59.112436 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-5842c"] Oct 30 14:32:59.188725 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:59.188700 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80180acb-ww7mh" Oct 30 14:32:59.259198 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:59.259120 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/260f52cf-e8e0-4122-9818-f9e8a149cb55-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/260f52cf-e8e0-4122-9818-f9e8a149cb55-maas-api-openshift-ca\") pod \"260f52cf-e8e0-4122-9818-f9e8a149cb55\" (UID: \"260f52cf-e8e0-4122-9818-f9e8a149cb55\") " Oct 30 14:32:59.259350 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:59.259287 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/260f52cf-e8e0-4122-9818-f9e8a149cb55-kube-api-access-tgq8m\" (UniqueName: \"kubernetes.io/projected/260f52cf-e8e0-4122-9818-f9e8a149cb55-kube-api-access-tgq8m\") pod \"260f52cf-e8e0-4122-9818-f9e8a149cb55\" (UID: \"260f52cf-e8e0-4122-9818-f9e8a149cb55\") " Oct 30 14:32:59.259498 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:59.259475 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/260f52cf-e8e0-4122-9818-f9e8a149cb55-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/260f52cf-e8e0-4122-9818-f9e8a149cb55-maas-api-xks-ca\") pod \"260f52cf-e8e0-4122-9818-f9e8a149cb55\" (UID: \"260f52cf-e8e0-4122-9818-f9e8a149cb55\") " Oct 30 14:32:59.259579 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:59.259540 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/260f52cf-e8e0-4122-9818-f9e8a149cb55-maas-api-openshift-ca" pod "260f52cf-e8e0-4122-9818-f9e8a149cb55" (UID: "260f52cf-e8e0-4122-9818-f9e8a149cb55"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:32:59.260115 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:59.260088 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/260f52cf-e8e0-4122-9818-f9e8a149cb55-maas-api-openshift-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:32:59.261477 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:59.261447 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/260f52cf-e8e0-4122-9818-f9e8a149cb55-kube-api-access-tgq8m" pod "260f52cf-e8e0-4122-9818-f9e8a149cb55" (UID: "260f52cf-e8e0-4122-9818-f9e8a149cb55"). InnerVolumeSpecName "kube-api-access-tgq8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:32:59.261602 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:59.261542 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/260f52cf-e8e0-4122-9818-f9e8a149cb55-maas-api-xks-ca" pod "260f52cf-e8e0-4122-9818-f9e8a149cb55" (UID: "260f52cf-e8e0-4122-9818-f9e8a149cb55"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:32:59.361243 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:59.361209 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgq8m\" (UniqueName: \"kubernetes.io/projected/260f52cf-e8e0-4122-9818-f9e8a149cb55-kube-api-access-tgq8m\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:32:59.361441 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:59.361252 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/260f52cf-e8e0-4122-9818-f9e8a149cb55-maas-api-xks-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:32:59.924931 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:59.924898 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:33:00.000031 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:32:59.999996 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp"] Oct 30 14:33:00.000306 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:00.000252 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp" podUID="e3ffe590-102f-420e-b108-3d6ea542c0b4" containerName="istio-proxy" containerID="cri-o://ebe47a4c527369313c27c3c37b1355d5191b33bd6ee24038ed99f1e40402f0cd" gracePeriod=30 Oct 30 14:33:00.009003 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:00.008975 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80180acb-ww7mh" Oct 30 14:33:00.009130 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:00.008999 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80180acb-ww7mh" event={"ID":"260f52cf-e8e0-4122-9818-f9e8a149cb55","Type":"ContainerDied","Data":"300ce0cddcf4b5238b2e32e7e18e720216777be696d1381c6d8f589133d309e6"} Oct 30 14:33:00.009130 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:00.009048 4982 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="300ce0cddcf4b5238b2e32e7e18e720216777be696d1381c6d8f589133d309e6" Oct 30 14:33:00.401945 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:00.401913 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f56cfd10-4ad6-4c14-b614-ae3b48b87dbe" path="/var/lib/kubelet/pods/f56cfd10-4ad6-4c14-b614-ae3b48b87dbe/volumes" Oct 30 14:33:00.563380 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:00.563344 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr"] Oct 30 14:33:00.563959 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:00.563943 4982 cpu_manager.go:414] "containerMap: removing container" podUID="f56cfd10-4ad6-4c14-b614-ae3b48b87dbe" containerName="istio-proxy" Oct 30 14:33:00.564015 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:00.563960 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f56cfd10-4ad6-4c14-b614-ae3b48b87dbe" containerName="istio-proxy" Oct 30 14:33:00.564015 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:00.563988 4982 cpu_manager.go:414] "containerMap: removing container" podUID="260f52cf-e8e0-4122-9818-f9e8a149cb55" containerName="revoke-keys" Oct 30 14:33:00.564015 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:00.563994 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="260f52cf-e8e0-4122-9818-f9e8a149cb55" containerName="revoke-keys" Oct 30 14:33:00.564015 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:00.564006 4982 cpu_manager.go:414] "containerMap: removing container" podUID="7a35dab5-bed1-4d54-95d0-370ccf127c4b" containerName="revoke-keys" Oct 30 14:33:00.564015 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:00.564012 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a35dab5-bed1-4d54-95d0-370ccf127c4b" containerName="revoke-keys" Oct 30 14:33:00.564162 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:00.564109 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="f56cfd10-4ad6-4c14-b614-ae3b48b87dbe" containerName="istio-proxy" Oct 30 14:33:00.564162 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:00.564122 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="260f52cf-e8e0-4122-9818-f9e8a149cb55" containerName="revoke-keys" Oct 30 14:33:00.564162 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:00.564128 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a35dab5-bed1-4d54-95d0-370ccf127c4b" containerName="revoke-keys" Oct 30 14:33:00.568899 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:00.568876 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr" Oct 30 14:33:00.592782 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:00.592742 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr"] Oct 30 14:33:00.600419 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:00.600393 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-e371c3-openshift-default-dockercfg-lb6vc\"" Oct 30 14:33:00.676959 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:00.676885 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e1d6dd2-aa38-431e-aacc-7ba23fd2310d-credential-socket\") pod \"e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr\" (UID: \"9e1d6dd2-aa38-431e-aacc-7ba23fd2310d\") " pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr" Oct 30 14:33:00.676959 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:00.676928 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e1d6dd2-aa38-431e-aacc-7ba23fd2310d-istio-token\") pod \"e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr\" (UID: \"9e1d6dd2-aa38-431e-aacc-7ba23fd2310d\") " pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr" Oct 30 14:33:00.677097 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:00.677014 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e1d6dd2-aa38-431e-aacc-7ba23fd2310d-istiod-ca-cert\") pod \"e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr\" (UID: \"9e1d6dd2-aa38-431e-aacc-7ba23fd2310d\") " pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr" Oct 30 14:33:00.677145 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:00.677109 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e1d6dd2-aa38-431e-aacc-7ba23fd2310d-istio-envoy\") pod \"e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr\" (UID: \"9e1d6dd2-aa38-431e-aacc-7ba23fd2310d\") " pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr" Oct 30 14:33:00.677185 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:00.677143 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdmpk\" (UniqueName: \"kubernetes.io/projected/9e1d6dd2-aa38-431e-aacc-7ba23fd2310d-kube-api-access-zdmpk\") pod \"e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr\" (UID: \"9e1d6dd2-aa38-431e-aacc-7ba23fd2310d\") " pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr" Oct 30 14:33:00.677236 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:00.677225 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e1d6dd2-aa38-431e-aacc-7ba23fd2310d-workload-certs\") pod \"e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr\" (UID: \"9e1d6dd2-aa38-431e-aacc-7ba23fd2310d\") " pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr" Oct 30 14:33:00.677282 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:00.677271 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e1d6dd2-aa38-431e-aacc-7ba23fd2310d-istio-podinfo\") pod \"e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr\" (UID: \"9e1d6dd2-aa38-431e-aacc-7ba23fd2310d\") " pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr" Oct 30 14:33:00.677409 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:00.677387 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e1d6dd2-aa38-431e-aacc-7ba23fd2310d-istio-data\") pod \"e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr\" (UID: \"9e1d6dd2-aa38-431e-aacc-7ba23fd2310d\") " pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr" Oct 30 14:33:00.677480 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:00.677468 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e1d6dd2-aa38-431e-aacc-7ba23fd2310d-workload-socket\") pod \"e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr\" (UID: \"9e1d6dd2-aa38-431e-aacc-7ba23fd2310d\") " pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr" Oct 30 14:33:00.779164 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:00.779137 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e1d6dd2-aa38-431e-aacc-7ba23fd2310d-istio-envoy\") pod \"e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr\" (UID: \"9e1d6dd2-aa38-431e-aacc-7ba23fd2310d\") " pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr" Oct 30 14:33:00.779280 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:00.779181 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdmpk\" (UniqueName: \"kubernetes.io/projected/9e1d6dd2-aa38-431e-aacc-7ba23fd2310d-kube-api-access-zdmpk\") pod \"e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr\" (UID: \"9e1d6dd2-aa38-431e-aacc-7ba23fd2310d\") " pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr" Oct 30 14:33:00.779280 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:00.779223 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e1d6dd2-aa38-431e-aacc-7ba23fd2310d-workload-certs\") pod \"e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr\" (UID: \"9e1d6dd2-aa38-431e-aacc-7ba23fd2310d\") " pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr" Oct 30 14:33:00.779280 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:00.779256 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e1d6dd2-aa38-431e-aacc-7ba23fd2310d-istio-podinfo\") pod \"e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr\" (UID: \"9e1d6dd2-aa38-431e-aacc-7ba23fd2310d\") " pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr" Oct 30 14:33:00.779420 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:00.779391 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e1d6dd2-aa38-431e-aacc-7ba23fd2310d-istio-data\") pod \"e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr\" (UID: \"9e1d6dd2-aa38-431e-aacc-7ba23fd2310d\") " pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr" Oct 30 14:33:00.779507 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:00.779492 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e1d6dd2-aa38-431e-aacc-7ba23fd2310d-workload-socket\") pod \"e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr\" (UID: \"9e1d6dd2-aa38-431e-aacc-7ba23fd2310d\") " pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr" Oct 30 14:33:00.779668 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:00.779588 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e1d6dd2-aa38-431e-aacc-7ba23fd2310d-credential-socket\") pod \"e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr\" (UID: \"9e1d6dd2-aa38-431e-aacc-7ba23fd2310d\") " pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr" Oct 30 14:33:00.779668 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:00.779622 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e1d6dd2-aa38-431e-aacc-7ba23fd2310d-workload-certs\") pod \"e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr\" (UID: \"9e1d6dd2-aa38-431e-aacc-7ba23fd2310d\") " pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr" Oct 30 14:33:00.779818 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:00.779624 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e1d6dd2-aa38-431e-aacc-7ba23fd2310d-istio-token\") pod \"e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr\" (UID: \"9e1d6dd2-aa38-431e-aacc-7ba23fd2310d\") " pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr" Oct 30 14:33:00.779818 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:00.779807 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e1d6dd2-aa38-431e-aacc-7ba23fd2310d-istiod-ca-cert\") pod \"e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr\" (UID: \"9e1d6dd2-aa38-431e-aacc-7ba23fd2310d\") " pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr" Oct 30 14:33:00.780215 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:00.780186 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e1d6dd2-aa38-431e-aacc-7ba23fd2310d-workload-socket\") pod \"e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr\" (UID: \"9e1d6dd2-aa38-431e-aacc-7ba23fd2310d\") " pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr" Oct 30 14:33:00.780687 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:00.780633 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e1d6dd2-aa38-431e-aacc-7ba23fd2310d-istiod-ca-cert\") pod \"e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr\" (UID: \"9e1d6dd2-aa38-431e-aacc-7ba23fd2310d\") " pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr" Oct 30 14:33:00.780774 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:00.780715 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e1d6dd2-aa38-431e-aacc-7ba23fd2310d-istio-data\") pod \"e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr\" (UID: \"9e1d6dd2-aa38-431e-aacc-7ba23fd2310d\") " pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr" Oct 30 14:33:00.781136 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:00.781115 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e1d6dd2-aa38-431e-aacc-7ba23fd2310d-credential-socket\") pod \"e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr\" (UID: \"9e1d6dd2-aa38-431e-aacc-7ba23fd2310d\") " pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr" Oct 30 14:33:00.781545 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:00.781519 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e1d6dd2-aa38-431e-aacc-7ba23fd2310d-istio-envoy\") pod \"e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr\" (UID: \"9e1d6dd2-aa38-431e-aacc-7ba23fd2310d\") " pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr" Oct 30 14:33:00.781855 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:00.781833 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e1d6dd2-aa38-431e-aacc-7ba23fd2310d-istio-podinfo\") pod \"e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr\" (UID: \"9e1d6dd2-aa38-431e-aacc-7ba23fd2310d\") " pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr" Oct 30 14:33:00.809400 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:00.809366 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e1d6dd2-aa38-431e-aacc-7ba23fd2310d-istio-token\") pod \"e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr\" (UID: \"9e1d6dd2-aa38-431e-aacc-7ba23fd2310d\") " pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr" Oct 30 14:33:00.809540 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:00.809443 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdmpk\" (UniqueName: \"kubernetes.io/projected/9e1d6dd2-aa38-431e-aacc-7ba23fd2310d-kube-api-access-zdmpk\") pod \"e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr\" (UID: \"9e1d6dd2-aa38-431e-aacc-7ba23fd2310d\") " pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr" Oct 30 14:33:00.897713 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:00.884908 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr" Oct 30 14:33:01.025857 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:01.025824 4982 generic.go:356] "Generic (PLEG): container finished" podID="b473ec20-3b31-418d-87ac-bc1269c41744" containerID="75bd369161fed518765352e20b7d7514b67e39becfd19fb6bd46ca7de3a10bc6" exitCode=0 Oct 30 14:33:01.026329 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:01.026052 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-fde75d-7d8d99ffcb-79hw8" event={"ID":"b473ec20-3b31-418d-87ac-bc1269c41744","Type":"ContainerDied","Data":"75bd369161fed518765352e20b7d7514b67e39becfd19fb6bd46ca7de3a10bc6"} Oct 30 14:33:01.028543 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:01.028518 4982 generic.go:356] "Generic (PLEG): container finished" podID="a7b9a748-bba1-45e2-9612-0fd47cc5f8a6" containerID="c6944f8c7ad6bb968408d0640d78188f7d98c042c5806245ac4c254bcad24437" exitCode=0 Oct 30 14:33:01.028655 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:01.028582 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fde75d-75cd788695-5n8hq" event={"ID":"a7b9a748-bba1-45e2-9612-0fd47cc5f8a6","Type":"ContainerDied","Data":"c6944f8c7ad6bb968408d0640d78188f7d98c042c5806245ac4c254bcad24437"} Oct 30 14:33:01.109462 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:01.109436 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-fde75d-7d8d99ffcb-79hw8" Oct 30 14:33:01.159669 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:01.159607 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fde75d-75cd788695-5n8hq" Oct 30 14:33:01.220632 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:01.220550 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b473ec20-3b31-418d-87ac-bc1269c41744-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b473ec20-3b31-418d-87ac-bc1269c41744-plugins-config-volume\") pod \"b473ec20-3b31-418d-87ac-bc1269c41744\" (UID: \"b473ec20-3b31-418d-87ac-bc1269c41744\") " Oct 30 14:33:01.220813 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:01.220624 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b473ec20-3b31-418d-87ac-bc1269c41744-kube-api-access-hlr6l\" (UniqueName: \"kubernetes.io/projected/b473ec20-3b31-418d-87ac-bc1269c41744-kube-api-access-hlr6l\") pod \"b473ec20-3b31-418d-87ac-bc1269c41744\" (UID: \"b473ec20-3b31-418d-87ac-bc1269c41744\") " Oct 30 14:33:01.221044 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:01.221015 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b473ec20-3b31-418d-87ac-bc1269c41744-plugins-config-volume" pod "b473ec20-3b31-418d-87ac-bc1269c41744" (UID: "b473ec20-3b31-418d-87ac-bc1269c41744"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:33:01.221234 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:01.221209 4982 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b473ec20-3b31-418d-87ac-bc1269c41744-plugins-config-volume\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:33:01.223239 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:01.223212 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b473ec20-3b31-418d-87ac-bc1269c41744-kube-api-access-hlr6l" pod "b473ec20-3b31-418d-87ac-bc1269c41744" (UID: "b473ec20-3b31-418d-87ac-bc1269c41744"). InnerVolumeSpecName "kube-api-access-hlr6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:33:01.306010 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:33:01.305981 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e1d6dd2_aa38_431e_aacc_7ba23fd2310d.slice/crio-50e8bcef0e439121f4c6bcb70a5ed3c5491daad888a1808e94c3bbf631444202 WatchSource:0}: Error finding container 50e8bcef0e439121f4c6bcb70a5ed3c5491daad888a1808e94c3bbf631444202: Status 404 returned error can't find the container with id 50e8bcef0e439121f4c6bcb70a5ed3c5491daad888a1808e94c3bbf631444202 Oct 30 14:33:01.307304 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:01.307277 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr"] Oct 30 14:33:01.309280 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:01.309250 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:33:01.309388 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:01.309312 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:33:01.309388 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:01.309341 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:33:01.322030 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:01.322005 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7b9a748-bba1-45e2-9612-0fd47cc5f8a6-kube-api-access-wl4rj\" (UniqueName: \"kubernetes.io/projected/a7b9a748-bba1-45e2-9612-0fd47cc5f8a6-kube-api-access-wl4rj\") pod \"a7b9a748-bba1-45e2-9612-0fd47cc5f8a6\" (UID: \"a7b9a748-bba1-45e2-9612-0fd47cc5f8a6\") " Oct 30 14:33:01.322233 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:01.322215 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7b9a748-bba1-45e2-9612-0fd47cc5f8a6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7b9a748-bba1-45e2-9612-0fd47cc5f8a6-plugins-config-volume\") pod \"a7b9a748-bba1-45e2-9612-0fd47cc5f8a6\" (UID: \"a7b9a748-bba1-45e2-9612-0fd47cc5f8a6\") " Oct 30 14:33:01.322653 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:01.322613 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7b9a748-bba1-45e2-9612-0fd47cc5f8a6-plugins-config-volume" pod "a7b9a748-bba1-45e2-9612-0fd47cc5f8a6" (UID: "a7b9a748-bba1-45e2-9612-0fd47cc5f8a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:33:01.322792 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:01.322726 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlr6l\" (UniqueName: \"kubernetes.io/projected/b473ec20-3b31-418d-87ac-bc1269c41744-kube-api-access-hlr6l\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:33:01.324339 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:01.324312 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7b9a748-bba1-45e2-9612-0fd47cc5f8a6-kube-api-access-wl4rj" pod "a7b9a748-bba1-45e2-9612-0fd47cc5f8a6" (UID: "a7b9a748-bba1-45e2-9612-0fd47cc5f8a6"). InnerVolumeSpecName "kube-api-access-wl4rj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:33:01.424765 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:01.424726 4982 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7b9a748-bba1-45e2-9612-0fd47cc5f8a6-plugins-config-volume\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:33:01.424765 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:01.424763 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wl4rj\" (UniqueName: \"kubernetes.io/projected/a7b9a748-bba1-45e2-9612-0fd47cc5f8a6-kube-api-access-wl4rj\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:33:01.746461 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:01.746429 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:33:01.949363 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:01.949282 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:33:02.038897 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:02.038870 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fde75d-75cd788695-5n8hq" Oct 30 14:33:02.038897 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:02.038883 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fde75d-75cd788695-5n8hq" event={"ID":"a7b9a748-bba1-45e2-9612-0fd47cc5f8a6","Type":"ContainerDied","Data":"3e17098b66d5c45c98e6c0d6d260aa7c8d4a69741923d46d45613a600798b4f4"} Oct 30 14:33:02.039143 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:02.038944 4982 scope.go:122] "RemoveContainer" containerID="c6944f8c7ad6bb968408d0640d78188f7d98c042c5806245ac4c254bcad24437" Oct 30 14:33:02.040667 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:02.040624 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr" event={"ID":"9e1d6dd2-aa38-431e-aacc-7ba23fd2310d","Type":"ContainerStarted","Data":"8622eda68b8255c5719ae056f4e68765fb911b6188ba3b001f5b9b0d3f4eebed"} Oct 30 14:33:02.040850 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:02.040692 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr" event={"ID":"9e1d6dd2-aa38-431e-aacc-7ba23fd2310d","Type":"ContainerStarted","Data":"50e8bcef0e439121f4c6bcb70a5ed3c5491daad888a1808e94c3bbf631444202"} Oct 30 14:33:02.042204 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:02.042183 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-fde75d-7d8d99ffcb-79hw8" Oct 30 14:33:02.042319 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:02.042183 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-fde75d-7d8d99ffcb-79hw8" event={"ID":"b473ec20-3b31-418d-87ac-bc1269c41744","Type":"ContainerDied","Data":"c6b03550b21a1d631c47441206a40be8999b21376f1ce8081ccbd8fbc005c3fc"} Oct 30 14:33:02.049687 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:02.049671 4982 scope.go:122] "RemoveContainer" containerID="75bd369161fed518765352e20b7d7514b67e39becfd19fb6bd46ca7de3a10bc6" Oct 30 14:33:02.097989 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:02.097962 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:33:02.099111 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:02.099091 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fde75d-7d8d99ffcb-79hw8"] Oct 30 14:33:02.104501 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:02.104477 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fde75d-7d8d99ffcb-79hw8"] Oct 30 14:33:02.148665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:02.147720 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr" podStartSLOduration=2.147700828 podStartE2EDuration="2.147700828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:33:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:33:02.146498992 +0000 UTC m=+2651.890633035" watchObservedRunningTime="2026-10-30 14:33:02.147700828 +0000 UTC m=+2651.891834869" Oct 30 14:33:02.196671 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:02.195690 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fde75d-75cd788695-5n8hq"] Oct 30 14:33:02.201711 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:02.199229 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fde75d-75cd788695-5n8hq"] Oct 30 14:33:02.421423 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:02.421381 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7b9a748-bba1-45e2-9612-0fd47cc5f8a6" path="/var/lib/kubelet/pods/a7b9a748-bba1-45e2-9612-0fd47cc5f8a6/volumes" Oct 30 14:33:02.422069 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:02.422044 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b473ec20-3b31-418d-87ac-bc1269c41744" path="/var/lib/kubelet/pods/b473ec20-3b31-418d-87ac-bc1269c41744/volumes" Oct 30 14:33:02.902547 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:02.902018 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr" Oct 30 14:33:02.911105 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:02.910770 4982 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 14:33:02.911105 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:02.910836 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr" podUID="9e1d6dd2-aa38-431e-aacc-7ba23fd2310d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 14:33:03.351015 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:03.350979 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:33:03.908663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:03.907756 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:33:03.925664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:03.909458 4982 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 14:33:03.925664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:03.909512 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr" podUID="9e1d6dd2-aa38-431e-aacc-7ba23fd2310d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 14:33:04.034685 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:04.034634 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b9c4b4-qn6wb"] Oct 30 14:33:04.046404 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:04.046374 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b9c4b4-qn6wb"] Oct 30 14:33:04.059230 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:04.059202 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:33:04.261184 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:04.261102 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:33:04.422258 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:04.422226 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d729832-e94c-4dc1-95cb-b67bbe754469" path="/var/lib/kubelet/pods/9d729832-e94c-4dc1-95cb-b67bbe754469/volumes" Oct 30 14:33:04.567392 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:04.567357 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:33:04.909745 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:04.902048 4982 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 14:33:04.909745 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:04.902109 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr" podUID="9e1d6dd2-aa38-431e-aacc-7ba23fd2310d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 14:33:05.358883 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:05.357668 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp" Oct 30 14:33:05.466519 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:05.466430 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:33:05.467287 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:05.467102 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3ffe590-102f-420e-b108-3d6ea542c0b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3ffe590-102f-420e-b108-3d6ea542c0b4-workload-certs\") pod \"e3ffe590-102f-420e-b108-3d6ea542c0b4\" (UID: \"e3ffe590-102f-420e-b108-3d6ea542c0b4\") " Oct 30 14:33:05.467287 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:05.467183 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3ffe590-102f-420e-b108-3d6ea542c0b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3ffe590-102f-420e-b108-3d6ea542c0b4-istiod-ca-cert\") pod \"e3ffe590-102f-420e-b108-3d6ea542c0b4\" (UID: \"e3ffe590-102f-420e-b108-3d6ea542c0b4\") " Oct 30 14:33:05.467287 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:05.467216 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3ffe590-102f-420e-b108-3d6ea542c0b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ffe590-102f-420e-b108-3d6ea542c0b4-workload-socket\") pod \"e3ffe590-102f-420e-b108-3d6ea542c0b4\" (UID: \"e3ffe590-102f-420e-b108-3d6ea542c0b4\") " Oct 30 14:33:05.467287 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:05.467270 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3ffe590-102f-420e-b108-3d6ea542c0b4-istio-token\" (UniqueName: \"kubernetes.io/projected/e3ffe590-102f-420e-b108-3d6ea542c0b4-istio-token\") pod \"e3ffe590-102f-420e-b108-3d6ea542c0b4\" (UID: \"e3ffe590-102f-420e-b108-3d6ea542c0b4\") " Oct 30 14:33:05.467596 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:05.467326 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3ffe590-102f-420e-b108-3d6ea542c0b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3ffe590-102f-420e-b108-3d6ea542c0b4-istio-data\") pod \"e3ffe590-102f-420e-b108-3d6ea542c0b4\" (UID: \"e3ffe590-102f-420e-b108-3d6ea542c0b4\") " Oct 30 14:33:05.467596 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:05.467425 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3ffe590-102f-420e-b108-3d6ea542c0b4-kube-api-access-ljpq5\" (UniqueName: \"kubernetes.io/projected/e3ffe590-102f-420e-b108-3d6ea542c0b4-kube-api-access-ljpq5\") pod \"e3ffe590-102f-420e-b108-3d6ea542c0b4\" (UID: \"e3ffe590-102f-420e-b108-3d6ea542c0b4\") " Oct 30 14:33:05.467596 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:05.467531 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3ffe590-102f-420e-b108-3d6ea542c0b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3ffe590-102f-420e-b108-3d6ea542c0b4-istio-envoy\") pod \"e3ffe590-102f-420e-b108-3d6ea542c0b4\" (UID: \"e3ffe590-102f-420e-b108-3d6ea542c0b4\") " Oct 30 14:33:05.467596 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:05.467583 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e3ffe590-102f-420e-b108-3d6ea542c0b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3ffe590-102f-420e-b108-3d6ea542c0b4-istio-podinfo\") pod \"e3ffe590-102f-420e-b108-3d6ea542c0b4\" (UID: \"e3ffe590-102f-420e-b108-3d6ea542c0b4\") " Oct 30 14:33:05.467873 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:05.467627 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3ffe590-102f-420e-b108-3d6ea542c0b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ffe590-102f-420e-b108-3d6ea542c0b4-credential-socket\") pod \"e3ffe590-102f-420e-b108-3d6ea542c0b4\" (UID: \"e3ffe590-102f-420e-b108-3d6ea542c0b4\") " Oct 30 14:33:05.468689 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:05.468658 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3ffe590-102f-420e-b108-3d6ea542c0b4-workload-socket" pod "e3ffe590-102f-420e-b108-3d6ea542c0b4" (UID: "e3ffe590-102f-420e-b108-3d6ea542c0b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:33:05.468959 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:05.468936 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3ffe590-102f-420e-b108-3d6ea542c0b4-istiod-ca-cert" pod "e3ffe590-102f-420e-b108-3d6ea542c0b4" (UID: "e3ffe590-102f-420e-b108-3d6ea542c0b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:33:05.469330 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:05.469278 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3ffe590-102f-420e-b108-3d6ea542c0b4-istio-data" pod "e3ffe590-102f-420e-b108-3d6ea542c0b4" (UID: "e3ffe590-102f-420e-b108-3d6ea542c0b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:33:05.469330 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:05.469316 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3ffe590-102f-420e-b108-3d6ea542c0b4-workload-certs" pod "e3ffe590-102f-420e-b108-3d6ea542c0b4" (UID: "e3ffe590-102f-420e-b108-3d6ea542c0b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:33:05.469507 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:05.469373 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3ffe590-102f-420e-b108-3d6ea542c0b4-credential-socket" pod "e3ffe590-102f-420e-b108-3d6ea542c0b4" (UID: "e3ffe590-102f-420e-b108-3d6ea542c0b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:33:05.477570 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:05.477371 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3ffe590-102f-420e-b108-3d6ea542c0b4-istio-podinfo" pod "e3ffe590-102f-420e-b108-3d6ea542c0b4" (UID: "e3ffe590-102f-420e-b108-3d6ea542c0b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:33:05.477570 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:05.477476 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3ffe590-102f-420e-b108-3d6ea542c0b4-istio-envoy" pod "e3ffe590-102f-420e-b108-3d6ea542c0b4" (UID: "e3ffe590-102f-420e-b108-3d6ea542c0b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:33:05.501912 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:05.501849 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3ffe590-102f-420e-b108-3d6ea542c0b4-istio-token" pod "e3ffe590-102f-420e-b108-3d6ea542c0b4" (UID: "e3ffe590-102f-420e-b108-3d6ea542c0b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:33:05.523873 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:05.523838 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3ffe590-102f-420e-b108-3d6ea542c0b4-kube-api-access-ljpq5" pod "e3ffe590-102f-420e-b108-3d6ea542c0b4" (UID: "e3ffe590-102f-420e-b108-3d6ea542c0b4"). InnerVolumeSpecName "kube-api-access-ljpq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:33:05.540698 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:05.540634 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b9c4b4-6498846c67-q52pq" podUID="b87e2746-abcd-4e8a-bda1-cf9badd05b04" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:33:05.568940 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:05.568893 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljpq5\" (UniqueName: \"kubernetes.io/projected/e3ffe590-102f-420e-b108-3d6ea542c0b4-kube-api-access-ljpq5\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:33:05.568940 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:05.568934 4982 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3ffe590-102f-420e-b108-3d6ea542c0b4-istio-envoy\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:33:05.569168 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:05.568952 4982 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3ffe590-102f-420e-b108-3d6ea542c0b4-istio-podinfo\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:33:05.569168 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:05.568964 4982 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ffe590-102f-420e-b108-3d6ea542c0b4-credential-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:33:05.569168 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:05.568975 4982 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3ffe590-102f-420e-b108-3d6ea542c0b4-workload-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:33:05.569168 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:05.568988 4982 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3ffe590-102f-420e-b108-3d6ea542c0b4-istiod-ca-cert\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:33:05.569168 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:05.568999 4982 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ffe590-102f-420e-b108-3d6ea542c0b4-workload-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:33:05.569168 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:05.569010 4982 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3ffe590-102f-420e-b108-3d6ea542c0b4-istio-token\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:33:05.569168 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:05.569021 4982 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3ffe590-102f-420e-b108-3d6ea542c0b4-istio-data\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:33:05.612161 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:05.612113 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:33:05.902666 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:05.899921 4982 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 14:33:05.902666 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:05.899983 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr" podUID="9e1d6dd2-aa38-431e-aacc-7ba23fd2310d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 14:33:06.015566 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:06.015528 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:33:06.159716 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:06.159457 4982 generic.go:356] "Generic (PLEG): container finished" podID="e3ffe590-102f-420e-b108-3d6ea542c0b4" containerID="ebe47a4c527369313c27c3c37b1355d5191b33bd6ee24038ed99f1e40402f0cd" exitCode=0 Oct 30 14:33:06.166658 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:06.161989 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp" event={"ID":"e3ffe590-102f-420e-b108-3d6ea542c0b4","Type":"ContainerDied","Data":"ebe47a4c527369313c27c3c37b1355d5191b33bd6ee24038ed99f1e40402f0cd"} Oct 30 14:33:06.166658 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:06.162057 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp" event={"ID":"e3ffe590-102f-420e-b108-3d6ea542c0b4","Type":"ContainerDied","Data":"e1dfee73ad96fb4293cf143176a1b9292f43a26cf52b51853863e13387cdb3f9"} Oct 30 14:33:06.166658 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:06.162088 4982 scope.go:122] "RemoveContainer" containerID="ebe47a4c527369313c27c3c37b1355d5191b33bd6ee24038ed99f1e40402f0cd" Oct 30 14:33:06.166658 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:06.162787 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp" Oct 30 14:33:06.198659 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:06.192954 4982 scope.go:122] "RemoveContainer" containerID="ebe47a4c527369313c27c3c37b1355d5191b33bd6ee24038ed99f1e40402f0cd" Oct 30 14:33:06.217663 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:33:06.212758 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebe47a4c527369313c27c3c37b1355d5191b33bd6ee24038ed99f1e40402f0cd\": container with ID starting with ebe47a4c527369313c27c3c37b1355d5191b33bd6ee24038ed99f1e40402f0cd not found: ID does not exist" containerID="ebe47a4c527369313c27c3c37b1355d5191b33bd6ee24038ed99f1e40402f0cd" Oct 30 14:33:06.217663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:06.212807 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebe47a4c527369313c27c3c37b1355d5191b33bd6ee24038ed99f1e40402f0cd"} err="failed to get container status \"ebe47a4c527369313c27c3c37b1355d5191b33bd6ee24038ed99f1e40402f0cd\": rpc error: code = NotFound desc = could not find container \"ebe47a4c527369313c27c3c37b1355d5191b33bd6ee24038ed99f1e40402f0cd\": container with ID starting with ebe47a4c527369313c27c3c37b1355d5191b33bd6ee24038ed99f1e40402f0cd not found: ID does not exist" Oct 30 14:33:06.239903 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:06.239874 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp"] Oct 30 14:33:06.253328 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:06.253307 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-t5xdp"] Oct 30 14:33:06.256521 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:33:06.256497 4982 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3ffe590_102f_420e_b108_3d6ea542c0b4.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3ffe590_102f_420e_b108_3d6ea542c0b4.slice/crio-e1dfee73ad96fb4293cf143176a1b9292f43a26cf52b51853863e13387cdb3f9\": RecentStats: unable to find data in memory cache]" Oct 30 14:33:06.433540 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:06.433442 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:33:06.494727 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:06.494693 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3ffe590-102f-420e-b108-3d6ea542c0b4" path="/var/lib/kubelet/pods/e3ffe590-102f-420e-b108-3d6ea542c0b4/volumes" Oct 30 14:33:06.906663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:06.906095 4982 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 14:33:06.906663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:06.906160 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr" podUID="9e1d6dd2-aa38-431e-aacc-7ba23fd2310d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 14:33:06.917970 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:06.910621 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e25b4aa-mg4n9"] Oct 30 14:33:06.922154 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:06.922117 4982 cpu_manager.go:414] "containerMap: removing container" podUID="a7b9a748-bba1-45e2-9612-0fd47cc5f8a6" containerName="payload-pre-processing" Oct 30 14:33:06.929925 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:06.922790 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7b9a748-bba1-45e2-9612-0fd47cc5f8a6" containerName="payload-pre-processing" Oct 30 14:33:06.929925 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:06.923030 4982 cpu_manager.go:414] "containerMap: removing container" podUID="b473ec20-3b31-418d-87ac-bc1269c41744" containerName="payload-processing" Oct 30 14:33:06.929925 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:06.923045 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b473ec20-3b31-418d-87ac-bc1269c41744" containerName="payload-processing" Oct 30 14:33:06.929925 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:06.923073 4982 cpu_manager.go:414] "containerMap: removing container" podUID="e3ffe590-102f-420e-b108-3d6ea542c0b4" containerName="istio-proxy" Oct 30 14:33:06.929925 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:06.923087 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3ffe590-102f-420e-b108-3d6ea542c0b4" containerName="istio-proxy" Oct 30 14:33:06.929925 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:06.923717 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="b473ec20-3b31-418d-87ac-bc1269c41744" containerName="payload-processing" Oct 30 14:33:06.929925 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:06.923740 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3ffe590-102f-420e-b108-3d6ea542c0b4" containerName="istio-proxy" Oct 30 14:33:06.929925 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:06.923773 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7b9a748-bba1-45e2-9612-0fd47cc5f8a6" containerName="payload-pre-processing" Oct 30 14:33:06.936752 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:06.934154 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e25b4aa-mg4n9"] Oct 30 14:33:06.936752 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:06.934348 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e25b4aa-mg4n9" Oct 30 14:33:06.955334 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:06.954459 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:33:06.955334 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:06.954920 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:33:07.035048 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:07.035007 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4be53bee-e002-4021-b073-f1609e19a1fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e25b4aa-mg4n9\" (UID: \"4be53bee-e002-4021-b073-f1609e19a1fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e25b4aa-mg4n9" Oct 30 14:33:07.035429 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:07.035172 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4be53bee-e002-4021-b073-f1609e19a1fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e25b4aa-mg4n9\" (UID: \"4be53bee-e002-4021-b073-f1609e19a1fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e25b4aa-mg4n9" Oct 30 14:33:07.035429 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:07.035227 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hczb\" (UniqueName: \"kubernetes.io/projected/4be53bee-e002-4021-b073-f1609e19a1fd-kube-api-access-2hczb\") pod \"maas-api-revoke-sub-4e25b4aa-mg4n9\" (UID: \"4be53bee-e002-4021-b073-f1609e19a1fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e25b4aa-mg4n9" Oct 30 14:33:07.137658 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:07.137029 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4be53bee-e002-4021-b073-f1609e19a1fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e25b4aa-mg4n9\" (UID: \"4be53bee-e002-4021-b073-f1609e19a1fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e25b4aa-mg4n9" Oct 30 14:33:07.137658 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:07.137105 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hczb\" (UniqueName: \"kubernetes.io/projected/4be53bee-e002-4021-b073-f1609e19a1fd-kube-api-access-2hczb\") pod \"maas-api-revoke-sub-4e25b4aa-mg4n9\" (UID: \"4be53bee-e002-4021-b073-f1609e19a1fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e25b4aa-mg4n9" Oct 30 14:33:07.137658 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:07.137218 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4be53bee-e002-4021-b073-f1609e19a1fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e25b4aa-mg4n9\" (UID: \"4be53bee-e002-4021-b073-f1609e19a1fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e25b4aa-mg4n9" Oct 30 14:33:07.138466 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:07.137920 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4be53bee-e002-4021-b073-f1609e19a1fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e25b4aa-mg4n9\" (UID: \"4be53bee-e002-4021-b073-f1609e19a1fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e25b4aa-mg4n9" Oct 30 14:33:07.140265 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:07.140236 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4be53bee-e002-4021-b073-f1609e19a1fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e25b4aa-mg4n9\" (UID: \"4be53bee-e002-4021-b073-f1609e19a1fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e25b4aa-mg4n9" Oct 30 14:33:07.164967 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:07.164892 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hczb\" (UniqueName: \"kubernetes.io/projected/4be53bee-e002-4021-b073-f1609e19a1fd-kube-api-access-2hczb\") pod \"maas-api-revoke-sub-4e25b4aa-mg4n9\" (UID: \"4be53bee-e002-4021-b073-f1609e19a1fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e25b4aa-mg4n9" Oct 30 14:33:07.257589 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:07.257550 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e25b4aa-mg4n9" Oct 30 14:33:07.605342 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:33:07.605302 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4be53bee_e002_4021_b073_f1609e19a1fd.slice/crio-f5237056509aac36c281c4ea7da46ec599d1fb6ea127084bc89a5cd00169b0c5 WatchSource:0}: Error finding container f5237056509aac36c281c4ea7da46ec599d1fb6ea127084bc89a5cd00169b0c5: Status 404 returned error can't find the container with id f5237056509aac36c281c4ea7da46ec599d1fb6ea127084bc89a5cd00169b0c5 Oct 30 14:33:07.606326 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:07.606298 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e25b4aa-mg4n9"] Oct 30 14:33:07.899173 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:07.899141 4982 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 14:33:07.899300 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:07.899192 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr" podUID="9e1d6dd2-aa38-431e-aacc-7ba23fd2310d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 14:33:07.948460 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:07.948426 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:33:08.183374 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:08.183280 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e25b4aa-mg4n9" event={"ID":"4be53bee-e002-4021-b073-f1609e19a1fd","Type":"ContainerStarted","Data":"7c8cdaa77d121b2b75e2b8ab3ae5d2c77b28e0253583ba6cf28c9d36915cdcb3"} Oct 30 14:33:08.183374 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:08.183348 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e25b4aa-mg4n9" event={"ID":"4be53bee-e002-4021-b073-f1609e19a1fd","Type":"ContainerStarted","Data":"f5237056509aac36c281c4ea7da46ec599d1fb6ea127084bc89a5cd00169b0c5"} Oct 30 14:33:08.228223 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:08.228172 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e25b4aa-mg4n9" podStartSLOduration=2.22815862 podStartE2EDuration="2.22815862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:33:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:33:08.226240233 +0000 UTC m=+2657.970374273" watchObservedRunningTime="2026-10-30 14:33:08.22815862 +0000 UTC m=+2657.972292660" Oct 30 14:33:08.904129 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:08.904096 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr" Oct 30 14:33:08.904527 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:08.904500 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr" Oct 30 14:33:08.905398 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:08.905372 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr" Oct 30 14:33:08.990376 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:08.990342 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:33:09.195251 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:09.195166 4982 generic.go:356] "Generic (PLEG): container finished" podID="4be53bee-e002-4021-b073-f1609e19a1fd" containerID="7c8cdaa77d121b2b75e2b8ab3ae5d2c77b28e0253583ba6cf28c9d36915cdcb3" exitCode=0 Oct 30 14:33:09.196077 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:09.196044 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e25b4aa-mg4n9" event={"ID":"4be53bee-e002-4021-b073-f1609e19a1fd","Type":"ContainerDied","Data":"7c8cdaa77d121b2b75e2b8ab3ae5d2c77b28e0253583ba6cf28c9d36915cdcb3"} Oct 30 14:33:09.735704 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:09.731493 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:33:09.844578 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:09.844551 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b9c4b4-6498846c67-q52pq" Oct 30 14:33:09.974293 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:09.974261 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b87e2746-abcd-4e8a-bda1-cf9badd05b04-kube-api-access-26gw5\" (UniqueName: \"kubernetes.io/projected/b87e2746-abcd-4e8a-bda1-cf9badd05b04-kube-api-access-26gw5\") pod \"b87e2746-abcd-4e8a-bda1-cf9badd05b04\" (UID: \"b87e2746-abcd-4e8a-bda1-cf9badd05b04\") " Oct 30 14:33:09.974444 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:09.974418 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b87e2746-abcd-4e8a-bda1-cf9badd05b04-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b87e2746-abcd-4e8a-bda1-cf9badd05b04-plugins-config-volume\") pod \"b87e2746-abcd-4e8a-bda1-cf9badd05b04\" (UID: \"b87e2746-abcd-4e8a-bda1-cf9badd05b04\") " Oct 30 14:33:09.974861 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:09.974831 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b87e2746-abcd-4e8a-bda1-cf9badd05b04-plugins-config-volume" pod "b87e2746-abcd-4e8a-bda1-cf9badd05b04" (UID: "b87e2746-abcd-4e8a-bda1-cf9badd05b04"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:33:09.976389 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:09.976367 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b87e2746-abcd-4e8a-bda1-cf9badd05b04-kube-api-access-26gw5" pod "b87e2746-abcd-4e8a-bda1-cf9badd05b04" (UID: "b87e2746-abcd-4e8a-bda1-cf9badd05b04"). InnerVolumeSpecName "kube-api-access-26gw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:33:10.076493 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:10.076418 4982 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b87e2746-abcd-4e8a-bda1-cf9badd05b04-plugins-config-volume\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:33:10.076493 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:10.076448 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26gw5\" (UniqueName: \"kubernetes.io/projected/b87e2746-abcd-4e8a-bda1-cf9badd05b04-kube-api-access-26gw5\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:33:10.205752 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:10.205715 4982 generic.go:356] "Generic (PLEG): container finished" podID="b87e2746-abcd-4e8a-bda1-cf9badd05b04" containerID="7daa8053c072645197f3bec6f19ffed89e19e22e92a3eacdf53da4fb3788d231" exitCode=0 Oct 30 14:33:10.206217 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:10.205763 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b9c4b4-6498846c67-q52pq" Oct 30 14:33:10.206217 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:10.205792 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b9c4b4-6498846c67-q52pq" event={"ID":"b87e2746-abcd-4e8a-bda1-cf9badd05b04","Type":"ContainerDied","Data":"7daa8053c072645197f3bec6f19ffed89e19e22e92a3eacdf53da4fb3788d231"} Oct 30 14:33:10.206217 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:10.205856 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b9c4b4-6498846c67-q52pq" event={"ID":"b87e2746-abcd-4e8a-bda1-cf9badd05b04","Type":"ContainerDied","Data":"37ab85dbbe9f8073f69faf585cc18eca80b229ee00cb91a234b72e0f288af9fc"} Oct 30 14:33:10.206217 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:10.205888 4982 scope.go:122] "RemoveContainer" containerID="7daa8053c072645197f3bec6f19ffed89e19e22e92a3eacdf53da4fb3788d231" Oct 30 14:33:10.227915 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:10.227890 4982 scope.go:122] "RemoveContainer" containerID="7daa8053c072645197f3bec6f19ffed89e19e22e92a3eacdf53da4fb3788d231" Oct 30 14:33:10.228510 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:33:10.228481 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7daa8053c072645197f3bec6f19ffed89e19e22e92a3eacdf53da4fb3788d231\": container with ID starting with 7daa8053c072645197f3bec6f19ffed89e19e22e92a3eacdf53da4fb3788d231 not found: ID does not exist" containerID="7daa8053c072645197f3bec6f19ffed89e19e22e92a3eacdf53da4fb3788d231" Oct 30 14:33:10.228608 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:10.228541 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7daa8053c072645197f3bec6f19ffed89e19e22e92a3eacdf53da4fb3788d231"} err="failed to get container status \"7daa8053c072645197f3bec6f19ffed89e19e22e92a3eacdf53da4fb3788d231\": rpc error: code = NotFound desc = could not find container \"7daa8053c072645197f3bec6f19ffed89e19e22e92a3eacdf53da4fb3788d231\": container with ID starting with 7daa8053c072645197f3bec6f19ffed89e19e22e92a3eacdf53da4fb3788d231 not found: ID does not exist" Oct 30 14:33:10.265952 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:10.265921 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b9c4b4-6498846c67-q52pq"] Oct 30 14:33:10.275360 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:10.273181 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b9c4b4-6498846c67-q52pq"] Oct 30 14:33:10.369757 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:10.369735 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e25b4aa-mg4n9" Oct 30 14:33:10.404090 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:10.404049 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b87e2746-abcd-4e8a-bda1-cf9badd05b04" path="/var/lib/kubelet/pods/b87e2746-abcd-4e8a-bda1-cf9badd05b04/volumes" Oct 30 14:33:10.482033 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:10.481998 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4be53bee-e002-4021-b073-f1609e19a1fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4be53bee-e002-4021-b073-f1609e19a1fd-maas-api-openshift-ca\") pod \"4be53bee-e002-4021-b073-f1609e19a1fd\" (UID: \"4be53bee-e002-4021-b073-f1609e19a1fd\") " Oct 30 14:33:10.482226 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:10.482170 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4be53bee-e002-4021-b073-f1609e19a1fd-kube-api-access-2hczb\" (UniqueName: \"kubernetes.io/projected/4be53bee-e002-4021-b073-f1609e19a1fd-kube-api-access-2hczb\") pod \"4be53bee-e002-4021-b073-f1609e19a1fd\" (UID: \"4be53bee-e002-4021-b073-f1609e19a1fd\") " Oct 30 14:33:10.482343 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:10.482322 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4be53bee-e002-4021-b073-f1609e19a1fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4be53bee-e002-4021-b073-f1609e19a1fd-maas-api-xks-ca\") pod \"4be53bee-e002-4021-b073-f1609e19a1fd\" (UID: \"4be53bee-e002-4021-b073-f1609e19a1fd\") " Oct 30 14:33:10.482428 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:10.482405 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4be53bee-e002-4021-b073-f1609e19a1fd-maas-api-openshift-ca" pod "4be53bee-e002-4021-b073-f1609e19a1fd" (UID: "4be53bee-e002-4021-b073-f1609e19a1fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:33:10.482877 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:10.482853 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4be53bee-e002-4021-b073-f1609e19a1fd-maas-api-openshift-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:33:10.484612 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:10.484590 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4be53bee-e002-4021-b073-f1609e19a1fd-maas-api-xks-ca" pod "4be53bee-e002-4021-b073-f1609e19a1fd" (UID: "4be53bee-e002-4021-b073-f1609e19a1fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:33:10.484873 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:10.484841 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4be53bee-e002-4021-b073-f1609e19a1fd-kube-api-access-2hczb" pod "4be53bee-e002-4021-b073-f1609e19a1fd" (UID: "4be53bee-e002-4021-b073-f1609e19a1fd"). InnerVolumeSpecName "kube-api-access-2hczb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:33:10.585051 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:10.585003 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hczb\" (UniqueName: \"kubernetes.io/projected/4be53bee-e002-4021-b073-f1609e19a1fd-kube-api-access-2hczb\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:33:10.585051 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:10.585037 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4be53bee-e002-4021-b073-f1609e19a1fd-maas-api-xks-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:33:11.217892 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:11.217853 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e25b4aa-mg4n9" Oct 30 14:33:11.217892 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:11.217876 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e25b4aa-mg4n9" event={"ID":"4be53bee-e002-4021-b073-f1609e19a1fd","Type":"ContainerDied","Data":"f5237056509aac36c281c4ea7da46ec599d1fb6ea127084bc89a5cd00169b0c5"} Oct 30 14:33:11.218434 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:11.217931 4982 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5237056509aac36c281c4ea7da46ec599d1fb6ea127084bc89a5cd00169b0c5" Oct 30 14:33:11.380214 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:11.380174 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:33:11.544130 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:11.544052 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:33:12.329628 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:12.329587 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:33:13.450209 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:13.450177 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:33:15.320122 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:15.320082 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:33:17.206005 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:17.205971 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:33:18.045401 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:18.045360 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c5e8070-5qdn2"] Oct 30 14:33:18.058137 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:18.058102 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c5e8070-5qdn2"] Oct 30 14:33:18.400552 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:18.400518 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cdc8048-a4db-4e5f-82d1-b0229c255311" path="/var/lib/kubelet/pods/8cdc8048-a4db-4e5f-82d1-b0229c255311/volumes" Oct 30 14:33:19.121425 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:19.121379 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:33:20.953324 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:20.953283 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:33:21.730569 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:21.730533 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:33:22.580383 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:22.580350 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:33:22.981729 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:22.981515 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:33:24.713455 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:24.713417 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:33:26.509331 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:26.509295 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:33:28.013901 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:33:28.013865 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:33:28.538065 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:28.538025 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:33:30.185625 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:30.185582 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:33:30.697333 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:30.697293 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:33:30.807799 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:30.807755 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0b1c15-t58mk"] Oct 30 14:33:30.820994 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:30.820960 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0b1c15-t58mk"] Oct 30 14:33:30.843233 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:30.843188 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:33:30.936736 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:30.936694 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-019e5b-rtxww"] Oct 30 14:33:30.937895 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:30.937864 4982 cpu_manager.go:414] "containerMap: removing container" podUID="b87e2746-abcd-4e8a-bda1-cf9badd05b04" containerName="payload-pre-processing" Oct 30 14:33:30.938059 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:30.938042 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b87e2746-abcd-4e8a-bda1-cf9badd05b04" containerName="payload-pre-processing" Oct 30 14:33:30.938185 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:30.938171 4982 cpu_manager.go:414] "containerMap: removing container" podUID="4be53bee-e002-4021-b073-f1609e19a1fd" containerName="revoke-keys" Oct 30 14:33:30.938277 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:30.938265 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4be53bee-e002-4021-b073-f1609e19a1fd" containerName="revoke-keys" Oct 30 14:33:30.943708 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:30.938551 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="b87e2746-abcd-4e8a-bda1-cf9badd05b04" containerName="payload-pre-processing" Oct 30 14:33:30.943708 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:30.941397 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="4be53bee-e002-4021-b073-f1609e19a1fd" containerName="revoke-keys" Oct 30 14:33:30.946882 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:30.946862 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-019e5b-rtxww"] Oct 30 14:33:30.947035 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:30.946964 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-019e5b-rtxww" Oct 30 14:33:30.971696 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:30.968510 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:33:30.971696 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:30.969245 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:33:30.994390 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:30.994321 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:33:31.119789 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:31.119754 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klsnn\" (UniqueName: \"kubernetes.io/projected/3f034e5f-5eaa-4f06-8737-0215cccb7c80-kube-api-access-klsnn\") pod \"maas-api-revoke-keys-e2e-inf-a-019e5b-rtxww\" (UID: \"3f034e5f-5eaa-4f06-8737-0215cccb7c80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-019e5b-rtxww" Oct 30 14:33:31.119994 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:31.119805 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f034e5f-5eaa-4f06-8737-0215cccb7c80-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-019e5b-rtxww\" (UID: \"3f034e5f-5eaa-4f06-8737-0215cccb7c80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-019e5b-rtxww" Oct 30 14:33:31.119994 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:31.119869 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f034e5f-5eaa-4f06-8737-0215cccb7c80-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-019e5b-rtxww\" (UID: \"3f034e5f-5eaa-4f06-8737-0215cccb7c80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-019e5b-rtxww" Oct 30 14:33:31.221531 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:31.221496 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klsnn\" (UniqueName: \"kubernetes.io/projected/3f034e5f-5eaa-4f06-8737-0215cccb7c80-kube-api-access-klsnn\") pod \"maas-api-revoke-keys-e2e-inf-a-019e5b-rtxww\" (UID: \"3f034e5f-5eaa-4f06-8737-0215cccb7c80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-019e5b-rtxww" Oct 30 14:33:31.222000 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:31.221567 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f034e5f-5eaa-4f06-8737-0215cccb7c80-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-019e5b-rtxww\" (UID: \"3f034e5f-5eaa-4f06-8737-0215cccb7c80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-019e5b-rtxww" Oct 30 14:33:31.222000 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:31.221623 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f034e5f-5eaa-4f06-8737-0215cccb7c80-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-019e5b-rtxww\" (UID: \"3f034e5f-5eaa-4f06-8737-0215cccb7c80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-019e5b-rtxww" Oct 30 14:33:31.222495 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:31.222470 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f034e5f-5eaa-4f06-8737-0215cccb7c80-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-019e5b-rtxww\" (UID: \"3f034e5f-5eaa-4f06-8737-0215cccb7c80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-019e5b-rtxww" Oct 30 14:33:31.224613 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:31.224591 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f034e5f-5eaa-4f06-8737-0215cccb7c80-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-019e5b-rtxww\" (UID: \"3f034e5f-5eaa-4f06-8737-0215cccb7c80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-019e5b-rtxww" Oct 30 14:33:31.247753 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:31.247726 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klsnn\" (UniqueName: \"kubernetes.io/projected/3f034e5f-5eaa-4f06-8737-0215cccb7c80-kube-api-access-klsnn\") pod \"maas-api-revoke-keys-e2e-inf-a-019e5b-rtxww\" (UID: \"3f034e5f-5eaa-4f06-8737-0215cccb7c80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-019e5b-rtxww" Oct 30 14:33:31.285015 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:31.284992 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-019e5b-rtxww" Oct 30 14:33:31.359034 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:31.354007 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:33:31.590562 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:33:31.590525 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f034e5f_5eaa_4f06_8737_0215cccb7c80.slice/crio-91a9058e940f56f1e5e1adfe914340ba0075dd04b7b9e05f3d67ef11c8e593ff WatchSource:0}: Error finding container 91a9058e940f56f1e5e1adfe914340ba0075dd04b7b9e05f3d67ef11c8e593ff: Status 404 returned error can't find the container with id 91a9058e940f56f1e5e1adfe914340ba0075dd04b7b9e05f3d67ef11c8e593ff Oct 30 14:33:31.592616 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:31.592573 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-019e5b-rtxww"] Oct 30 14:33:32.035203 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:32.035171 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4edd955-47wlv"] Oct 30 14:33:32.041910 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:32.041886 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4edd955-47wlv"] Oct 30 14:33:32.219436 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:32.219396 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:33:32.411664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:32.411022 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a35dab5-bed1-4d54-95d0-370ccf127c4b" path="/var/lib/kubelet/pods/7a35dab5-bed1-4d54-95d0-370ccf127c4b/volumes" Oct 30 14:33:32.412171 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:32.411724 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2b1bddb-21f5-4576-b5cb-ae696fc925f4" path="/var/lib/kubelet/pods/b2b1bddb-21f5-4576-b5cb-ae696fc925f4/volumes" Oct 30 14:33:32.456008 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:32.455962 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-019e5b-rtxww" event={"ID":"3f034e5f-5eaa-4f06-8737-0215cccb7c80","Type":"ContainerStarted","Data":"ec5aaec370b4d4fc4276ba6c9306319b043c9d9eb1808fcc812f9c46548b5ea3"} Oct 30 14:33:32.456179 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:32.456029 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-019e5b-rtxww" event={"ID":"3f034e5f-5eaa-4f06-8737-0215cccb7c80","Type":"ContainerStarted","Data":"91a9058e940f56f1e5e1adfe914340ba0075dd04b7b9e05f3d67ef11c8e593ff"} Oct 30 14:33:32.507867 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:32.507804 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-019e5b-rtxww" podStartSLOduration=2.5077869059999998 podStartE2EDuration="2.507786906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:33:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:33:32.501778559 +0000 UTC m=+2682.245912599" watchObservedRunningTime="2026-10-30 14:33:32.507786906 +0000 UTC m=+2682.251920947" Oct 30 14:33:32.846593 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:32.846553 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:33:33.018098 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:33.018057 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:33:33.321661 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:33.320568 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:33:33.473796 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:33.473762 4982 generic.go:356] "Generic (PLEG): container finished" podID="3f034e5f-5eaa-4f06-8737-0215cccb7c80" containerID="ec5aaec370b4d4fc4276ba6c9306319b043c9d9eb1808fcc812f9c46548b5ea3" exitCode=0 Oct 30 14:33:33.474201 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:33.473799 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-019e5b-rtxww" event={"ID":"3f034e5f-5eaa-4f06-8737-0215cccb7c80","Type":"ContainerDied","Data":"ec5aaec370b4d4fc4276ba6c9306319b043c9d9eb1808fcc812f9c46548b5ea3"} Oct 30 14:33:33.491546 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:33.491512 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:33:34.676488 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:34.676461 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-019e5b-rtxww" Oct 30 14:33:34.798137 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:34.798051 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f034e5f-5eaa-4f06-8737-0215cccb7c80-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f034e5f-5eaa-4f06-8737-0215cccb7c80-maas-api-xks-ca\") pod \"3f034e5f-5eaa-4f06-8737-0215cccb7c80\" (UID: \"3f034e5f-5eaa-4f06-8737-0215cccb7c80\") " Oct 30 14:33:34.798309 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:34.798221 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f034e5f-5eaa-4f06-8737-0215cccb7c80-kube-api-access-klsnn\" (UniqueName: \"kubernetes.io/projected/3f034e5f-5eaa-4f06-8737-0215cccb7c80-kube-api-access-klsnn\") pod \"3f034e5f-5eaa-4f06-8737-0215cccb7c80\" (UID: \"3f034e5f-5eaa-4f06-8737-0215cccb7c80\") " Oct 30 14:33:34.798377 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:34.798347 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f034e5f-5eaa-4f06-8737-0215cccb7c80-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f034e5f-5eaa-4f06-8737-0215cccb7c80-maas-api-openshift-ca\") pod \"3f034e5f-5eaa-4f06-8737-0215cccb7c80\" (UID: \"3f034e5f-5eaa-4f06-8737-0215cccb7c80\") " Oct 30 14:33:34.798829 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:34.798760 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f034e5f-5eaa-4f06-8737-0215cccb7c80-maas-api-openshift-ca" pod "3f034e5f-5eaa-4f06-8737-0215cccb7c80" (UID: "3f034e5f-5eaa-4f06-8737-0215cccb7c80"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:33:34.799072 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:34.799043 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f034e5f-5eaa-4f06-8737-0215cccb7c80-maas-api-openshift-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:33:34.800448 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:34.800419 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f034e5f-5eaa-4f06-8737-0215cccb7c80-kube-api-access-klsnn" pod "3f034e5f-5eaa-4f06-8737-0215cccb7c80" (UID: "3f034e5f-5eaa-4f06-8737-0215cccb7c80"). InnerVolumeSpecName "kube-api-access-klsnn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:33:34.800618 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:34.800600 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f034e5f-5eaa-4f06-8737-0215cccb7c80-maas-api-xks-ca" pod "3f034e5f-5eaa-4f06-8737-0215cccb7c80" (UID: "3f034e5f-5eaa-4f06-8737-0215cccb7c80"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:33:34.900501 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:34.900462 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f034e5f-5eaa-4f06-8737-0215cccb7c80-maas-api-xks-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:33:34.900501 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:34.900501 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klsnn\" (UniqueName: \"kubernetes.io/projected/3f034e5f-5eaa-4f06-8737-0215cccb7c80-kube-api-access-klsnn\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:33:35.042007 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:35.041967 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:33:35.492955 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:35.492924 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-019e5b-rtxww" Oct 30 14:33:35.493127 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:35.492971 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-019e5b-rtxww" event={"ID":"3f034e5f-5eaa-4f06-8737-0215cccb7c80","Type":"ContainerDied","Data":"91a9058e940f56f1e5e1adfe914340ba0075dd04b7b9e05f3d67ef11c8e593ff"} Oct 30 14:33:35.493127 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:35.493026 4982 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91a9058e940f56f1e5e1adfe914340ba0075dd04b7b9e05f3d67ef11c8e593ff" Oct 30 14:33:36.451533 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:36.451500 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:33:36.616129 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:36.616095 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:33:37.254613 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:37.254558 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:33:38.109179 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:38.109134 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:33:38.264199 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:38.264159 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:33:38.898552 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:38.898516 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:33:39.048192 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:39.048157 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:33:39.195702 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:39.193280 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh"] Oct 30 14:33:39.195702 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:39.193569 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh" podUID="d57125b3-c6b9-4d3b-b04c-7f6b30094325" containerName="istio-proxy" containerID="cri-o://fadf1c5e2f38abe61af6385309baa37457b191d90d279076e79beb34a1c5b9d1" gracePeriod=30 Oct 30 14:33:39.213587 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:39.213558 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:33:39.387581 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:39.387542 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:33:41.131833 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:41.131789 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:33:41.353450 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:41.353412 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-019e5b-75f68bd755-f22gh"] Oct 30 14:33:41.521333 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:41.521256 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:33:41.735800 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:41.735760 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:33:42.762771 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:42.762730 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:33:42.902274 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:42.902242 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:33:43.052998 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:43.052920 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:33:44.486785 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:44.486731 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh" Oct 30 14:33:44.564214 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:44.564177 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d57125b3-c6b9-4d3b-b04c-7f6b30094325-kube-api-access-6mlrq\" (UniqueName: \"kubernetes.io/projected/d57125b3-c6b9-4d3b-b04c-7f6b30094325-kube-api-access-6mlrq\") pod \"d57125b3-c6b9-4d3b-b04c-7f6b30094325\" (UID: \"d57125b3-c6b9-4d3b-b04c-7f6b30094325\") " Oct 30 14:33:44.564395 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:44.564277 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d57125b3-c6b9-4d3b-b04c-7f6b30094325-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d57125b3-c6b9-4d3b-b04c-7f6b30094325-istio-data\") pod \"d57125b3-c6b9-4d3b-b04c-7f6b30094325\" (UID: \"d57125b3-c6b9-4d3b-b04c-7f6b30094325\") " Oct 30 14:33:44.564395 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:44.564388 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d57125b3-c6b9-4d3b-b04c-7f6b30094325-istio-token\" (UniqueName: \"kubernetes.io/projected/d57125b3-c6b9-4d3b-b04c-7f6b30094325-istio-token\") pod \"d57125b3-c6b9-4d3b-b04c-7f6b30094325\" (UID: \"d57125b3-c6b9-4d3b-b04c-7f6b30094325\") " Oct 30 14:33:44.564545 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:44.564448 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d57125b3-c6b9-4d3b-b04c-7f6b30094325-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d57125b3-c6b9-4d3b-b04c-7f6b30094325-istio-podinfo\") pod \"d57125b3-c6b9-4d3b-b04c-7f6b30094325\" (UID: \"d57125b3-c6b9-4d3b-b04c-7f6b30094325\") " Oct 30 14:33:44.564545 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:44.564480 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d57125b3-c6b9-4d3b-b04c-7f6b30094325-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d57125b3-c6b9-4d3b-b04c-7f6b30094325-credential-socket\") pod \"d57125b3-c6b9-4d3b-b04c-7f6b30094325\" (UID: \"d57125b3-c6b9-4d3b-b04c-7f6b30094325\") " Oct 30 14:33:44.564545 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:44.564514 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d57125b3-c6b9-4d3b-b04c-7f6b30094325-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d57125b3-c6b9-4d3b-b04c-7f6b30094325-istio-envoy\") pod \"d57125b3-c6b9-4d3b-b04c-7f6b30094325\" (UID: \"d57125b3-c6b9-4d3b-b04c-7f6b30094325\") " Oct 30 14:33:44.564804 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:44.564602 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d57125b3-c6b9-4d3b-b04c-7f6b30094325-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d57125b3-c6b9-4d3b-b04c-7f6b30094325-istiod-ca-cert\") pod \"d57125b3-c6b9-4d3b-b04c-7f6b30094325\" (UID: \"d57125b3-c6b9-4d3b-b04c-7f6b30094325\") " Oct 30 14:33:44.564804 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:44.564697 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d57125b3-c6b9-4d3b-b04c-7f6b30094325-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d57125b3-c6b9-4d3b-b04c-7f6b30094325-workload-socket\") pod \"d57125b3-c6b9-4d3b-b04c-7f6b30094325\" (UID: \"d57125b3-c6b9-4d3b-b04c-7f6b30094325\") " Oct 30 14:33:44.564804 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:44.564767 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d57125b3-c6b9-4d3b-b04c-7f6b30094325-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d57125b3-c6b9-4d3b-b04c-7f6b30094325-workload-certs\") pod \"d57125b3-c6b9-4d3b-b04c-7f6b30094325\" (UID: \"d57125b3-c6b9-4d3b-b04c-7f6b30094325\") " Oct 30 14:33:44.565714 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:44.565685 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d57125b3-c6b9-4d3b-b04c-7f6b30094325-workload-certs" pod "d57125b3-c6b9-4d3b-b04c-7f6b30094325" (UID: "d57125b3-c6b9-4d3b-b04c-7f6b30094325"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:33:44.568437 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:44.568411 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d57125b3-c6b9-4d3b-b04c-7f6b30094325-credential-socket" pod "d57125b3-c6b9-4d3b-b04c-7f6b30094325" (UID: "d57125b3-c6b9-4d3b-b04c-7f6b30094325"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:33:44.568860 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:44.568833 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d57125b3-c6b9-4d3b-b04c-7f6b30094325-istio-data" pod "d57125b3-c6b9-4d3b-b04c-7f6b30094325" (UID: "d57125b3-c6b9-4d3b-b04c-7f6b30094325"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:33:44.569141 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:44.569116 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d57125b3-c6b9-4d3b-b04c-7f6b30094325-kube-api-access-6mlrq" pod "d57125b3-c6b9-4d3b-b04c-7f6b30094325" (UID: "d57125b3-c6b9-4d3b-b04c-7f6b30094325"). InnerVolumeSpecName "kube-api-access-6mlrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:33:44.569461 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:44.569417 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d57125b3-c6b9-4d3b-b04c-7f6b30094325-workload-socket" pod "d57125b3-c6b9-4d3b-b04c-7f6b30094325" (UID: "d57125b3-c6b9-4d3b-b04c-7f6b30094325"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:33:44.571007 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:44.570982 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d57125b3-c6b9-4d3b-b04c-7f6b30094325-istiod-ca-cert" pod "d57125b3-c6b9-4d3b-b04c-7f6b30094325" (UID: "d57125b3-c6b9-4d3b-b04c-7f6b30094325"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:33:44.571345 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:44.571294 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d57125b3-c6b9-4d3b-b04c-7f6b30094325-istio-token" pod "d57125b3-c6b9-4d3b-b04c-7f6b30094325" (UID: "d57125b3-c6b9-4d3b-b04c-7f6b30094325"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:33:44.571635 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:44.571612 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d57125b3-c6b9-4d3b-b04c-7f6b30094325-istio-podinfo" pod "d57125b3-c6b9-4d3b-b04c-7f6b30094325" (UID: "d57125b3-c6b9-4d3b-b04c-7f6b30094325"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:33:44.571773 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:44.571753 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d57125b3-c6b9-4d3b-b04c-7f6b30094325-istio-envoy" pod "d57125b3-c6b9-4d3b-b04c-7f6b30094325" (UID: "d57125b3-c6b9-4d3b-b04c-7f6b30094325"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:33:44.571915 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:44.571893 4982 generic.go:356] "Generic (PLEG): container finished" podID="d57125b3-c6b9-4d3b-b04c-7f6b30094325" containerID="fadf1c5e2f38abe61af6385309baa37457b191d90d279076e79beb34a1c5b9d1" exitCode=0 Oct 30 14:33:44.571970 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:44.571952 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh" Oct 30 14:33:44.572023 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:44.571971 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh" event={"ID":"d57125b3-c6b9-4d3b-b04c-7f6b30094325","Type":"ContainerDied","Data":"fadf1c5e2f38abe61af6385309baa37457b191d90d279076e79beb34a1c5b9d1"} Oct 30 14:33:44.572075 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:44.572036 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh" event={"ID":"d57125b3-c6b9-4d3b-b04c-7f6b30094325","Type":"ContainerDied","Data":"90a6976e769b7ec63f22ec95bed16c5235f76f73e299a00c09c11e45561594d7"} Oct 30 14:33:44.572075 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:44.572069 4982 scope.go:122] "RemoveContainer" containerID="fadf1c5e2f38abe61af6385309baa37457b191d90d279076e79beb34a1c5b9d1" Oct 30 14:33:44.610681 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:44.610579 4982 scope.go:122] "RemoveContainer" containerID="d2f5f4d9ec23ca09019f62b706839660f2af42d297cc465750ba89ed7c052cd8" Oct 30 14:33:44.623220 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:44.623199 4982 scope.go:122] "RemoveContainer" containerID="fadf1c5e2f38abe61af6385309baa37457b191d90d279076e79beb34a1c5b9d1" Oct 30 14:33:44.623536 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:33:44.623506 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fadf1c5e2f38abe61af6385309baa37457b191d90d279076e79beb34a1c5b9d1\": container with ID starting with fadf1c5e2f38abe61af6385309baa37457b191d90d279076e79beb34a1c5b9d1 not found: ID does not exist" containerID="fadf1c5e2f38abe61af6385309baa37457b191d90d279076e79beb34a1c5b9d1" Oct 30 14:33:44.623677 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:44.623548 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fadf1c5e2f38abe61af6385309baa37457b191d90d279076e79beb34a1c5b9d1"} err="failed to get container status \"fadf1c5e2f38abe61af6385309baa37457b191d90d279076e79beb34a1c5b9d1\": rpc error: code = NotFound desc = could not find container \"fadf1c5e2f38abe61af6385309baa37457b191d90d279076e79beb34a1c5b9d1\": container with ID starting with fadf1c5e2f38abe61af6385309baa37457b191d90d279076e79beb34a1c5b9d1 not found: ID does not exist" Oct 30 14:33:44.623677 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:44.623571 4982 scope.go:122] "RemoveContainer" containerID="d2f5f4d9ec23ca09019f62b706839660f2af42d297cc465750ba89ed7c052cd8" Oct 30 14:33:44.623931 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:33:44.623907 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2f5f4d9ec23ca09019f62b706839660f2af42d297cc465750ba89ed7c052cd8\": container with ID starting with d2f5f4d9ec23ca09019f62b706839660f2af42d297cc465750ba89ed7c052cd8 not found: ID does not exist" containerID="d2f5f4d9ec23ca09019f62b706839660f2af42d297cc465750ba89ed7c052cd8" Oct 30 14:33:44.623987 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:44.623942 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2f5f4d9ec23ca09019f62b706839660f2af42d297cc465750ba89ed7c052cd8"} err="failed to get container status \"d2f5f4d9ec23ca09019f62b706839660f2af42d297cc465750ba89ed7c052cd8\": rpc error: code = NotFound desc = could not find container \"d2f5f4d9ec23ca09019f62b706839660f2af42d297cc465750ba89ed7c052cd8\": container with ID starting with d2f5f4d9ec23ca09019f62b706839660f2af42d297cc465750ba89ed7c052cd8 not found: ID does not exist" Oct 30 14:33:44.647950 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:44.647915 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh"] Oct 30 14:33:44.654447 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:44.654423 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-zpkwh"] Oct 30 14:33:44.666757 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:44.666731 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mlrq\" (UniqueName: \"kubernetes.io/projected/d57125b3-c6b9-4d3b-b04c-7f6b30094325-kube-api-access-6mlrq\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:33:44.666757 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:44.666755 4982 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d57125b3-c6b9-4d3b-b04c-7f6b30094325-istio-data\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:33:44.666878 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:44.666766 4982 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d57125b3-c6b9-4d3b-b04c-7f6b30094325-istio-token\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:33:44.666878 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:44.666776 4982 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d57125b3-c6b9-4d3b-b04c-7f6b30094325-istio-podinfo\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:33:44.666878 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:44.666784 4982 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d57125b3-c6b9-4d3b-b04c-7f6b30094325-credential-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:33:44.666878 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:44.666792 4982 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d57125b3-c6b9-4d3b-b04c-7f6b30094325-istio-envoy\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:33:44.666878 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:44.666800 4982 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d57125b3-c6b9-4d3b-b04c-7f6b30094325-istiod-ca-cert\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:33:44.666878 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:44.666809 4982 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d57125b3-c6b9-4d3b-b04c-7f6b30094325-workload-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:33:44.666878 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:44.666818 4982 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d57125b3-c6b9-4d3b-b04c-7f6b30094325-workload-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:33:44.822084 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:44.822056 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:33:45.756921 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:45.756890 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:33:46.354676 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:46.354626 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-019e5b-75f68bd755-f22gh" podUID="60275a20-4dba-4da0-8aff-5150df7abb02" containerName="payload-pre-processing" containerID="cri-o://d7d145c412ee52018e2e88fa760757eb6a777e42fbda20876a823f4fde4970d5" gracePeriod=25 Oct 30 14:33:46.401325 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:46.401293 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d57125b3-c6b9-4d3b-b04c-7f6b30094325" path="/var/lib/kubelet/pods/d57125b3-c6b9-4d3b-b04c-7f6b30094325/volumes" Oct 30 14:33:46.833133 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:46.833097 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:33:48.711805 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:48.711760 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:33:50.536431 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:50.536393 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:33:50.808898 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:50.808866 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ntlqq_dc967875-1546-40bb-8376-fde09416c204/console-operator/2.log" Oct 30 14:33:50.814409 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:50.814386 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhxjf_aca47fe8-7068-4743-b6ab-283ee0874f66/ovn-acl-logging/0.log" Oct 30 14:33:50.816894 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:50.816873 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ntlqq_dc967875-1546-40bb-8376-fde09416c204/console-operator/2.log" Oct 30 14:33:50.822477 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:50.822457 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhxjf_aca47fe8-7068-4743-b6ab-283ee0874f66/ovn-acl-logging/0.log" Oct 30 14:33:52.376613 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:52.374504 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:33:54.234043 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:54.234004 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:33:55.081094 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:55.081049 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-019e5b-75f68bd755-f22gh" podUID="60275a20-4dba-4da0-8aff-5150df7abb02" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:33:56.285658 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:56.285595 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:33:58.014670 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:33:58.014619 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:33:58.341461 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:33:58.341412 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:34:00.160768 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:00.158342 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:34:00.399703 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:00.399669 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:34:02.098690 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:02.098653 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:34:04.127717 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:04.127676 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:34:05.080880 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:05.080845 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-019e5b-75f68bd755-f22gh" podUID="60275a20-4dba-4da0-8aff-5150df7abb02" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:34:05.624536 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:05.624498 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:34:05.959142 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:05.959059 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:34:06.650445 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:06.650416 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-019e5b-75f68bd755-f22gh" Oct 30 14:34:06.747565 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:06.747531 4982 generic.go:356] "Generic (PLEG): container finished" podID="60275a20-4dba-4da0-8aff-5150df7abb02" containerID="d7d145c412ee52018e2e88fa760757eb6a777e42fbda20876a823f4fde4970d5" exitCode=0 Oct 30 14:34:06.747780 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:06.747581 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-019e5b-75f68bd755-f22gh" Oct 30 14:34:06.747780 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:06.747609 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-019e5b-75f68bd755-f22gh" event={"ID":"60275a20-4dba-4da0-8aff-5150df7abb02","Type":"ContainerDied","Data":"d7d145c412ee52018e2e88fa760757eb6a777e42fbda20876a823f4fde4970d5"} Oct 30 14:34:06.747780 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:06.747679 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-019e5b-75f68bd755-f22gh" event={"ID":"60275a20-4dba-4da0-8aff-5150df7abb02","Type":"ContainerDied","Data":"c6022ff8241ff8a20bfd01c0910e99f598ef4bd9e78cbb7a5b0d1183c2b36546"} Oct 30 14:34:06.747780 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:06.747700 4982 scope.go:122] "RemoveContainer" containerID="d7d145c412ee52018e2e88fa760757eb6a777e42fbda20876a823f4fde4970d5" Oct 30 14:34:06.758238 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:06.758218 4982 scope.go:122] "RemoveContainer" containerID="d7d145c412ee52018e2e88fa760757eb6a777e42fbda20876a823f4fde4970d5" Oct 30 14:34:06.758528 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:34:06.758506 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7d145c412ee52018e2e88fa760757eb6a777e42fbda20876a823f4fde4970d5\": container with ID starting with d7d145c412ee52018e2e88fa760757eb6a777e42fbda20876a823f4fde4970d5 not found: ID does not exist" containerID="d7d145c412ee52018e2e88fa760757eb6a777e42fbda20876a823f4fde4970d5" Oct 30 14:34:06.758606 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:06.758536 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7d145c412ee52018e2e88fa760757eb6a777e42fbda20876a823f4fde4970d5"} err="failed to get container status \"d7d145c412ee52018e2e88fa760757eb6a777e42fbda20876a823f4fde4970d5\": rpc error: code = NotFound desc = could not find container \"d7d145c412ee52018e2e88fa760757eb6a777e42fbda20876a823f4fde4970d5\": container with ID starting with d7d145c412ee52018e2e88fa760757eb6a777e42fbda20876a823f4fde4970d5 not found: ID does not exist" Oct 30 14:34:06.814992 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:06.814968 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60275a20-4dba-4da0-8aff-5150df7abb02-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60275a20-4dba-4da0-8aff-5150df7abb02-plugins-config-volume\") pod \"60275a20-4dba-4da0-8aff-5150df7abb02\" (UID: \"60275a20-4dba-4da0-8aff-5150df7abb02\") " Oct 30 14:34:06.815154 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:06.814998 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60275a20-4dba-4da0-8aff-5150df7abb02-kube-api-access-4xkm4\" (UniqueName: \"kubernetes.io/projected/60275a20-4dba-4da0-8aff-5150df7abb02-kube-api-access-4xkm4\") pod \"60275a20-4dba-4da0-8aff-5150df7abb02\" (UID: \"60275a20-4dba-4da0-8aff-5150df7abb02\") " Oct 30 14:34:06.815389 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:06.815367 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60275a20-4dba-4da0-8aff-5150df7abb02-plugins-config-volume" pod "60275a20-4dba-4da0-8aff-5150df7abb02" (UID: "60275a20-4dba-4da0-8aff-5150df7abb02"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:34:06.815532 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:06.815515 4982 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60275a20-4dba-4da0-8aff-5150df7abb02-plugins-config-volume\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:34:06.817112 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:06.817085 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60275a20-4dba-4da0-8aff-5150df7abb02-kube-api-access-4xkm4" pod "60275a20-4dba-4da0-8aff-5150df7abb02" (UID: "60275a20-4dba-4da0-8aff-5150df7abb02"). InnerVolumeSpecName "kube-api-access-4xkm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:34:06.917605 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:06.917519 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xkm4\" (UniqueName: \"kubernetes.io/projected/60275a20-4dba-4da0-8aff-5150df7abb02-kube-api-access-4xkm4\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:34:07.098606 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:07.098569 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-019e5b-75f68bd755-f22gh"] Oct 30 14:34:07.104285 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:07.104245 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-019e5b-75f68bd755-f22gh"] Oct 30 14:34:07.776075 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:07.776032 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:34:08.399424 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:08.399387 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60275a20-4dba-4da0-8aff-5150df7abb02" path="/var/lib/kubelet/pods/60275a20-4dba-4da0-8aff-5150df7abb02/volumes" Oct 30 14:34:09.739588 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:09.739545 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:34:11.655284 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:11.655248 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:34:12.969706 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:12.969672 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:34:13.485004 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:13.484965 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:34:15.688488 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:15.688453 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:34:15.832281 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:15.832244 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:34:17.552911 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:17.552878 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:34:18.783943 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:18.783907 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:34:19.390479 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:19.390446 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:34:21.237954 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:21.237905 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:34:21.883880 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:21.883849 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:34:22.008398 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:22.008361 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-019e5b-rtxww"] Oct 30 14:34:22.020380 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:22.020348 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-019e5b-rtxww"] Oct 30 14:34:22.032481 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:22.032455 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:34:22.177197 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:22.177096 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:34:22.399257 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:22.399223 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f034e5f-5eaa-4f06-8737-0215cccb7c80" path="/var/lib/kubelet/pods/3f034e5f-5eaa-4f06-8737-0215cccb7c80/volumes" Oct 30 14:34:22.526518 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:22.526439 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:34:24.167411 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:24.167361 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:34:25.927431 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:25.927394 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:34:27.729849 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:27.729814 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:34:28.014707 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:34:28.014197 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:34:29.781973 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:29.781937 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:34:29.936394 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:29.936364 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:34:30.162784 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:30.162747 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:34:30.628919 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:30.628878 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1d3696-55jvx"] Oct 30 14:34:30.629562 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:30.629538 4982 cpu_manager.go:414] "containerMap: removing container" podUID="d57125b3-c6b9-4d3b-b04c-7f6b30094325" containerName="istio-proxy" Oct 30 14:34:30.629694 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:30.629564 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d57125b3-c6b9-4d3b-b04c-7f6b30094325" containerName="istio-proxy" Oct 30 14:34:30.629694 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:30.629620 4982 cpu_manager.go:414] "containerMap: removing container" podUID="60275a20-4dba-4da0-8aff-5150df7abb02" containerName="payload-pre-processing" Oct 30 14:34:30.629694 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:30.629630 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60275a20-4dba-4da0-8aff-5150df7abb02" containerName="payload-pre-processing" Oct 30 14:34:30.629694 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:30.629688 4982 cpu_manager.go:414] "containerMap: removing container" podUID="3f034e5f-5eaa-4f06-8737-0215cccb7c80" containerName="revoke-keys" Oct 30 14:34:30.629907 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:30.629696 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f034e5f-5eaa-4f06-8737-0215cccb7c80" containerName="revoke-keys" Oct 30 14:34:30.629907 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:30.629869 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="60275a20-4dba-4da0-8aff-5150df7abb02" containerName="payload-pre-processing" Oct 30 14:34:30.629907 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:30.629891 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f034e5f-5eaa-4f06-8737-0215cccb7c80" containerName="revoke-keys" Oct 30 14:34:30.629907 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:30.629900 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="d57125b3-c6b9-4d3b-b04c-7f6b30094325" containerName="istio-proxy" Oct 30 14:34:30.630316 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:30.629914 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="d57125b3-c6b9-4d3b-b04c-7f6b30094325" containerName="istio-proxy" Oct 30 14:34:30.633955 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:30.633930 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1d3696-55jvx" Oct 30 14:34:30.653960 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:30.653926 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1d3696-55jvx"] Oct 30 14:34:30.717743 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:30.717705 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:34:30.717924 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:30.717750 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:34:30.755185 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:30.755148 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/069cf653-0bd7-46d6-8030-6853aca17b5b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1d3696-55jvx\" (UID: \"069cf653-0bd7-46d6-8030-6853aca17b5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1d3696-55jvx" Oct 30 14:34:30.755185 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:30.755188 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhrp7\" (UniqueName: \"kubernetes.io/projected/069cf653-0bd7-46d6-8030-6853aca17b5b-kube-api-access-rhrp7\") pod \"maas-api-revoke-keys-e2e-inf-b-1d3696-55jvx\" (UID: \"069cf653-0bd7-46d6-8030-6853aca17b5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1d3696-55jvx" Oct 30 14:34:30.755410 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:30.755349 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/069cf653-0bd7-46d6-8030-6853aca17b5b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1d3696-55jvx\" (UID: \"069cf653-0bd7-46d6-8030-6853aca17b5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1d3696-55jvx" Oct 30 14:34:30.856670 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:30.856594 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/069cf653-0bd7-46d6-8030-6853aca17b5b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1d3696-55jvx\" (UID: \"069cf653-0bd7-46d6-8030-6853aca17b5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1d3696-55jvx" Oct 30 14:34:30.857085 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:30.856852 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/069cf653-0bd7-46d6-8030-6853aca17b5b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1d3696-55jvx\" (UID: \"069cf653-0bd7-46d6-8030-6853aca17b5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1d3696-55jvx" Oct 30 14:34:30.857085 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:30.856884 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhrp7\" (UniqueName: \"kubernetes.io/projected/069cf653-0bd7-46d6-8030-6853aca17b5b-kube-api-access-rhrp7\") pod \"maas-api-revoke-keys-e2e-inf-b-1d3696-55jvx\" (UID: \"069cf653-0bd7-46d6-8030-6853aca17b5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1d3696-55jvx" Oct 30 14:34:30.857582 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:30.857563 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/069cf653-0bd7-46d6-8030-6853aca17b5b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1d3696-55jvx\" (UID: \"069cf653-0bd7-46d6-8030-6853aca17b5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1d3696-55jvx" Oct 30 14:34:30.859080 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:30.859061 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/069cf653-0bd7-46d6-8030-6853aca17b5b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1d3696-55jvx\" (UID: \"069cf653-0bd7-46d6-8030-6853aca17b5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1d3696-55jvx" Oct 30 14:34:30.882973 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:30.882893 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhrp7\" (UniqueName: \"kubernetes.io/projected/069cf653-0bd7-46d6-8030-6853aca17b5b-kube-api-access-rhrp7\") pod \"maas-api-revoke-keys-e2e-inf-b-1d3696-55jvx\" (UID: \"069cf653-0bd7-46d6-8030-6853aca17b5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1d3696-55jvx" Oct 30 14:34:30.947425 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:30.947388 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1d3696-55jvx" Oct 30 14:34:31.255410 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:34:31.255380 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod069cf653_0bd7_46d6_8030_6853aca17b5b.slice/crio-57aff4b064685e28670dbdcfa08eafde338829c3b18c952bab10966585d55c80 WatchSource:0}: Error finding container 57aff4b064685e28670dbdcfa08eafde338829c3b18c952bab10966585d55c80: Status 404 returned error can't find the container with id 57aff4b064685e28670dbdcfa08eafde338829c3b18c952bab10966585d55c80 Oct 30 14:34:31.255538 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:31.255432 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1d3696-55jvx"] Oct 30 14:34:31.718998 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:31.718965 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:34:31.951212 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:31.951113 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1d3696-55jvx" event={"ID":"069cf653-0bd7-46d6-8030-6853aca17b5b","Type":"ContainerStarted","Data":"5c8230a326978d3ba66850dd85582972fef91e3d19c02f33fbc67b0a248d4752"} Oct 30 14:34:31.951212 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:31.951181 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1d3696-55jvx" event={"ID":"069cf653-0bd7-46d6-8030-6853aca17b5b","Type":"ContainerStarted","Data":"57aff4b064685e28670dbdcfa08eafde338829c3b18c952bab10966585d55c80"} Oct 30 14:34:31.995461 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:31.995411 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1d3696-55jvx" podStartSLOduration=1.9953962189999999 podStartE2EDuration="1.995396219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:34:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:34:31.994422936 +0000 UTC m=+2741.738556977" watchObservedRunningTime="2026-10-30 14:34:31.995396219 +0000 UTC m=+2741.739530261" Oct 30 14:34:32.961279 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:32.961246 4982 generic.go:356] "Generic (PLEG): container finished" podID="069cf653-0bd7-46d6-8030-6853aca17b5b" containerID="5c8230a326978d3ba66850dd85582972fef91e3d19c02f33fbc67b0a248d4752" exitCode=0 Oct 30 14:34:32.961712 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:32.961308 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1d3696-55jvx" event={"ID":"069cf653-0bd7-46d6-8030-6853aca17b5b","Type":"ContainerDied","Data":"5c8230a326978d3ba66850dd85582972fef91e3d19c02f33fbc67b0a248d4752"} Oct 30 14:34:33.758334 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:33.758293 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:34:34.140678 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:34.140625 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1d3696-55jvx" Oct 30 14:34:34.205111 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:34.205076 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/069cf653-0bd7-46d6-8030-6853aca17b5b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/069cf653-0bd7-46d6-8030-6853aca17b5b-maas-api-xks-ca\") pod \"069cf653-0bd7-46d6-8030-6853aca17b5b\" (UID: \"069cf653-0bd7-46d6-8030-6853aca17b5b\") " Oct 30 14:34:34.205313 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:34.205140 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/069cf653-0bd7-46d6-8030-6853aca17b5b-kube-api-access-rhrp7\" (UniqueName: \"kubernetes.io/projected/069cf653-0bd7-46d6-8030-6853aca17b5b-kube-api-access-rhrp7\") pod \"069cf653-0bd7-46d6-8030-6853aca17b5b\" (UID: \"069cf653-0bd7-46d6-8030-6853aca17b5b\") " Oct 30 14:34:34.205313 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:34.205220 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/069cf653-0bd7-46d6-8030-6853aca17b5b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/069cf653-0bd7-46d6-8030-6853aca17b5b-maas-api-openshift-ca\") pod \"069cf653-0bd7-46d6-8030-6853aca17b5b\" (UID: \"069cf653-0bd7-46d6-8030-6853aca17b5b\") " Oct 30 14:34:34.205826 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:34.205797 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/069cf653-0bd7-46d6-8030-6853aca17b5b-maas-api-openshift-ca" pod "069cf653-0bd7-46d6-8030-6853aca17b5b" (UID: "069cf653-0bd7-46d6-8030-6853aca17b5b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:34:34.206077 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:34.206057 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/069cf653-0bd7-46d6-8030-6853aca17b5b-maas-api-openshift-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:34:34.207955 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:34.207924 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/069cf653-0bd7-46d6-8030-6853aca17b5b-maas-api-xks-ca" pod "069cf653-0bd7-46d6-8030-6853aca17b5b" (UID: "069cf653-0bd7-46d6-8030-6853aca17b5b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:34:34.208044 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:34.207954 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/069cf653-0bd7-46d6-8030-6853aca17b5b-kube-api-access-rhrp7" pod "069cf653-0bd7-46d6-8030-6853aca17b5b" (UID: "069cf653-0bd7-46d6-8030-6853aca17b5b"). InnerVolumeSpecName "kube-api-access-rhrp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:34:34.307860 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:34.307825 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/069cf653-0bd7-46d6-8030-6853aca17b5b-maas-api-xks-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:34:34.307860 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:34.307858 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhrp7\" (UniqueName: \"kubernetes.io/projected/069cf653-0bd7-46d6-8030-6853aca17b5b-kube-api-access-rhrp7\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:34:34.979176 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:34.979148 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1d3696-55jvx" Oct 30 14:34:34.979351 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:34.979185 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1d3696-55jvx" event={"ID":"069cf653-0bd7-46d6-8030-6853aca17b5b","Type":"ContainerDied","Data":"57aff4b064685e28670dbdcfa08eafde338829c3b18c952bab10966585d55c80"} Oct 30 14:34:34.979351 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:34.979239 4982 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57aff4b064685e28670dbdcfa08eafde338829c3b18c952bab10966585d55c80" Oct 30 14:34:35.805248 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:35.805199 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:34:37.530320 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:37.530275 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb035005-wvc7d"] Oct 30 14:34:37.531172 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:37.531145 4982 cpu_manager.go:414] "containerMap: removing container" podUID="069cf653-0bd7-46d6-8030-6853aca17b5b" containerName="revoke-keys" Oct 30 14:34:37.531271 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:37.531173 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="069cf653-0bd7-46d6-8030-6853aca17b5b" containerName="revoke-keys" Oct 30 14:34:37.531271 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:37.531236 4982 cpu_manager.go:414] "containerMap: removing container" podUID="d57125b3-c6b9-4d3b-b04c-7f6b30094325" containerName="istio-proxy" Oct 30 14:34:37.531271 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:37.531245 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d57125b3-c6b9-4d3b-b04c-7f6b30094325" containerName="istio-proxy" Oct 30 14:34:37.531445 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:37.531388 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="069cf653-0bd7-46d6-8030-6853aca17b5b" containerName="revoke-keys" Oct 30 14:34:37.535755 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:37.535701 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb035005-wvc7d" Oct 30 14:34:37.557794 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:37.556658 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb035005-wvc7d"] Oct 30 14:34:37.564330 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:37.564306 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:34:37.564622 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:37.564605 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:34:37.583661 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:37.571741 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f3b8815-8ec6-486f-997f-a0bd815c0817-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb035005-wvc7d\" (UID: \"0f3b8815-8ec6-486f-997f-a0bd815c0817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb035005-wvc7d" Oct 30 14:34:37.583661 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:37.571856 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xltr\" (UniqueName: \"kubernetes.io/projected/0f3b8815-8ec6-486f-997f-a0bd815c0817-kube-api-access-6xltr\") pod \"maas-api-revoke-sub-fb035005-wvc7d\" (UID: \"0f3b8815-8ec6-486f-997f-a0bd815c0817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb035005-wvc7d" Oct 30 14:34:37.583661 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:37.571949 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f3b8815-8ec6-486f-997f-a0bd815c0817-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb035005-wvc7d\" (UID: \"0f3b8815-8ec6-486f-997f-a0bd815c0817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb035005-wvc7d" Oct 30 14:34:37.673315 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:37.673277 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f3b8815-8ec6-486f-997f-a0bd815c0817-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb035005-wvc7d\" (UID: \"0f3b8815-8ec6-486f-997f-a0bd815c0817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb035005-wvc7d" Oct 30 14:34:37.673535 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:37.673457 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f3b8815-8ec6-486f-997f-a0bd815c0817-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb035005-wvc7d\" (UID: \"0f3b8815-8ec6-486f-997f-a0bd815c0817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb035005-wvc7d" Oct 30 14:34:37.673601 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:37.673550 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xltr\" (UniqueName: \"kubernetes.io/projected/0f3b8815-8ec6-486f-997f-a0bd815c0817-kube-api-access-6xltr\") pod \"maas-api-revoke-sub-fb035005-wvc7d\" (UID: \"0f3b8815-8ec6-486f-997f-a0bd815c0817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb035005-wvc7d" Oct 30 14:34:37.674139 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:37.674113 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f3b8815-8ec6-486f-997f-a0bd815c0817-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb035005-wvc7d\" (UID: \"0f3b8815-8ec6-486f-997f-a0bd815c0817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb035005-wvc7d" Oct 30 14:34:37.676958 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:37.676934 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f3b8815-8ec6-486f-997f-a0bd815c0817-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb035005-wvc7d\" (UID: \"0f3b8815-8ec6-486f-997f-a0bd815c0817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb035005-wvc7d" Oct 30 14:34:37.699490 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:37.699462 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xltr\" (UniqueName: \"kubernetes.io/projected/0f3b8815-8ec6-486f-997f-a0bd815c0817-kube-api-access-6xltr\") pod \"maas-api-revoke-sub-fb035005-wvc7d\" (UID: \"0f3b8815-8ec6-486f-997f-a0bd815c0817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb035005-wvc7d" Oct 30 14:34:37.859698 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:37.859632 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb035005-wvc7d" Oct 30 14:34:37.940114 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:37.939919 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:34:38.189989 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:38.189161 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb035005-wvc7d"] Oct 30 14:34:38.200461 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:34:38.200436 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f3b8815_8ec6_486f_997f_a0bd815c0817.slice/crio-8a09f91f1dd1f593ca17b145dcc08645cdad166e945cb55b31bdb45ae1e49328 WatchSource:0}: Error finding container 8a09f91f1dd1f593ca17b145dcc08645cdad166e945cb55b31bdb45ae1e49328: Status 404 returned error can't find the container with id 8a09f91f1dd1f593ca17b145dcc08645cdad166e945cb55b31bdb45ae1e49328 Oct 30 14:34:39.073594 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:39.073553 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb035005-wvc7d" event={"ID":"0f3b8815-8ec6-486f-997f-a0bd815c0817","Type":"ContainerStarted","Data":"62d6b2d46350e915035a9b986c1d050601ca058ad60be62bdfe6a5e76d1fbfc5"} Oct 30 14:34:39.074098 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:39.073616 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb035005-wvc7d" event={"ID":"0f3b8815-8ec6-486f-997f-a0bd815c0817","Type":"ContainerStarted","Data":"8a09f91f1dd1f593ca17b145dcc08645cdad166e945cb55b31bdb45ae1e49328"} Oct 30 14:34:39.121161 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:39.121098 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb035005-wvc7d" podStartSLOduration=2.121078746 podStartE2EDuration="2.121078746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:34:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:34:39.118279339 +0000 UTC m=+2748.862413380" watchObservedRunningTime="2026-10-30 14:34:39.121078746 +0000 UTC m=+2748.865212789" Oct 30 14:34:39.484982 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:39.484936 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:34:39.756940 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:39.756853 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:34:40.082263 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:40.082228 4982 generic.go:356] "Generic (PLEG): container finished" podID="0f3b8815-8ec6-486f-997f-a0bd815c0817" containerID="62d6b2d46350e915035a9b986c1d050601ca058ad60be62bdfe6a5e76d1fbfc5" exitCode=0 Oct 30 14:34:40.082689 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:40.082275 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb035005-wvc7d" event={"ID":"0f3b8815-8ec6-486f-997f-a0bd815c0817","Type":"ContainerDied","Data":"62d6b2d46350e915035a9b986c1d050601ca058ad60be62bdfe6a5e76d1fbfc5"} Oct 30 14:34:41.203692 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:41.202510 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1d3696-69b88b47c5-kpq6k"] Oct 30 14:34:41.267033 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:41.267010 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb035005-wvc7d" Oct 30 14:34:41.320982 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:41.320953 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:34:41.333212 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:41.333181 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f3b8815-8ec6-486f-997f-a0bd815c0817-kube-api-access-6xltr\" (UniqueName: \"kubernetes.io/projected/0f3b8815-8ec6-486f-997f-a0bd815c0817-kube-api-access-6xltr\") pod \"0f3b8815-8ec6-486f-997f-a0bd815c0817\" (UID: \"0f3b8815-8ec6-486f-997f-a0bd815c0817\") " Oct 30 14:34:41.333367 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:41.333305 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f3b8815-8ec6-486f-997f-a0bd815c0817-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f3b8815-8ec6-486f-997f-a0bd815c0817-maas-api-xks-ca\") pod \"0f3b8815-8ec6-486f-997f-a0bd815c0817\" (UID: \"0f3b8815-8ec6-486f-997f-a0bd815c0817\") " Oct 30 14:34:41.333367 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:41.333343 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f3b8815-8ec6-486f-997f-a0bd815c0817-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f3b8815-8ec6-486f-997f-a0bd815c0817-maas-api-openshift-ca\") pod \"0f3b8815-8ec6-486f-997f-a0bd815c0817\" (UID: \"0f3b8815-8ec6-486f-997f-a0bd815c0817\") " Oct 30 14:34:41.334602 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:41.334572 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f3b8815-8ec6-486f-997f-a0bd815c0817-maas-api-openshift-ca" pod "0f3b8815-8ec6-486f-997f-a0bd815c0817" (UID: "0f3b8815-8ec6-486f-997f-a0bd815c0817"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:34:41.336591 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:41.336565 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f3b8815-8ec6-486f-997f-a0bd815c0817-kube-api-access-6xltr" pod "0f3b8815-8ec6-486f-997f-a0bd815c0817" (UID: "0f3b8815-8ec6-486f-997f-a0bd815c0817"). InnerVolumeSpecName "kube-api-access-6xltr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:34:41.347818 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:41.347761 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f3b8815-8ec6-486f-997f-a0bd815c0817-maas-api-xks-ca" pod "0f3b8815-8ec6-486f-997f-a0bd815c0817" (UID: "0f3b8815-8ec6-486f-997f-a0bd815c0817"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:34:41.434907 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:41.434871 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f3b8815-8ec6-486f-997f-a0bd815c0817-maas-api-openshift-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:34:41.434907 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:41.434899 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xltr\" (UniqueName: \"kubernetes.io/projected/0f3b8815-8ec6-486f-997f-a0bd815c0817-kube-api-access-6xltr\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:34:41.434907 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:41.434910 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f3b8815-8ec6-486f-997f-a0bd815c0817-maas-api-xks-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:34:41.474543 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:41.474506 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:34:41.733960 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:41.733876 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:34:42.101551 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:42.101522 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb035005-wvc7d" Oct 30 14:34:42.101769 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:42.101596 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb035005-wvc7d" event={"ID":"0f3b8815-8ec6-486f-997f-a0bd815c0817","Type":"ContainerDied","Data":"8a09f91f1dd1f593ca17b145dcc08645cdad166e945cb55b31bdb45ae1e49328"} Oct 30 14:34:42.101769 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:42.101692 4982 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a09f91f1dd1f593ca17b145dcc08645cdad166e945cb55b31bdb45ae1e49328" Oct 30 14:34:43.520194 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:43.520163 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:34:45.458787 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:45.458006 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:34:45.944418 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:45.944378 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:34:46.203774 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:46.203688 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1d3696-69b88b47c5-kpq6k" podUID="1d7bb15a-a136-49f4-b8cd-6739f38bfb99" containerName="payload-pre-processing" containerID="cri-o://8f87ccdb4e872c8e560c2a6e8700a93d3438ca9f836e1428149057601b16bab2" gracePeriod=25 Oct 30 14:34:47.346265 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:47.346230 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:34:47.931917 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:47.931884 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d8e34e9-5r5zs"] Oct 30 14:34:47.932525 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:47.932506 4982 cpu_manager.go:414] "containerMap: removing container" podUID="0f3b8815-8ec6-486f-997f-a0bd815c0817" containerName="revoke-keys" Oct 30 14:34:47.932525 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:47.932521 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f3b8815-8ec6-486f-997f-a0bd815c0817" containerName="revoke-keys" Oct 30 14:34:47.932751 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:47.932613 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f3b8815-8ec6-486f-997f-a0bd815c0817" containerName="revoke-keys" Oct 30 14:34:47.936386 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:47.936369 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8e34e9-5r5zs" Oct 30 14:34:47.956734 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:47.956711 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d8e34e9-5r5zs"] Oct 30 14:34:47.966166 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:47.966138 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:34:47.966166 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:47.966142 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:34:48.021325 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:48.021295 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01d8ac6a-ba15-48c1-b33a-7b18910666e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d8e34e9-5r5zs\" (UID: \"01d8ac6a-ba15-48c1-b33a-7b18910666e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8e34e9-5r5zs" Oct 30 14:34:48.021522 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:48.021330 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78h9p\" (UniqueName: \"kubernetes.io/projected/01d8ac6a-ba15-48c1-b33a-7b18910666e7-kube-api-access-78h9p\") pod \"maas-api-revoke-sub-4d8e34e9-5r5zs\" (UID: \"01d8ac6a-ba15-48c1-b33a-7b18910666e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8e34e9-5r5zs" Oct 30 14:34:48.021595 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:48.021513 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01d8ac6a-ba15-48c1-b33a-7b18910666e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d8e34e9-5r5zs\" (UID: \"01d8ac6a-ba15-48c1-b33a-7b18910666e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8e34e9-5r5zs" Oct 30 14:34:48.035451 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:48.035423 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e83f182f-4d9d2"] Oct 30 14:34:48.038433 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:48.038411 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e83f182f-4d9d2"] Oct 30 14:34:48.123194 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:48.123154 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01d8ac6a-ba15-48c1-b33a-7b18910666e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d8e34e9-5r5zs\" (UID: \"01d8ac6a-ba15-48c1-b33a-7b18910666e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8e34e9-5r5zs" Oct 30 14:34:48.123357 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:48.123202 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78h9p\" (UniqueName: \"kubernetes.io/projected/01d8ac6a-ba15-48c1-b33a-7b18910666e7-kube-api-access-78h9p\") pod \"maas-api-revoke-sub-4d8e34e9-5r5zs\" (UID: \"01d8ac6a-ba15-48c1-b33a-7b18910666e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8e34e9-5r5zs" Oct 30 14:34:48.123357 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:48.123333 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01d8ac6a-ba15-48c1-b33a-7b18910666e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d8e34e9-5r5zs\" (UID: \"01d8ac6a-ba15-48c1-b33a-7b18910666e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8e34e9-5r5zs" Oct 30 14:34:48.123906 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:48.123872 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01d8ac6a-ba15-48c1-b33a-7b18910666e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d8e34e9-5r5zs\" (UID: \"01d8ac6a-ba15-48c1-b33a-7b18910666e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8e34e9-5r5zs" Oct 30 14:34:48.125818 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:48.125793 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01d8ac6a-ba15-48c1-b33a-7b18910666e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d8e34e9-5r5zs\" (UID: \"01d8ac6a-ba15-48c1-b33a-7b18910666e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8e34e9-5r5zs" Oct 30 14:34:48.149478 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:48.149452 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78h9p\" (UniqueName: \"kubernetes.io/projected/01d8ac6a-ba15-48c1-b33a-7b18910666e7-kube-api-access-78h9p\") pod \"maas-api-revoke-sub-4d8e34e9-5r5zs\" (UID: \"01d8ac6a-ba15-48c1-b33a-7b18910666e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8e34e9-5r5zs" Oct 30 14:34:48.250219 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:48.250154 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8e34e9-5r5zs" Oct 30 14:34:48.403231 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:48.403191 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4771a9e0-445a-40ae-8f79-2e6de53a3660" path="/var/lib/kubelet/pods/4771a9e0-445a-40ae-8f79-2e6de53a3660/volumes" Oct 30 14:34:48.556595 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:34:48.556568 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01d8ac6a_ba15_48c1_b33a_7b18910666e7.slice/crio-2547026962f87c1addaef87cb0f9e92f2250eac12ede0d8ceea5db0c98af114a WatchSource:0}: Error finding container 2547026962f87c1addaef87cb0f9e92f2250eac12ede0d8ceea5db0c98af114a: Status 404 returned error can't find the container with id 2547026962f87c1addaef87cb0f9e92f2250eac12ede0d8ceea5db0c98af114a Oct 30 14:34:48.557741 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:48.557712 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d8e34e9-5r5zs"] Oct 30 14:34:49.170780 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:49.170687 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8e34e9-5r5zs" event={"ID":"01d8ac6a-ba15-48c1-b33a-7b18910666e7","Type":"ContainerStarted","Data":"6a4279d54171dd88b81fe4be414dd635f445d2d497e6e33cb423e0e2e1c1dbdd"} Oct 30 14:34:49.170780 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:49.170774 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8e34e9-5r5zs" event={"ID":"01d8ac6a-ba15-48c1-b33a-7b18910666e7","Type":"ContainerStarted","Data":"2547026962f87c1addaef87cb0f9e92f2250eac12ede0d8ceea5db0c98af114a"} Oct 30 14:34:49.214945 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:49.214888 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8e34e9-5r5zs" podStartSLOduration=2.214872933 podStartE2EDuration="2.214872933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:34:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:34:49.213426984 +0000 UTC m=+2758.957561025" watchObservedRunningTime="2026-10-30 14:34:49.214872933 +0000 UTC m=+2758.959006977" Oct 30 14:34:49.334380 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:49.329810 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:34:50.182421 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:50.182377 4982 generic.go:356] "Generic (PLEG): container finished" podID="01d8ac6a-ba15-48c1-b33a-7b18910666e7" containerID="6a4279d54171dd88b81fe4be414dd635f445d2d497e6e33cb423e0e2e1c1dbdd" exitCode=0 Oct 30 14:34:50.182962 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:50.182563 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8e34e9-5r5zs" event={"ID":"01d8ac6a-ba15-48c1-b33a-7b18910666e7","Type":"ContainerDied","Data":"6a4279d54171dd88b81fe4be414dd635f445d2d497e6e33cb423e0e2e1c1dbdd"} Oct 30 14:34:51.107790 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:51.107753 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:34:51.341203 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:51.341176 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8e34e9-5r5zs" Oct 30 14:34:51.470264 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:51.470179 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01d8ac6a-ba15-48c1-b33a-7b18910666e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01d8ac6a-ba15-48c1-b33a-7b18910666e7-maas-api-openshift-ca\") pod \"01d8ac6a-ba15-48c1-b33a-7b18910666e7\" (UID: \"01d8ac6a-ba15-48c1-b33a-7b18910666e7\") " Oct 30 14:34:51.470436 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:51.470330 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01d8ac6a-ba15-48c1-b33a-7b18910666e7-kube-api-access-78h9p\" (UniqueName: \"kubernetes.io/projected/01d8ac6a-ba15-48c1-b33a-7b18910666e7-kube-api-access-78h9p\") pod \"01d8ac6a-ba15-48c1-b33a-7b18910666e7\" (UID: \"01d8ac6a-ba15-48c1-b33a-7b18910666e7\") " Oct 30 14:34:51.470522 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:51.470503 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01d8ac6a-ba15-48c1-b33a-7b18910666e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01d8ac6a-ba15-48c1-b33a-7b18910666e7-maas-api-xks-ca\") pod \"01d8ac6a-ba15-48c1-b33a-7b18910666e7\" (UID: \"01d8ac6a-ba15-48c1-b33a-7b18910666e7\") " Oct 30 14:34:51.470664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:51.470501 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01d8ac6a-ba15-48c1-b33a-7b18910666e7-maas-api-openshift-ca" pod "01d8ac6a-ba15-48c1-b33a-7b18910666e7" (UID: "01d8ac6a-ba15-48c1-b33a-7b18910666e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:34:51.470994 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:51.470970 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01d8ac6a-ba15-48c1-b33a-7b18910666e7-maas-api-openshift-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:34:51.472560 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:51.472534 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01d8ac6a-ba15-48c1-b33a-7b18910666e7-maas-api-xks-ca" pod "01d8ac6a-ba15-48c1-b33a-7b18910666e7" (UID: "01d8ac6a-ba15-48c1-b33a-7b18910666e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:34:51.472782 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:51.472767 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01d8ac6a-ba15-48c1-b33a-7b18910666e7-kube-api-access-78h9p" pod "01d8ac6a-ba15-48c1-b33a-7b18910666e7" (UID: "01d8ac6a-ba15-48c1-b33a-7b18910666e7"). InnerVolumeSpecName "kube-api-access-78h9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:34:51.572696 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:51.572664 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01d8ac6a-ba15-48c1-b33a-7b18910666e7-maas-api-xks-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:34:51.572696 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:51.572691 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78h9p\" (UniqueName: \"kubernetes.io/projected/01d8ac6a-ba15-48c1-b33a-7b18910666e7-kube-api-access-78h9p\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:34:51.629261 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:51.629222 4982 scope.go:122] "RemoveContainer" containerID="4c8f72d19b911eabce0835956f6e51b7eed25ebdb91132ffef97f228c4efb453" Oct 30 14:34:51.641855 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:51.641821 4982 scope.go:122] "RemoveContainer" containerID="07bb29a424bed8849c24f080f5f4b7c04a0a8d0bfd37d4da75cdc4b81b122d95" Oct 30 14:34:52.212191 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:52.212148 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8e34e9-5r5zs" event={"ID":"01d8ac6a-ba15-48c1-b33a-7b18910666e7","Type":"ContainerDied","Data":"2547026962f87c1addaef87cb0f9e92f2250eac12ede0d8ceea5db0c98af114a"} Oct 30 14:34:52.212434 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:52.212197 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8e34e9-5r5zs" Oct 30 14:34:52.212434 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:52.212244 4982 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2547026962f87c1addaef87cb0f9e92f2250eac12ede0d8ceea5db0c98af114a" Oct 30 14:34:52.351387 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:52.351344 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1d3696-69b88b47c5-kpq6k" podUID="1d7bb15a-a136-49f4-b8cd-6739f38bfb99" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:34:52.925346 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:52.925307 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:34:53.575709 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:53.575662 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:34:54.856145 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:54.856109 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:34:56.730931 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:56.730895 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:34:57.546137 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:57.546098 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx"] Oct 30 14:34:57.546972 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:57.546951 4982 cpu_manager.go:414] "containerMap: removing container" podUID="01d8ac6a-ba15-48c1-b33a-7b18910666e7" containerName="revoke-keys" Oct 30 14:34:57.547030 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:57.546975 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01d8ac6a-ba15-48c1-b33a-7b18910666e7" containerName="revoke-keys" Oct 30 14:34:57.547169 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:57.547158 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="01d8ac6a-ba15-48c1-b33a-7b18910666e7" containerName="revoke-keys" Oct 30 14:34:57.552789 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:57.552766 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx" Oct 30 14:34:57.562697 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:57.562672 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8q65\" (UniqueName: \"kubernetes.io/projected/13d8ee0f-f4c1-4630-85d7-5fdee28f3187-kube-api-access-r8q65\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx\" (UID: \"13d8ee0f-f4c1-4630-85d7-5fdee28f3187\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx" Oct 30 14:34:57.562809 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:57.562710 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13d8ee0f-f4c1-4630-85d7-5fdee28f3187-workload-socket\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx\" (UID: \"13d8ee0f-f4c1-4630-85d7-5fdee28f3187\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx" Oct 30 14:34:57.562809 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:57.562738 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13d8ee0f-f4c1-4630-85d7-5fdee28f3187-istio-podinfo\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx\" (UID: \"13d8ee0f-f4c1-4630-85d7-5fdee28f3187\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx" Oct 30 14:34:57.562909 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:57.562865 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13d8ee0f-f4c1-4630-85d7-5fdee28f3187-workload-certs\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx\" (UID: \"13d8ee0f-f4c1-4630-85d7-5fdee28f3187\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx" Oct 30 14:34:57.562978 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:57.562954 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13d8ee0f-f4c1-4630-85d7-5fdee28f3187-istio-envoy\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx\" (UID: \"13d8ee0f-f4c1-4630-85d7-5fdee28f3187\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx" Oct 30 14:34:57.563068 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:57.563054 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13d8ee0f-f4c1-4630-85d7-5fdee28f3187-istiod-ca-cert\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx\" (UID: \"13d8ee0f-f4c1-4630-85d7-5fdee28f3187\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx" Oct 30 14:34:57.563142 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:57.563085 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13d8ee0f-f4c1-4630-85d7-5fdee28f3187-istio-data\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx\" (UID: \"13d8ee0f-f4c1-4630-85d7-5fdee28f3187\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx" Oct 30 14:34:57.563181 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:57.563142 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13d8ee0f-f4c1-4630-85d7-5fdee28f3187-istio-token\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx\" (UID: \"13d8ee0f-f4c1-4630-85d7-5fdee28f3187\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx" Oct 30 14:34:57.563261 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:57.563248 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13d8ee0f-f4c1-4630-85d7-5fdee28f3187-credential-socket\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx\" (UID: \"13d8ee0f-f4c1-4630-85d7-5fdee28f3187\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx" Oct 30 14:34:57.590748 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:57.590700 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-a59ecb-openshift-default-dockercfg-q4p9h\"" Oct 30 14:34:57.595329 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:57.595299 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx"] Oct 30 14:34:57.665140 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:57.665096 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13d8ee0f-f4c1-4630-85d7-5fdee28f3187-istio-envoy\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx\" (UID: \"13d8ee0f-f4c1-4630-85d7-5fdee28f3187\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx" Oct 30 14:34:57.665321 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:57.665219 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13d8ee0f-f4c1-4630-85d7-5fdee28f3187-istiod-ca-cert\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx\" (UID: \"13d8ee0f-f4c1-4630-85d7-5fdee28f3187\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx" Oct 30 14:34:57.665321 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:57.665248 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13d8ee0f-f4c1-4630-85d7-5fdee28f3187-istio-data\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx\" (UID: \"13d8ee0f-f4c1-4630-85d7-5fdee28f3187\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx" Oct 30 14:34:57.665321 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:57.665289 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13d8ee0f-f4c1-4630-85d7-5fdee28f3187-istio-token\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx\" (UID: \"13d8ee0f-f4c1-4630-85d7-5fdee28f3187\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx" Oct 30 14:34:57.665567 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:57.665447 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13d8ee0f-f4c1-4630-85d7-5fdee28f3187-credential-socket\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx\" (UID: \"13d8ee0f-f4c1-4630-85d7-5fdee28f3187\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx" Oct 30 14:34:57.665567 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:57.665511 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8q65\" (UniqueName: \"kubernetes.io/projected/13d8ee0f-f4c1-4630-85d7-5fdee28f3187-kube-api-access-r8q65\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx\" (UID: \"13d8ee0f-f4c1-4630-85d7-5fdee28f3187\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx" Oct 30 14:34:57.665567 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:57.665543 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13d8ee0f-f4c1-4630-85d7-5fdee28f3187-workload-socket\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx\" (UID: \"13d8ee0f-f4c1-4630-85d7-5fdee28f3187\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx" Oct 30 14:34:57.665738 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:57.665572 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13d8ee0f-f4c1-4630-85d7-5fdee28f3187-istio-podinfo\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx\" (UID: \"13d8ee0f-f4c1-4630-85d7-5fdee28f3187\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx" Oct 30 14:34:57.665738 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:57.665629 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13d8ee0f-f4c1-4630-85d7-5fdee28f3187-workload-certs\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx\" (UID: \"13d8ee0f-f4c1-4630-85d7-5fdee28f3187\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx" Oct 30 14:34:57.666014 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:57.665991 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13d8ee0f-f4c1-4630-85d7-5fdee28f3187-istio-data\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx\" (UID: \"13d8ee0f-f4c1-4630-85d7-5fdee28f3187\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx" Oct 30 14:34:57.666203 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:57.666046 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13d8ee0f-f4c1-4630-85d7-5fdee28f3187-credential-socket\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx\" (UID: \"13d8ee0f-f4c1-4630-85d7-5fdee28f3187\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx" Oct 30 14:34:57.666203 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:57.666084 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13d8ee0f-f4c1-4630-85d7-5fdee28f3187-workload-socket\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx\" (UID: \"13d8ee0f-f4c1-4630-85d7-5fdee28f3187\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx" Oct 30 14:34:57.666324 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:57.666098 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13d8ee0f-f4c1-4630-85d7-5fdee28f3187-workload-certs\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx\" (UID: \"13d8ee0f-f4c1-4630-85d7-5fdee28f3187\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx" Oct 30 14:34:57.666324 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:57.666230 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13d8ee0f-f4c1-4630-85d7-5fdee28f3187-istiod-ca-cert\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx\" (UID: \"13d8ee0f-f4c1-4630-85d7-5fdee28f3187\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx" Oct 30 14:34:57.668024 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:57.667997 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13d8ee0f-f4c1-4630-85d7-5fdee28f3187-istio-podinfo\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx\" (UID: \"13d8ee0f-f4c1-4630-85d7-5fdee28f3187\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx" Oct 30 14:34:57.668110 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:57.668002 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13d8ee0f-f4c1-4630-85d7-5fdee28f3187-istio-envoy\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx\" (UID: \"13d8ee0f-f4c1-4630-85d7-5fdee28f3187\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx" Oct 30 14:34:57.693434 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:57.693408 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13d8ee0f-f4c1-4630-85d7-5fdee28f3187-istio-token\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx\" (UID: \"13d8ee0f-f4c1-4630-85d7-5fdee28f3187\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx" Oct 30 14:34:57.693567 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:57.693550 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8q65\" (UniqueName: \"kubernetes.io/projected/13d8ee0f-f4c1-4630-85d7-5fdee28f3187-kube-api-access-r8q65\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx\" (UID: \"13d8ee0f-f4c1-4630-85d7-5fdee28f3187\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx" Oct 30 14:34:57.869389 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:57.869352 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx" Oct 30 14:34:58.017193 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:34:58.017119 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:34:58.197816 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:58.197792 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx"] Oct 30 14:34:58.198781 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:34:58.198754 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13d8ee0f_f4c1_4630_85d7_5fdee28f3187.slice/crio-b39f104d40cb1cabb6ab5ec2fc9a4d982d37fe6983a99dc77b4e45a26310f5bd WatchSource:0}: Error finding container b39f104d40cb1cabb6ab5ec2fc9a4d982d37fe6983a99dc77b4e45a26310f5bd: Status 404 returned error can't find the container with id b39f104d40cb1cabb6ab5ec2fc9a4d982d37fe6983a99dc77b4e45a26310f5bd Oct 30 14:34:58.201313 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:58.201278 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:34:58.201401 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:58.201346 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:34:58.201401 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:58.201374 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:34:58.254344 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:58.254317 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx" event={"ID":"13d8ee0f-f4c1-4630-85d7-5fdee28f3187","Type":"ContainerStarted","Data":"b39f104d40cb1cabb6ab5ec2fc9a4d982d37fe6983a99dc77b4e45a26310f5bd"} Oct 30 14:34:58.661558 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:58.661519 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:34:59.263304 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:59.263271 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx" event={"ID":"13d8ee0f-f4c1-4630-85d7-5fdee28f3187","Type":"ContainerStarted","Data":"c368a4e6fd60dcada4f4f3248ba1737ceb64c48779562ea3b450cd6162b359c3"} Oct 30 14:34:59.351119 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:59.351062 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx" podStartSLOduration=2.351047976 podStartE2EDuration="2.351047976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:34:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:34:59.336073763 +0000 UTC m=+2769.080207804" watchObservedRunningTime="2026-10-30 14:34:59.351047976 +0000 UTC m=+2769.095182016" Oct 30 14:34:59.869594 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:59.869555 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx" Oct 30 14:34:59.871377 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:59.871353 4982 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 14:34:59.871444 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:34:59.871401 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx" podUID="13d8ee0f-f4c1-4630-85d7-5fdee28f3187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 14:35:00.497090 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:00.497053 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:35:00.870632 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:00.870600 4982 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 14:35:00.870857 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:00.870663 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx" podUID="13d8ee0f-f4c1-4630-85d7-5fdee28f3187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 14:35:01.869831 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:01.869794 4982 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 14:35:01.870202 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:01.869846 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx" podUID="13d8ee0f-f4c1-4630-85d7-5fdee28f3187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 14:35:02.045713 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:02.045677 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1d3696-9b5954949-dfgdl"] Oct 30 14:35:02.045991 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:02.045945 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d3696-9b5954949-dfgdl" podUID="180b2f0c-daa7-4a34-9a5a-bb4fa79e8bcd" containerName="maas-api" containerID="cri-o://fdb867d87f41bc8f76c047aa14ec48658aab16986209ffdb3f4789112a218cc6" gracePeriod=30 Oct 30 14:35:02.142678 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:02.142549 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:35:02.443359 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:02.443269 4982 generic.go:356] "Generic (PLEG): container finished" podID="180b2f0c-daa7-4a34-9a5a-bb4fa79e8bcd" containerID="fdb867d87f41bc8f76c047aa14ec48658aab16986209ffdb3f4789112a218cc6" exitCode=0 Oct 30 14:35:02.443950 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:02.443912 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d3696-9b5954949-dfgdl" event={"ID":"180b2f0c-daa7-4a34-9a5a-bb4fa79e8bcd","Type":"ContainerDied","Data":"fdb867d87f41bc8f76c047aa14ec48658aab16986209ffdb3f4789112a218cc6"} Oct 30 14:35:02.581146 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:02.580733 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1d3696-69b88b47c5-kpq6k" podUID="1d7bb15a-a136-49f4-b8cd-6739f38bfb99" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:35:02.737004 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:02.736968 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:35:02.771254 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:02.771221 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d3696-9b5954949-dfgdl" Oct 30 14:35:02.842117 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:02.842080 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1d3696-55jvx"] Oct 30 14:35:02.869158 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:02.868984 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1d3696-55jvx"] Oct 30 14:35:02.872472 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:02.872443 4982 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 14:35:02.872930 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:02.872900 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx" podUID="13d8ee0f-f4c1-4630-85d7-5fdee28f3187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 14:35:02.888357 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:02.888317 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/180b2f0c-daa7-4a34-9a5a-bb4fa79e8bcd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/180b2f0c-daa7-4a34-9a5a-bb4fa79e8bcd-maas-api-tls\") pod \"180b2f0c-daa7-4a34-9a5a-bb4fa79e8bcd\" (UID: \"180b2f0c-daa7-4a34-9a5a-bb4fa79e8bcd\") " Oct 30 14:35:02.888756 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:02.888729 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/180b2f0c-daa7-4a34-9a5a-bb4fa79e8bcd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/180b2f0c-daa7-4a34-9a5a-bb4fa79e8bcd-openshift-service-ca\") pod \"180b2f0c-daa7-4a34-9a5a-bb4fa79e8bcd\" (UID: \"180b2f0c-daa7-4a34-9a5a-bb4fa79e8bcd\") " Oct 30 14:35:02.896664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:02.888951 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/180b2f0c-daa7-4a34-9a5a-bb4fa79e8bcd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/180b2f0c-daa7-4a34-9a5a-bb4fa79e8bcd-metrics-certs\") pod \"180b2f0c-daa7-4a34-9a5a-bb4fa79e8bcd\" (UID: \"180b2f0c-daa7-4a34-9a5a-bb4fa79e8bcd\") " Oct 30 14:35:02.896905 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:02.896877 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/180b2f0c-daa7-4a34-9a5a-bb4fa79e8bcd-kube-api-access-2z59n\" (UniqueName: \"kubernetes.io/projected/180b2f0c-daa7-4a34-9a5a-bb4fa79e8bcd-kube-api-access-2z59n\") pod \"180b2f0c-daa7-4a34-9a5a-bb4fa79e8bcd\" (UID: \"180b2f0c-daa7-4a34-9a5a-bb4fa79e8bcd\") " Oct 30 14:35:02.921704 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:02.911028 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/180b2f0c-daa7-4a34-9a5a-bb4fa79e8bcd-maas-api-tls" pod "180b2f0c-daa7-4a34-9a5a-bb4fa79e8bcd" (UID: "180b2f0c-daa7-4a34-9a5a-bb4fa79e8bcd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:35:02.921704 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:02.911515 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:35:02.921704 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:02.914784 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/180b2f0c-daa7-4a34-9a5a-bb4fa79e8bcd-kube-api-access-2z59n" pod "180b2f0c-daa7-4a34-9a5a-bb4fa79e8bcd" (UID: "180b2f0c-daa7-4a34-9a5a-bb4fa79e8bcd"). InnerVolumeSpecName "kube-api-access-2z59n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:35:02.921704 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:02.916078 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/180b2f0c-daa7-4a34-9a5a-bb4fa79e8bcd-metrics-certs" pod "180b2f0c-daa7-4a34-9a5a-bb4fa79e8bcd" (UID: "180b2f0c-daa7-4a34-9a5a-bb4fa79e8bcd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:35:02.930987 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:02.930947 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/180b2f0c-daa7-4a34-9a5a-bb4fa79e8bcd-openshift-service-ca" pod "180b2f0c-daa7-4a34-9a5a-bb4fa79e8bcd" (UID: "180b2f0c-daa7-4a34-9a5a-bb4fa79e8bcd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:35:03.004717 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:03.003768 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/180b2f0c-daa7-4a34-9a5a-bb4fa79e8bcd-maas-api-tls\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:35:03.004717 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:03.003806 4982 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/180b2f0c-daa7-4a34-9a5a-bb4fa79e8bcd-openshift-service-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:35:03.004717 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:03.003822 4982 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/180b2f0c-daa7-4a34-9a5a-bb4fa79e8bcd-metrics-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:35:03.004717 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:03.003835 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2z59n\" (UniqueName: \"kubernetes.io/projected/180b2f0c-daa7-4a34-9a5a-bb4fa79e8bcd-kube-api-access-2z59n\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:35:03.454031 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:03.453994 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d3696-9b5954949-dfgdl" event={"ID":"180b2f0c-daa7-4a34-9a5a-bb4fa79e8bcd","Type":"ContainerDied","Data":"ba4cdcf53f19679c9fedde196be1fbad84d6198794958b5722faaa3657c4c328"} Oct 30 14:35:03.454031 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:03.454018 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d3696-9b5954949-dfgdl" Oct 30 14:35:03.454266 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:03.454062 4982 scope.go:122] "RemoveContainer" containerID="fdb867d87f41bc8f76c047aa14ec48658aab16986209ffdb3f4789112a218cc6" Oct 30 14:35:03.517252 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:03.517215 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1d3696-9b5954949-dfgdl"] Oct 30 14:35:03.529804 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:03.529764 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1d3696-9b5954949-dfgdl"] Oct 30 14:35:03.870544 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:03.870506 4982 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 14:35:03.870749 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:03.870571 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx" podUID="13d8ee0f-f4c1-4630-85d7-5fdee28f3187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 14:35:04.400488 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:04.400455 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="069cf653-0bd7-46d6-8030-6853aca17b5b" path="/var/lib/kubelet/pods/069cf653-0bd7-46d6-8030-6853aca17b5b/volumes" Oct 30 14:35:04.400943 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:04.400927 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="180b2f0c-daa7-4a34-9a5a-bb4fa79e8bcd" path="/var/lib/kubelet/pods/180b2f0c-daa7-4a34-9a5a-bb4fa79e8bcd/volumes" Oct 30 14:35:04.568664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:04.568440 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-e371c3/auth-test-model-e371c3-kserve-866c468ff7-2bf7k"] Oct 30 14:35:04.605679 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:04.605423 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:35:04.799983 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:04.799897 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:35:04.874536 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:04.874506 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx" Oct 30 14:35:04.874980 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:04.874961 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx" Oct 30 14:35:04.875605 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:04.875585 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx" Oct 30 14:35:06.486305 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:06.486231 4982 generic.go:356] "Generic (PLEG): container finished" podID="1d7bb15a-a136-49f4-b8cd-6739f38bfb99" containerID="8f87ccdb4e872c8e560c2a6e8700a93d3438ca9f836e1428149057601b16bab2" exitCode=0 Oct 30 14:35:06.487094 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:06.487062 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1d3696-69b88b47c5-kpq6k" event={"ID":"1d7bb15a-a136-49f4-b8cd-6739f38bfb99","Type":"ContainerDied","Data":"8f87ccdb4e872c8e560c2a6e8700a93d3438ca9f836e1428149057601b16bab2"} Oct 30 14:35:06.519447 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:06.519422 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1d3696-69b88b47c5-kpq6k" Oct 30 14:35:06.580380 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:06.580347 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d7bb15a-a136-49f4-b8cd-6739f38bfb99-kube-api-access-9484k\" (UniqueName: \"kubernetes.io/projected/1d7bb15a-a136-49f4-b8cd-6739f38bfb99-kube-api-access-9484k\") pod \"1d7bb15a-a136-49f4-b8cd-6739f38bfb99\" (UID: \"1d7bb15a-a136-49f4-b8cd-6739f38bfb99\") " Oct 30 14:35:06.580506 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:06.580482 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d7bb15a-a136-49f4-b8cd-6739f38bfb99-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d7bb15a-a136-49f4-b8cd-6739f38bfb99-plugins-config-volume\") pod \"1d7bb15a-a136-49f4-b8cd-6739f38bfb99\" (UID: \"1d7bb15a-a136-49f4-b8cd-6739f38bfb99\") " Oct 30 14:35:06.581049 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:06.581023 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d7bb15a-a136-49f4-b8cd-6739f38bfb99-plugins-config-volume" pod "1d7bb15a-a136-49f4-b8cd-6739f38bfb99" (UID: "1d7bb15a-a136-49f4-b8cd-6739f38bfb99"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:35:06.581387 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:06.581364 4982 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d7bb15a-a136-49f4-b8cd-6739f38bfb99-plugins-config-volume\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:35:06.582926 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:06.582897 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d7bb15a-a136-49f4-b8cd-6739f38bfb99-kube-api-access-9484k" pod "1d7bb15a-a136-49f4-b8cd-6739f38bfb99" (UID: "1d7bb15a-a136-49f4-b8cd-6739f38bfb99"). InnerVolumeSpecName "kube-api-access-9484k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:35:06.638155 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:06.638135 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:35:06.683294 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:06.683250 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9484k\" (UniqueName: \"kubernetes.io/projected/1d7bb15a-a136-49f4-b8cd-6739f38bfb99-kube-api-access-9484k\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:35:07.495205 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:07.495169 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1d3696-69b88b47c5-kpq6k" event={"ID":"1d7bb15a-a136-49f4-b8cd-6739f38bfb99","Type":"ContainerDied","Data":"80c404cc4d5894bf92d880e52c6dc08135d74baf785a970f89affb433f90f6bf"} Oct 30 14:35:07.495799 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:07.495238 4982 scope.go:122] "RemoveContainer" containerID="8f87ccdb4e872c8e560c2a6e8700a93d3438ca9f836e1428149057601b16bab2" Oct 30 14:35:07.495978 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:07.495937 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1d3696-69b88b47c5-kpq6k" Oct 30 14:35:07.554326 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:07.554296 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1d3696-69b88b47c5-kpq6k"] Oct 30 14:35:07.567204 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:07.567179 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1d3696-69b88b47c5-kpq6k"] Oct 30 14:35:08.210131 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:08.210095 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:35:08.400193 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:08.400158 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d7bb15a-a136-49f4-b8cd-6739f38bfb99" path="/var/lib/kubelet/pods/1d7bb15a-a136-49f4-b8cd-6739f38bfb99/volumes" Oct 30 14:35:08.553531 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:08.553454 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:35:08.701022 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:08.700985 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:35:09.052374 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:09.052342 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54495360-zbkbn"] Oct 30 14:35:09.065123 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:09.065085 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54495360-zbkbn"] Oct 30 14:35:09.176429 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:09.176395 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:35:09.327190 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:09.327155 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:35:10.432222 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:10.432186 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f3320ba-10c9-4018-a33b-479d8b18d02e" path="/var/lib/kubelet/pods/2f3320ba-10c9-4018-a33b-479d8b18d02e/volumes" Oct 30 14:35:10.459109 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:10.459078 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:35:11.938277 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:11.938238 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:35:12.381177 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:12.381139 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:35:12.534783 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:12.534752 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:35:12.681305 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:12.681221 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:35:13.359281 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:13.359243 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:35:14.191667 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:14.191602 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:35:14.360022 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:14.359982 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:35:14.503208 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:14.503080 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:35:16.359415 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:16.359366 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:35:18.468408 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:18.465142 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:35:19.599710 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:19.599631 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-e371c3/auth-test-model-e371c3-kserve-866c468ff7-2bf7k" podUID="3770ada4-af76-45ff-8782-f84ea8f5cc26" containerName="main" containerID="cri-o://69d64e4971db8430a6e9cfd63f7dbaeb401d86bbc14c5214a25eba9900a92089" gracePeriod=45 Oct 30 14:35:20.072187 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:20.072156 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-e371c3/auth-test-model-e371c3-kserve-866c468ff7-2bf7k" Oct 30 14:35:20.087054 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:20.087017 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3770ada4-af76-45ff-8782-f84ea8f5cc26-tls-certs\" (UniqueName: \"kubernetes.io/secret/3770ada4-af76-45ff-8782-f84ea8f5cc26-tls-certs\") pod \"3770ada4-af76-45ff-8782-f84ea8f5cc26\" (UID: \"3770ada4-af76-45ff-8782-f84ea8f5cc26\") " Oct 30 14:35:20.087206 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:20.087124 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3770ada4-af76-45ff-8782-f84ea8f5cc26-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3770ada4-af76-45ff-8782-f84ea8f5cc26-model-cache\") pod \"3770ada4-af76-45ff-8782-f84ea8f5cc26\" (UID: \"3770ada4-af76-45ff-8782-f84ea8f5cc26\") " Oct 30 14:35:20.087287 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:20.087250 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3770ada4-af76-45ff-8782-f84ea8f5cc26-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3770ada4-af76-45ff-8782-f84ea8f5cc26-dshm\") pod \"3770ada4-af76-45ff-8782-f84ea8f5cc26\" (UID: \"3770ada4-af76-45ff-8782-f84ea8f5cc26\") " Oct 30 14:35:20.087354 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:20.087319 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3770ada4-af76-45ff-8782-f84ea8f5cc26-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3770ada4-af76-45ff-8782-f84ea8f5cc26-tmp-dir\") pod \"3770ada4-af76-45ff-8782-f84ea8f5cc26\" (UID: \"3770ada4-af76-45ff-8782-f84ea8f5cc26\") " Oct 30 14:35:20.087429 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:20.087363 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3770ada4-af76-45ff-8782-f84ea8f5cc26-home\" (UniqueName: \"kubernetes.io/empty-dir/3770ada4-af76-45ff-8782-f84ea8f5cc26-home\") pod \"3770ada4-af76-45ff-8782-f84ea8f5cc26\" (UID: \"3770ada4-af76-45ff-8782-f84ea8f5cc26\") " Oct 30 14:35:20.087543 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:20.087432 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3770ada4-af76-45ff-8782-f84ea8f5cc26-kube-api-access-gjml6\" (UniqueName: \"kubernetes.io/projected/3770ada4-af76-45ff-8782-f84ea8f5cc26-kube-api-access-gjml6\") pod \"3770ada4-af76-45ff-8782-f84ea8f5cc26\" (UID: \"3770ada4-af76-45ff-8782-f84ea8f5cc26\") " Oct 30 14:35:20.092070 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:20.088758 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3770ada4-af76-45ff-8782-f84ea8f5cc26-home" pod "3770ada4-af76-45ff-8782-f84ea8f5cc26" (UID: "3770ada4-af76-45ff-8782-f84ea8f5cc26"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:35:20.092070 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:20.089594 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3770ada4-af76-45ff-8782-f84ea8f5cc26-tmp-dir" pod "3770ada4-af76-45ff-8782-f84ea8f5cc26" (UID: "3770ada4-af76-45ff-8782-f84ea8f5cc26"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:35:20.092070 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:20.089815 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3770ada4-af76-45ff-8782-f84ea8f5cc26-model-cache" pod "3770ada4-af76-45ff-8782-f84ea8f5cc26" (UID: "3770ada4-af76-45ff-8782-f84ea8f5cc26"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:35:20.092602 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:20.092575 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3770ada4-af76-45ff-8782-f84ea8f5cc26-kube-api-access-gjml6" pod "3770ada4-af76-45ff-8782-f84ea8f5cc26" (UID: "3770ada4-af76-45ff-8782-f84ea8f5cc26"). InnerVolumeSpecName "kube-api-access-gjml6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:35:20.092994 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:20.092969 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3770ada4-af76-45ff-8782-f84ea8f5cc26-dshm" pod "3770ada4-af76-45ff-8782-f84ea8f5cc26" (UID: "3770ada4-af76-45ff-8782-f84ea8f5cc26"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:35:20.098982 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:20.094956 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3770ada4-af76-45ff-8782-f84ea8f5cc26-tls-certs" pod "3770ada4-af76-45ff-8782-f84ea8f5cc26" (UID: "3770ada4-af76-45ff-8782-f84ea8f5cc26"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:35:20.193066 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:20.191296 4982 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3770ada4-af76-45ff-8782-f84ea8f5cc26-dshm\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:35:20.193066 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:20.191336 4982 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3770ada4-af76-45ff-8782-f84ea8f5cc26-tmp-dir\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:35:20.193066 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:20.191353 4982 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3770ada4-af76-45ff-8782-f84ea8f5cc26-home\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:35:20.193066 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:20.191367 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjml6\" (UniqueName: \"kubernetes.io/projected/3770ada4-af76-45ff-8782-f84ea8f5cc26-kube-api-access-gjml6\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:35:20.193066 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:20.191383 4982 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3770ada4-af76-45ff-8782-f84ea8f5cc26-tls-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:35:20.193066 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:20.191398 4982 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3770ada4-af76-45ff-8782-f84ea8f5cc26-model-cache\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:35:20.599262 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:20.599218 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:35:20.626299 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:20.625834 4982 generic.go:356] "Generic (PLEG): container finished" podID="3770ada4-af76-45ff-8782-f84ea8f5cc26" containerID="69d64e4971db8430a6e9cfd63f7dbaeb401d86bbc14c5214a25eba9900a92089" exitCode=0 Oct 30 14:35:20.626299 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:20.625875 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-e371c3/auth-test-model-e371c3-kserve-866c468ff7-2bf7k" Oct 30 14:35:20.626299 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:20.625927 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-e371c3/auth-test-model-e371c3-kserve-866c468ff7-2bf7k" event={"ID":"3770ada4-af76-45ff-8782-f84ea8f5cc26","Type":"ContainerDied","Data":"69d64e4971db8430a6e9cfd63f7dbaeb401d86bbc14c5214a25eba9900a92089"} Oct 30 14:35:20.626299 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:20.625963 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-e371c3/auth-test-model-e371c3-kserve-866c468ff7-2bf7k" event={"ID":"3770ada4-af76-45ff-8782-f84ea8f5cc26","Type":"ContainerDied","Data":"ba617501a24ac1c56067439dac3f97b377ffb58c580984fde866ab8cfdfb5aa1"} Oct 30 14:35:20.626299 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:20.625986 4982 scope.go:122] "RemoveContainer" containerID="69d64e4971db8430a6e9cfd63f7dbaeb401d86bbc14c5214a25eba9900a92089" Oct 30 14:35:20.648538 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:20.648510 4982 scope.go:122] "RemoveContainer" containerID="69d64e4971db8430a6e9cfd63f7dbaeb401d86bbc14c5214a25eba9900a92089" Oct 30 14:35:20.649008 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:35:20.648979 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69d64e4971db8430a6e9cfd63f7dbaeb401d86bbc14c5214a25eba9900a92089\": container with ID starting with 69d64e4971db8430a6e9cfd63f7dbaeb401d86bbc14c5214a25eba9900a92089 not found: ID does not exist" containerID="69d64e4971db8430a6e9cfd63f7dbaeb401d86bbc14c5214a25eba9900a92089" Oct 30 14:35:20.649120 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:20.649020 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69d64e4971db8430a6e9cfd63f7dbaeb401d86bbc14c5214a25eba9900a92089"} err="failed to get container status \"69d64e4971db8430a6e9cfd63f7dbaeb401d86bbc14c5214a25eba9900a92089\": rpc error: code = NotFound desc = could not find container \"69d64e4971db8430a6e9cfd63f7dbaeb401d86bbc14c5214a25eba9900a92089\": container with ID starting with 69d64e4971db8430a6e9cfd63f7dbaeb401d86bbc14c5214a25eba9900a92089 not found: ID does not exist" Oct 30 14:35:20.698568 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:20.698528 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-e371c3/auth-test-model-e371c3-kserve-866c468ff7-2bf7k"] Oct 30 14:35:20.717154 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:20.712973 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-e371c3/auth-test-model-e371c3-kserve-866c468ff7-2bf7k"] Oct 30 14:35:21.943670 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:21.942885 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:35:22.350977 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:22.350939 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z"] Oct 30 14:35:22.351233 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:22.351195 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z" podUID="33aa1ab8-e5ee-416a-beec-abd4714629e8" containerName="istio-proxy" containerID="cri-o://6df1a589e0dbc8ee3ebd0663fd0c30e518afd48d062e115e05d332aa24387046" gracePeriod=30 Oct 30 14:35:22.379369 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:22.375021 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:35:22.412665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:22.412248 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3770ada4-af76-45ff-8782-f84ea8f5cc26" path="/var/lib/kubelet/pods/3770ada4-af76-45ff-8782-f84ea8f5cc26/volumes" Oct 30 14:35:22.527663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:22.527446 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:35:24.438004 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:24.437959 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:35:26.130580 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:26.130533 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:35:26.283614 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:26.283493 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:35:27.642469 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:27.642444 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z" Oct 30 14:35:27.679987 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:27.679955 4982 generic.go:356] "Generic (PLEG): container finished" podID="33aa1ab8-e5ee-416a-beec-abd4714629e8" containerID="6df1a589e0dbc8ee3ebd0663fd0c30e518afd48d062e115e05d332aa24387046" exitCode=0 Oct 30 14:35:27.680132 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:27.680007 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z" Oct 30 14:35:27.680132 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:27.680028 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z" event={"ID":"33aa1ab8-e5ee-416a-beec-abd4714629e8","Type":"ContainerDied","Data":"6df1a589e0dbc8ee3ebd0663fd0c30e518afd48d062e115e05d332aa24387046"} Oct 30 14:35:27.680132 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:27.680092 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z" event={"ID":"33aa1ab8-e5ee-416a-beec-abd4714629e8","Type":"ContainerDied","Data":"f9bc5fbcb9ce0a9cd1cb0d9f60cc20e9b30956166afeed65fafc2bb3c6e0738c"} Oct 30 14:35:27.680132 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:27.680122 4982 scope.go:122] "RemoveContainer" containerID="6df1a589e0dbc8ee3ebd0663fd0c30e518afd48d062e115e05d332aa24387046" Oct 30 14:35:27.690958 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:27.690938 4982 scope.go:122] "RemoveContainer" containerID="6df1a589e0dbc8ee3ebd0663fd0c30e518afd48d062e115e05d332aa24387046" Oct 30 14:35:27.691275 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:35:27.691249 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6df1a589e0dbc8ee3ebd0663fd0c30e518afd48d062e115e05d332aa24387046\": container with ID starting with 6df1a589e0dbc8ee3ebd0663fd0c30e518afd48d062e115e05d332aa24387046 not found: ID does not exist" containerID="6df1a589e0dbc8ee3ebd0663fd0c30e518afd48d062e115e05d332aa24387046" Oct 30 14:35:27.691363 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:27.691287 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6df1a589e0dbc8ee3ebd0663fd0c30e518afd48d062e115e05d332aa24387046"} err="failed to get container status \"6df1a589e0dbc8ee3ebd0663fd0c30e518afd48d062e115e05d332aa24387046\": rpc error: code = NotFound desc = could not find container \"6df1a589e0dbc8ee3ebd0663fd0c30e518afd48d062e115e05d332aa24387046\": container with ID starting with 6df1a589e0dbc8ee3ebd0663fd0c30e518afd48d062e115e05d332aa24387046 not found: ID does not exist" Oct 30 14:35:27.749412 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:27.749377 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/33aa1ab8-e5ee-416a-beec-abd4714629e8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33aa1ab8-e5ee-416a-beec-abd4714629e8-istio-podinfo\") pod \"33aa1ab8-e5ee-416a-beec-abd4714629e8\" (UID: \"33aa1ab8-e5ee-416a-beec-abd4714629e8\") " Oct 30 14:35:27.749584 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:27.749515 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33aa1ab8-e5ee-416a-beec-abd4714629e8-kube-api-access-hsddv\" (UniqueName: \"kubernetes.io/projected/33aa1ab8-e5ee-416a-beec-abd4714629e8-kube-api-access-hsddv\") pod \"33aa1ab8-e5ee-416a-beec-abd4714629e8\" (UID: \"33aa1ab8-e5ee-416a-beec-abd4714629e8\") " Oct 30 14:35:27.749661 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:27.749614 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33aa1ab8-e5ee-416a-beec-abd4714629e8-istio-token\" (UniqueName: \"kubernetes.io/projected/33aa1ab8-e5ee-416a-beec-abd4714629e8-istio-token\") pod \"33aa1ab8-e5ee-416a-beec-abd4714629e8\" (UID: \"33aa1ab8-e5ee-416a-beec-abd4714629e8\") " Oct 30 14:35:27.750033 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:27.750006 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33aa1ab8-e5ee-416a-beec-abd4714629e8-credential-socket" pod "33aa1ab8-e5ee-416a-beec-abd4714629e8" (UID: "33aa1ab8-e5ee-416a-beec-abd4714629e8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:35:27.750263 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:27.750242 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33aa1ab8-e5ee-416a-beec-abd4714629e8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33aa1ab8-e5ee-416a-beec-abd4714629e8-credential-socket\") pod \"33aa1ab8-e5ee-416a-beec-abd4714629e8\" (UID: \"33aa1ab8-e5ee-416a-beec-abd4714629e8\") " Oct 30 14:35:27.750383 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:27.750365 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33aa1ab8-e5ee-416a-beec-abd4714629e8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33aa1ab8-e5ee-416a-beec-abd4714629e8-istio-envoy\") pod \"33aa1ab8-e5ee-416a-beec-abd4714629e8\" (UID: \"33aa1ab8-e5ee-416a-beec-abd4714629e8\") " Oct 30 14:35:27.750762 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:27.750731 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33aa1ab8-e5ee-416a-beec-abd4714629e8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33aa1ab8-e5ee-416a-beec-abd4714629e8-workload-certs\") pod \"33aa1ab8-e5ee-416a-beec-abd4714629e8\" (UID: \"33aa1ab8-e5ee-416a-beec-abd4714629e8\") " Oct 30 14:35:27.750892 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:27.750786 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33aa1ab8-e5ee-416a-beec-abd4714629e8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33aa1ab8-e5ee-416a-beec-abd4714629e8-workload-socket\") pod \"33aa1ab8-e5ee-416a-beec-abd4714629e8\" (UID: \"33aa1ab8-e5ee-416a-beec-abd4714629e8\") " Oct 30 14:35:27.750993 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:27.750915 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33aa1ab8-e5ee-416a-beec-abd4714629e8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33aa1ab8-e5ee-416a-beec-abd4714629e8-istio-data\") pod \"33aa1ab8-e5ee-416a-beec-abd4714629e8\" (UID: \"33aa1ab8-e5ee-416a-beec-abd4714629e8\") " Oct 30 14:35:27.751057 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:27.751018 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33aa1ab8-e5ee-416a-beec-abd4714629e8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33aa1ab8-e5ee-416a-beec-abd4714629e8-istiod-ca-cert\") pod \"33aa1ab8-e5ee-416a-beec-abd4714629e8\" (UID: \"33aa1ab8-e5ee-416a-beec-abd4714629e8\") " Oct 30 14:35:27.751119 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:27.751095 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33aa1ab8-e5ee-416a-beec-abd4714629e8-workload-socket" pod "33aa1ab8-e5ee-416a-beec-abd4714629e8" (UID: "33aa1ab8-e5ee-416a-beec-abd4714629e8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:35:27.751280 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:27.751261 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33aa1ab8-e5ee-416a-beec-abd4714629e8-workload-certs" pod "33aa1ab8-e5ee-416a-beec-abd4714629e8" (UID: "33aa1ab8-e5ee-416a-beec-abd4714629e8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:35:27.751445 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:27.751428 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33aa1ab8-e5ee-416a-beec-abd4714629e8-istio-data" pod "33aa1ab8-e5ee-416a-beec-abd4714629e8" (UID: "33aa1ab8-e5ee-416a-beec-abd4714629e8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:35:27.751677 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:27.751622 4982 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33aa1ab8-e5ee-416a-beec-abd4714629e8-istio-data\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:35:27.751677 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:27.751658 4982 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33aa1ab8-e5ee-416a-beec-abd4714629e8-credential-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:35:27.751677 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:27.751675 4982 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33aa1ab8-e5ee-416a-beec-abd4714629e8-workload-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:35:27.751914 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:27.751688 4982 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33aa1ab8-e5ee-416a-beec-abd4714629e8-workload-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:35:27.751914 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:27.751869 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33aa1ab8-e5ee-416a-beec-abd4714629e8-istiod-ca-cert" pod "33aa1ab8-e5ee-416a-beec-abd4714629e8" (UID: "33aa1ab8-e5ee-416a-beec-abd4714629e8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:35:27.753824 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:27.753782 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/33aa1ab8-e5ee-416a-beec-abd4714629e8-istio-podinfo" pod "33aa1ab8-e5ee-416a-beec-abd4714629e8" (UID: "33aa1ab8-e5ee-416a-beec-abd4714629e8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:35:27.754106 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:27.754083 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33aa1ab8-e5ee-416a-beec-abd4714629e8-istio-envoy" pod "33aa1ab8-e5ee-416a-beec-abd4714629e8" (UID: "33aa1ab8-e5ee-416a-beec-abd4714629e8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:35:27.754187 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:27.754109 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33aa1ab8-e5ee-416a-beec-abd4714629e8-istio-token" pod "33aa1ab8-e5ee-416a-beec-abd4714629e8" (UID: "33aa1ab8-e5ee-416a-beec-abd4714629e8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:35:27.755360 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:27.755335 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33aa1ab8-e5ee-416a-beec-abd4714629e8-kube-api-access-hsddv" pod "33aa1ab8-e5ee-416a-beec-abd4714629e8" (UID: "33aa1ab8-e5ee-416a-beec-abd4714629e8"). InnerVolumeSpecName "kube-api-access-hsddv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:35:27.853084 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:27.853050 4982 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33aa1ab8-e5ee-416a-beec-abd4714629e8-istiod-ca-cert\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:35:27.853084 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:27.853079 4982 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33aa1ab8-e5ee-416a-beec-abd4714629e8-istio-podinfo\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:35:27.853084 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:27.853090 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsddv\" (UniqueName: \"kubernetes.io/projected/33aa1ab8-e5ee-416a-beec-abd4714629e8-kube-api-access-hsddv\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:35:27.853309 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:27.853100 4982 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33aa1ab8-e5ee-416a-beec-abd4714629e8-istio-token\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:35:27.853309 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:27.853109 4982 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33aa1ab8-e5ee-416a-beec-abd4714629e8-istio-envoy\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:35:28.014418 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:35:28.014384 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:35:28.050124 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:28.050092 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z"] Oct 30 14:35:28.066025 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:28.065996 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-1d3696-openshift-default-8769f4c5f-nkl9z"] Oct 30 14:35:28.152031 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:28.151982 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:35:28.403234 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:28.403201 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33aa1ab8-e5ee-416a-beec-abd4714629e8" path="/var/lib/kubelet/pods/33aa1ab8-e5ee-416a-beec-abd4714629e8/volumes" Oct 30 14:35:30.052193 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:30.052157 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:35:31.841358 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:31.841321 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:35:33.175376 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:33.175343 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:35:33.727009 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:33.726973 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:35:33.870456 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:33.870421 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:35:34.013425 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:34.013327 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:35:34.256458 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:34.256416 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:35:34.403463 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:34.403432 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:35:35.573674 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:35.567990 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:35:37.572018 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:37.571982 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:35:37.728579 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:37.728524 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:35:38.175060 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:38.175014 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:35:38.696514 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:38.696472 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:35:39.726277 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:39.726236 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:35:40.024086 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:40.024005 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:35:40.163181 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:40.163141 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c7c37b-6b9c5c5999-kdfrr"] Oct 30 14:35:40.163783 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:40.163763 4982 cpu_manager.go:414] "containerMap: removing container" podUID="3770ada4-af76-45ff-8782-f84ea8f5cc26" containerName="main" Oct 30 14:35:40.163783 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:40.163782 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3770ada4-af76-45ff-8782-f84ea8f5cc26" containerName="main" Oct 30 14:35:40.163948 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:40.163794 4982 cpu_manager.go:414] "containerMap: removing container" podUID="1d7bb15a-a136-49f4-b8cd-6739f38bfb99" containerName="payload-pre-processing" Oct 30 14:35:40.163948 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:40.163800 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d7bb15a-a136-49f4-b8cd-6739f38bfb99" containerName="payload-pre-processing" Oct 30 14:35:40.163948 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:40.163827 4982 cpu_manager.go:414] "containerMap: removing container" podUID="180b2f0c-daa7-4a34-9a5a-bb4fa79e8bcd" containerName="maas-api" Oct 30 14:35:40.163948 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:40.163832 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="180b2f0c-daa7-4a34-9a5a-bb4fa79e8bcd" containerName="maas-api" Oct 30 14:35:40.163948 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:40.163838 4982 cpu_manager.go:414] "containerMap: removing container" podUID="33aa1ab8-e5ee-416a-beec-abd4714629e8" containerName="istio-proxy" Oct 30 14:35:40.163948 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:40.163843 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33aa1ab8-e5ee-416a-beec-abd4714629e8" containerName="istio-proxy" Oct 30 14:35:40.163948 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:40.163941 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="3770ada4-af76-45ff-8782-f84ea8f5cc26" containerName="main" Oct 30 14:35:40.163948 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:40.163949 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d7bb15a-a136-49f4-b8cd-6739f38bfb99" containerName="payload-pre-processing" Oct 30 14:35:40.164363 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:40.163959 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="180b2f0c-daa7-4a34-9a5a-bb4fa79e8bcd" containerName="maas-api" Oct 30 14:35:40.164363 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:40.163965 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="33aa1ab8-e5ee-416a-beec-abd4714629e8" containerName="istio-proxy" Oct 30 14:35:40.168762 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:40.168734 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c7c37b-6b9c5c5999-kdfrr" Oct 30 14:35:40.178696 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:40.178674 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:35:40.188065 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:40.188031 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c7c37b-6b9c5c5999-kdfrr"] Oct 30 14:35:40.197463 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:40.197435 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-c7c37b\"" Oct 30 14:35:40.226202 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:40.226166 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4cdb56a-9a5e-4edc-8cec-ff32aa54de9d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-c7c37b-6b9c5c5999-kdfrr\" (UID: \"d4cdb56a-9a5e-4edc-8cec-ff32aa54de9d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c7c37b-6b9c5c5999-kdfrr" Oct 30 14:35:40.226465 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:40.226442 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vm9t\" (UniqueName: \"kubernetes.io/projected/d4cdb56a-9a5e-4edc-8cec-ff32aa54de9d-kube-api-access-7vm9t\") pod \"payload-pre-processing-e2e-shared-a-w1-c7c37b-6b9c5c5999-kdfrr\" (UID: \"d4cdb56a-9a5e-4edc-8cec-ff32aa54de9d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c7c37b-6b9c5c5999-kdfrr" Oct 30 14:35:40.299388 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:40.275178 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-c7c37b-dockercfg-28bts\"" Oct 30 14:35:40.324666 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:40.322446 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-c7c37b-67bb95b964-cqz6m"] Oct 30 14:35:40.328664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:40.328094 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vm9t\" (UniqueName: \"kubernetes.io/projected/d4cdb56a-9a5e-4edc-8cec-ff32aa54de9d-kube-api-access-7vm9t\") pod \"payload-pre-processing-e2e-shared-a-w1-c7c37b-6b9c5c5999-kdfrr\" (UID: \"d4cdb56a-9a5e-4edc-8cec-ff32aa54de9d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c7c37b-6b9c5c5999-kdfrr" Oct 30 14:35:40.328664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:40.328189 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4cdb56a-9a5e-4edc-8cec-ff32aa54de9d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-c7c37b-6b9c5c5999-kdfrr\" (UID: \"d4cdb56a-9a5e-4edc-8cec-ff32aa54de9d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c7c37b-6b9c5c5999-kdfrr" Oct 30 14:35:40.333740 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:40.330061 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4cdb56a-9a5e-4edc-8cec-ff32aa54de9d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-c7c37b-6b9c5c5999-kdfrr\" (UID: \"d4cdb56a-9a5e-4edc-8cec-ff32aa54de9d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c7c37b-6b9c5c5999-kdfrr" Oct 30 14:35:40.333740 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:40.332696 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-c7c37b-67bb95b964-cqz6m"] Oct 30 14:35:40.333740 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:40.332825 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c7c37b-67bb95b964-cqz6m" Oct 30 14:35:40.388205 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:40.388172 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vm9t\" (UniqueName: \"kubernetes.io/projected/d4cdb56a-9a5e-4edc-8cec-ff32aa54de9d-kube-api-access-7vm9t\") pod \"payload-pre-processing-e2e-shared-a-w1-c7c37b-6b9c5c5999-kdfrr\" (UID: \"d4cdb56a-9a5e-4edc-8cec-ff32aa54de9d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c7c37b-6b9c5c5999-kdfrr" Oct 30 14:35:40.429586 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:40.429550 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzbxv\" (UniqueName: \"kubernetes.io/projected/fb2ea041-440f-42d9-8454-6ff8bc952a5e-kube-api-access-bzbxv\") pod \"payload-processing-e2e-shared-a-w1-c7c37b-67bb95b964-cqz6m\" (UID: \"fb2ea041-440f-42d9-8454-6ff8bc952a5e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c7c37b-67bb95b964-cqz6m" Oct 30 14:35:40.429817 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:40.429760 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb2ea041-440f-42d9-8454-6ff8bc952a5e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-c7c37b-67bb95b964-cqz6m\" (UID: \"fb2ea041-440f-42d9-8454-6ff8bc952a5e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c7c37b-67bb95b964-cqz6m" Oct 30 14:35:40.483743 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:40.483712 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c7c37b-6b9c5c5999-kdfrr" Oct 30 14:35:40.531627 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:40.531590 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzbxv\" (UniqueName: \"kubernetes.io/projected/fb2ea041-440f-42d9-8454-6ff8bc952a5e-kube-api-access-bzbxv\") pod \"payload-processing-e2e-shared-a-w1-c7c37b-67bb95b964-cqz6m\" (UID: \"fb2ea041-440f-42d9-8454-6ff8bc952a5e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c7c37b-67bb95b964-cqz6m" Oct 30 14:35:40.531843 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:40.531715 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb2ea041-440f-42d9-8454-6ff8bc952a5e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-c7c37b-67bb95b964-cqz6m\" (UID: \"fb2ea041-440f-42d9-8454-6ff8bc952a5e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c7c37b-67bb95b964-cqz6m" Oct 30 14:35:40.532676 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:40.532526 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb2ea041-440f-42d9-8454-6ff8bc952a5e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-c7c37b-67bb95b964-cqz6m\" (UID: \"fb2ea041-440f-42d9-8454-6ff8bc952a5e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c7c37b-67bb95b964-cqz6m" Oct 30 14:35:40.563726 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:40.558943 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzbxv\" (UniqueName: \"kubernetes.io/projected/fb2ea041-440f-42d9-8454-6ff8bc952a5e-kube-api-access-bzbxv\") pod \"payload-processing-e2e-shared-a-w1-c7c37b-67bb95b964-cqz6m\" (UID: \"fb2ea041-440f-42d9-8454-6ff8bc952a5e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c7c37b-67bb95b964-cqz6m" Oct 30 14:35:40.687792 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:40.687764 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c7c37b-67bb95b964-cqz6m" Oct 30 14:35:40.810340 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:40.810307 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c7c37b-6b9c5c5999-kdfrr"] Oct 30 14:35:40.996967 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:35:40.996923 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb2ea041_440f_42d9_8454_6ff8bc952a5e.slice/crio-d0774ddfba81cb8074ecabd6d93f1b16982d65442726b56374d2075628ffea05 WatchSource:0}: Error finding container d0774ddfba81cb8074ecabd6d93f1b16982d65442726b56374d2075628ffea05: Status 404 returned error can't find the container with id d0774ddfba81cb8074ecabd6d93f1b16982d65442726b56374d2075628ffea05 Oct 30 14:35:40.998781 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:40.998755 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-c7c37b-67bb95b964-cqz6m"] Oct 30 14:35:41.651076 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:41.651036 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:35:41.796551 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:41.796508 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c7c37b-6b9c5c5999-kdfrr" event={"ID":"d4cdb56a-9a5e-4edc-8cec-ff32aa54de9d","Type":"ContainerStarted","Data":"d6606d65c58e04ee138c32d0eda581cf0d51ee5425d375ba18b84376cde310bb"} Oct 30 14:35:41.796795 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:41.796771 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c7c37b-6b9c5c5999-kdfrr" Oct 30 14:35:41.796861 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:41.796826 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c7c37b-6b9c5c5999-kdfrr" event={"ID":"d4cdb56a-9a5e-4edc-8cec-ff32aa54de9d","Type":"ContainerStarted","Data":"f3e9ed0d7231df82d4ddcc30f7b2f7eb9f2ba4253a8143d4f8df6e74438814b8"} Oct 30 14:35:41.798317 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:41.798289 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c7c37b-67bb95b964-cqz6m" event={"ID":"fb2ea041-440f-42d9-8454-6ff8bc952a5e","Type":"ContainerStarted","Data":"65cb85cfc52abff3dbf77c62f5b14356e5fb0146fc827d8bb7dac8349f6f0d08"} Oct 30 14:35:41.798447 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:41.798329 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c7c37b-67bb95b964-cqz6m" event={"ID":"fb2ea041-440f-42d9-8454-6ff8bc952a5e","Type":"ContainerStarted","Data":"d0774ddfba81cb8074ecabd6d93f1b16982d65442726b56374d2075628ffea05"} Oct 30 14:35:41.798447 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:41.798365 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c7c37b-67bb95b964-cqz6m" Oct 30 14:35:41.842362 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:41.842317 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c7c37b-6b9c5c5999-kdfrr" podStartSLOduration=1.8422990540000002 podStartE2EDuration="1.842299054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:35:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:35:41.839390677 +0000 UTC m=+2811.583524722" watchObservedRunningTime="2026-10-30 14:35:41.842299054 +0000 UTC m=+2811.586433102" Oct 30 14:35:41.885365 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:41.885319 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c7c37b-67bb95b964-cqz6m" podStartSLOduration=1.885305553 podStartE2EDuration="1.885305553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:35:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:35:41.882738118 +0000 UTC m=+2811.626872161" watchObservedRunningTime="2026-10-30 14:35:41.885305553 +0000 UTC m=+2811.629439594" Oct 30 14:35:43.450422 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:43.450386 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:35:45.336680 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:45.336634 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:35:46.052685 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:46.052651 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:35:47.131131 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:47.131080 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:35:48.883345 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:48.883305 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:35:49.032779 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:49.032745 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:35:49.191424 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:49.191334 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:35:49.355464 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:49.355429 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:35:51.046169 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:51.046135 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:35:51.687379 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:51.687343 4982 scope.go:122] "RemoveContainer" containerID="15b3d5c0581f12e08adf710a308653770fe2fb7144946fc36b4b667899d96850" Oct 30 14:35:51.700586 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:51.700559 4982 scope.go:122] "RemoveContainer" containerID="7d98b5551028f271e97b3fbd2dff2488030dc7d2279c54e836e48c8d84f6b5dd" Oct 30 14:35:52.809210 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:52.809180 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c7c37b-6b9c5c5999-kdfrr" Oct 30 14:35:52.809679 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:52.809257 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c7c37b-67bb95b964-cqz6m" Oct 30 14:35:52.839064 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:52.839027 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:35:54.637915 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:54.637868 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:35:55.208246 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:55.208207 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-e371c3/rate-test-model-e371c3-kserve-fb686cfb5-bn68k"] Oct 30 14:35:55.217632 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:55.217607 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-e371c3/rate-test-model-e371c3-kserve-fb686cfb5-bn68k" Oct 30 14:35:55.232435 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:55.232402 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-e371c3/rate-test-model-e371c3-kserve-fb686cfb5-bn68k"] Oct 30 14:35:55.240475 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:55.240446 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-e371c3\"/\"default-dockercfg-gwfbq\"" Oct 30 14:35:55.240475 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:55.240468 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-e371c3\"/\"rate-test-model-e371c3-kserve-self-signed-certs\"" Oct 30 14:35:55.245307 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:55.245279 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec8a22d7-6673-4ebe-aada-d44647d2f005-dshm\") pod \"rate-test-model-e371c3-kserve-fb686cfb5-bn68k\" (UID: \"ec8a22d7-6673-4ebe-aada-d44647d2f005\") " pod="ai-tenant-e2e-shared-a-w4-e371c3/rate-test-model-e371c3-kserve-fb686cfb5-bn68k" Oct 30 14:35:55.245426 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:55.245376 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec8a22d7-6673-4ebe-aada-d44647d2f005-tmp-dir\") pod \"rate-test-model-e371c3-kserve-fb686cfb5-bn68k\" (UID: \"ec8a22d7-6673-4ebe-aada-d44647d2f005\") " pod="ai-tenant-e2e-shared-a-w4-e371c3/rate-test-model-e371c3-kserve-fb686cfb5-bn68k" Oct 30 14:35:55.245426 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:55.245400 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec8a22d7-6673-4ebe-aada-d44647d2f005-tls-certs\") pod \"rate-test-model-e371c3-kserve-fb686cfb5-bn68k\" (UID: \"ec8a22d7-6673-4ebe-aada-d44647d2f005\") " pod="ai-tenant-e2e-shared-a-w4-e371c3/rate-test-model-e371c3-kserve-fb686cfb5-bn68k" Oct 30 14:35:55.245568 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:55.245493 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfq8h\" (UniqueName: \"kubernetes.io/projected/ec8a22d7-6673-4ebe-aada-d44647d2f005-kube-api-access-hfq8h\") pod \"rate-test-model-e371c3-kserve-fb686cfb5-bn68k\" (UID: \"ec8a22d7-6673-4ebe-aada-d44647d2f005\") " pod="ai-tenant-e2e-shared-a-w4-e371c3/rate-test-model-e371c3-kserve-fb686cfb5-bn68k" Oct 30 14:35:55.245568 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:55.245520 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec8a22d7-6673-4ebe-aada-d44647d2f005-model-cache\") pod \"rate-test-model-e371c3-kserve-fb686cfb5-bn68k\" (UID: \"ec8a22d7-6673-4ebe-aada-d44647d2f005\") " pod="ai-tenant-e2e-shared-a-w4-e371c3/rate-test-model-e371c3-kserve-fb686cfb5-bn68k" Oct 30 14:35:55.245568 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:55.245558 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec8a22d7-6673-4ebe-aada-d44647d2f005-home\") pod \"rate-test-model-e371c3-kserve-fb686cfb5-bn68k\" (UID: \"ec8a22d7-6673-4ebe-aada-d44647d2f005\") " pod="ai-tenant-e2e-shared-a-w4-e371c3/rate-test-model-e371c3-kserve-fb686cfb5-bn68k" Oct 30 14:35:55.248195 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:55.248171 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-e371c3\"/\"openshift-service-ca.crt\"" Oct 30 14:35:55.248284 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:55.248208 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-e371c3\"/\"kube-root-ca.crt\"" Oct 30 14:35:55.287663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:55.287617 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:35:55.347260 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:55.347228 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec8a22d7-6673-4ebe-aada-d44647d2f005-dshm\") pod \"rate-test-model-e371c3-kserve-fb686cfb5-bn68k\" (UID: \"ec8a22d7-6673-4ebe-aada-d44647d2f005\") " pod="ai-tenant-e2e-shared-a-w4-e371c3/rate-test-model-e371c3-kserve-fb686cfb5-bn68k" Oct 30 14:35:55.347430 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:55.347336 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec8a22d7-6673-4ebe-aada-d44647d2f005-tmp-dir\") pod \"rate-test-model-e371c3-kserve-fb686cfb5-bn68k\" (UID: \"ec8a22d7-6673-4ebe-aada-d44647d2f005\") " pod="ai-tenant-e2e-shared-a-w4-e371c3/rate-test-model-e371c3-kserve-fb686cfb5-bn68k" Oct 30 14:35:55.347430 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:55.347370 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec8a22d7-6673-4ebe-aada-d44647d2f005-tls-certs\") pod \"rate-test-model-e371c3-kserve-fb686cfb5-bn68k\" (UID: \"ec8a22d7-6673-4ebe-aada-d44647d2f005\") " pod="ai-tenant-e2e-shared-a-w4-e371c3/rate-test-model-e371c3-kserve-fb686cfb5-bn68k" Oct 30 14:35:55.347560 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:55.347463 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfq8h\" (UniqueName: \"kubernetes.io/projected/ec8a22d7-6673-4ebe-aada-d44647d2f005-kube-api-access-hfq8h\") pod \"rate-test-model-e371c3-kserve-fb686cfb5-bn68k\" (UID: \"ec8a22d7-6673-4ebe-aada-d44647d2f005\") " pod="ai-tenant-e2e-shared-a-w4-e371c3/rate-test-model-e371c3-kserve-fb686cfb5-bn68k" Oct 30 14:35:55.347560 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:55.347499 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec8a22d7-6673-4ebe-aada-d44647d2f005-model-cache\") pod \"rate-test-model-e371c3-kserve-fb686cfb5-bn68k\" (UID: \"ec8a22d7-6673-4ebe-aada-d44647d2f005\") " pod="ai-tenant-e2e-shared-a-w4-e371c3/rate-test-model-e371c3-kserve-fb686cfb5-bn68k" Oct 30 14:35:55.347560 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:55.347552 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec8a22d7-6673-4ebe-aada-d44647d2f005-home\") pod \"rate-test-model-e371c3-kserve-fb686cfb5-bn68k\" (UID: \"ec8a22d7-6673-4ebe-aada-d44647d2f005\") " pod="ai-tenant-e2e-shared-a-w4-e371c3/rate-test-model-e371c3-kserve-fb686cfb5-bn68k" Oct 30 14:35:55.347837 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:55.347808 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec8a22d7-6673-4ebe-aada-d44647d2f005-tmp-dir\") pod \"rate-test-model-e371c3-kserve-fb686cfb5-bn68k\" (UID: \"ec8a22d7-6673-4ebe-aada-d44647d2f005\") " pod="ai-tenant-e2e-shared-a-w4-e371c3/rate-test-model-e371c3-kserve-fb686cfb5-bn68k" Oct 30 14:35:55.347963 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:55.347869 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec8a22d7-6673-4ebe-aada-d44647d2f005-model-cache\") pod \"rate-test-model-e371c3-kserve-fb686cfb5-bn68k\" (UID: \"ec8a22d7-6673-4ebe-aada-d44647d2f005\") " pod="ai-tenant-e2e-shared-a-w4-e371c3/rate-test-model-e371c3-kserve-fb686cfb5-bn68k" Oct 30 14:35:55.348026 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:55.347980 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec8a22d7-6673-4ebe-aada-d44647d2f005-home\") pod \"rate-test-model-e371c3-kserve-fb686cfb5-bn68k\" (UID: \"ec8a22d7-6673-4ebe-aada-d44647d2f005\") " pod="ai-tenant-e2e-shared-a-w4-e371c3/rate-test-model-e371c3-kserve-fb686cfb5-bn68k" Oct 30 14:35:55.349750 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:55.349727 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec8a22d7-6673-4ebe-aada-d44647d2f005-dshm\") pod \"rate-test-model-e371c3-kserve-fb686cfb5-bn68k\" (UID: \"ec8a22d7-6673-4ebe-aada-d44647d2f005\") " pod="ai-tenant-e2e-shared-a-w4-e371c3/rate-test-model-e371c3-kserve-fb686cfb5-bn68k" Oct 30 14:35:55.350240 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:55.350217 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec8a22d7-6673-4ebe-aada-d44647d2f005-tls-certs\") pod \"rate-test-model-e371c3-kserve-fb686cfb5-bn68k\" (UID: \"ec8a22d7-6673-4ebe-aada-d44647d2f005\") " pod="ai-tenant-e2e-shared-a-w4-e371c3/rate-test-model-e371c3-kserve-fb686cfb5-bn68k" Oct 30 14:35:55.373936 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:55.373913 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfq8h\" (UniqueName: \"kubernetes.io/projected/ec8a22d7-6673-4ebe-aada-d44647d2f005-kube-api-access-hfq8h\") pod \"rate-test-model-e371c3-kserve-fb686cfb5-bn68k\" (UID: \"ec8a22d7-6673-4ebe-aada-d44647d2f005\") " pod="ai-tenant-e2e-shared-a-w4-e371c3/rate-test-model-e371c3-kserve-fb686cfb5-bn68k" Oct 30 14:35:55.537450 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:55.537098 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-e371c3/rate-test-model-e371c3-kserve-fb686cfb5-bn68k" Oct 30 14:35:55.863979 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:55.863953 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-e371c3/rate-test-model-e371c3-kserve-fb686cfb5-bn68k"] Oct 30 14:35:55.865390 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:35:55.865361 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podec8a22d7_6673_4ebe_aada_d44647d2f005.slice/crio-5857eb274be8566f8d3355ab7e9e0cd9ae964819c5f1a71cd8cd8a0cb3ec4017 WatchSource:0}: Error finding container 5857eb274be8566f8d3355ab7e9e0cd9ae964819c5f1a71cd8cd8a0cb3ec4017: Status 404 returned error can't find the container with id 5857eb274be8566f8d3355ab7e9e0cd9ae964819c5f1a71cd8cd8a0cb3ec4017 Oct 30 14:35:55.923106 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:55.922997 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-e371c3/rate-test-model-e371c3-kserve-fb686cfb5-bn68k" event={"ID":"ec8a22d7-6673-4ebe-aada-d44647d2f005","Type":"ContainerStarted","Data":"5857eb274be8566f8d3355ab7e9e0cd9ae964819c5f1a71cd8cd8a0cb3ec4017"} Oct 30 14:35:56.652407 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:56.652368 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:35:56.932930 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:56.932849 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-e371c3/rate-test-model-e371c3-kserve-fb686cfb5-bn68k" event={"ID":"ec8a22d7-6673-4ebe-aada-d44647d2f005","Type":"ContainerStarted","Data":"eaf50028ca79ee5852b13645f6c0707704c0d784f6662a222c567b2987d12c22"} Oct 30 14:35:56.979820 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:56.979760 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-e371c3/rate-test-model-e371c3-kserve-fb686cfb5-bn68k" podStartSLOduration=1.9797460089999999 podStartE2EDuration="1.979746009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:35:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:35:56.976519188 +0000 UTC m=+2826.720653229" watchObservedRunningTime="2026-10-30 14:35:56.979746009 +0000 UTC m=+2826.723880097" Oct 30 14:35:57.538016 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:57.537987 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-e371c3/rate-test-model-e371c3-kserve-fb686cfb5-bn68k" Oct 30 14:35:58.014139 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:35:58.014101 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:35:58.443006 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:58.442967 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:35:59.043207 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:59.043166 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cab7d423-bf5jd"] Oct 30 14:35:59.105276 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:35:59.105235 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cab7d423-bf5jd"] Oct 30 14:36:00.354316 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:00.354278 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:36:00.402792 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:00.402755 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0783d93b-e60a-4769-b757-430aa9bf97a7" path="/var/lib/kubelet/pods/0783d93b-e60a-4769-b757-430aa9bf97a7/volumes" Oct 30 14:36:01.997431 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:01.997394 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9346a31-qnmmw"] Oct 30 14:36:02.002778 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:02.002759 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9346a31-qnmmw" Oct 30 14:36:02.021756 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:02.021732 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9346a31-qnmmw"] Oct 30 14:36:02.033676 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:02.031439 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:36:02.033676 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:02.031450 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:36:02.066852 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:02.066820 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4acdc09c-aeff-4e7d-a67f-b8fe04ce5736-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9346a31-qnmmw\" (UID: \"4acdc09c-aeff-4e7d-a67f-b8fe04ce5736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9346a31-qnmmw" Oct 30 14:36:02.066852 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:02.066853 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8xv6\" (UniqueName: \"kubernetes.io/projected/4acdc09c-aeff-4e7d-a67f-b8fe04ce5736-kube-api-access-p8xv6\") pod \"maas-api-revoke-sub-d9346a31-qnmmw\" (UID: \"4acdc09c-aeff-4e7d-a67f-b8fe04ce5736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9346a31-qnmmw" Oct 30 14:36:02.067060 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:02.066981 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4acdc09c-aeff-4e7d-a67f-b8fe04ce5736-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9346a31-qnmmw\" (UID: \"4acdc09c-aeff-4e7d-a67f-b8fe04ce5736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9346a31-qnmmw" Oct 30 14:36:02.168327 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:02.168286 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4acdc09c-aeff-4e7d-a67f-b8fe04ce5736-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9346a31-qnmmw\" (UID: \"4acdc09c-aeff-4e7d-a67f-b8fe04ce5736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9346a31-qnmmw" Oct 30 14:36:02.168490 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:02.168336 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8xv6\" (UniqueName: \"kubernetes.io/projected/4acdc09c-aeff-4e7d-a67f-b8fe04ce5736-kube-api-access-p8xv6\") pod \"maas-api-revoke-sub-d9346a31-qnmmw\" (UID: \"4acdc09c-aeff-4e7d-a67f-b8fe04ce5736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9346a31-qnmmw" Oct 30 14:36:02.168490 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:02.168398 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4acdc09c-aeff-4e7d-a67f-b8fe04ce5736-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9346a31-qnmmw\" (UID: \"4acdc09c-aeff-4e7d-a67f-b8fe04ce5736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9346a31-qnmmw" Oct 30 14:36:02.169098 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:02.169077 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4acdc09c-aeff-4e7d-a67f-b8fe04ce5736-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9346a31-qnmmw\" (UID: \"4acdc09c-aeff-4e7d-a67f-b8fe04ce5736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9346a31-qnmmw" Oct 30 14:36:02.170785 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:02.170764 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4acdc09c-aeff-4e7d-a67f-b8fe04ce5736-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9346a31-qnmmw\" (UID: \"4acdc09c-aeff-4e7d-a67f-b8fe04ce5736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9346a31-qnmmw" Oct 30 14:36:02.194636 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:02.194611 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8xv6\" (UniqueName: \"kubernetes.io/projected/4acdc09c-aeff-4e7d-a67f-b8fe04ce5736-kube-api-access-p8xv6\") pod \"maas-api-revoke-sub-d9346a31-qnmmw\" (UID: \"4acdc09c-aeff-4e7d-a67f-b8fe04ce5736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9346a31-qnmmw" Oct 30 14:36:02.314069 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:02.314024 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:36:02.318327 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:02.318301 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9346a31-qnmmw" Oct 30 14:36:02.649765 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:02.649739 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9346a31-qnmmw"] Oct 30 14:36:02.652483 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:36:02.652447 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4acdc09c_aeff_4e7d_a67f_b8fe04ce5736.slice/crio-7ec5766ae6624f759a10a28aafed919c1fbd152a7d57b5056e96dd2e69a8a292 WatchSource:0}: Error finding container 7ec5766ae6624f759a10a28aafed919c1fbd152a7d57b5056e96dd2e69a8a292: Status 404 returned error can't find the container with id 7ec5766ae6624f759a10a28aafed919c1fbd152a7d57b5056e96dd2e69a8a292 Oct 30 14:36:02.969733 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:02.969700 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:36:02.997613 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:02.997586 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9346a31-qnmmw" event={"ID":"4acdc09c-aeff-4e7d-a67f-b8fe04ce5736","Type":"ContainerStarted","Data":"7ec5766ae6624f759a10a28aafed919c1fbd152a7d57b5056e96dd2e69a8a292"} Oct 30 14:36:03.212924 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:03.212821 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:36:04.007016 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:04.006828 4982 generic.go:356] "Generic (PLEG): container finished" podID="4acdc09c-aeff-4e7d-a67f-b8fe04ce5736" containerID="d9cdcdb199292e201af22223558e46bea6481d9d29d1ca5f54b174365b8f9645" exitCode=0 Oct 30 14:36:04.007016 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:04.006882 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9346a31-qnmmw" event={"ID":"4acdc09c-aeff-4e7d-a67f-b8fe04ce5736","Type":"ContainerDied","Data":"d9cdcdb199292e201af22223558e46bea6481d9d29d1ca5f54b174365b8f9645"} Oct 30 14:36:04.590708 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:04.589345 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:36:04.740933 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:04.740834 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:36:04.747253 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:04.747226 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d3a829-6fcfc75547-xlcmh"] Oct 30 14:36:04.751723 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:04.751706 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d3a829-6fcfc75547-xlcmh" Oct 30 14:36:04.773628 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:04.773601 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d3a829-6fcfc75547-xlcmh"] Oct 30 14:36:04.782436 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:04.782414 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-d3a829\"" Oct 30 14:36:04.782549 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:04.782470 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-d3a829-dockercfg-q24m5\"" Oct 30 14:36:04.850662 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:04.823506 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a566e97f-4f8a-499c-9630-7e5de078910a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d3a829-6fcfc75547-xlcmh\" (UID: \"a566e97f-4f8a-499c-9630-7e5de078910a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d3a829-6fcfc75547-xlcmh" Oct 30 14:36:04.850662 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:04.823680 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvbz9\" (UniqueName: \"kubernetes.io/projected/a566e97f-4f8a-499c-9630-7e5de078910a-kube-api-access-wvbz9\") pod \"payload-pre-processing-e2e-ipp-a-d3a829-6fcfc75547-xlcmh\" (UID: \"a566e97f-4f8a-499c-9630-7e5de078910a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d3a829-6fcfc75547-xlcmh" Oct 30 14:36:04.896299 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:04.896256 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d3a829-698854d99d-8f69p"] Oct 30 14:36:04.902601 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:04.902573 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d3a829-698854d99d-8f69p" Oct 30 14:36:04.926504 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:04.925314 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a566e97f-4f8a-499c-9630-7e5de078910a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d3a829-6fcfc75547-xlcmh\" (UID: \"a566e97f-4f8a-499c-9630-7e5de078910a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d3a829-6fcfc75547-xlcmh" Oct 30 14:36:04.926504 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:04.925456 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvbz9\" (UniqueName: \"kubernetes.io/projected/a566e97f-4f8a-499c-9630-7e5de078910a-kube-api-access-wvbz9\") pod \"payload-pre-processing-e2e-ipp-a-d3a829-6fcfc75547-xlcmh\" (UID: \"a566e97f-4f8a-499c-9630-7e5de078910a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d3a829-6fcfc75547-xlcmh" Oct 30 14:36:04.926937 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:04.926913 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a566e97f-4f8a-499c-9630-7e5de078910a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d3a829-6fcfc75547-xlcmh\" (UID: \"a566e97f-4f8a-499c-9630-7e5de078910a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d3a829-6fcfc75547-xlcmh" Oct 30 14:36:04.937073 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:04.937047 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d3a829-698854d99d-8f69p"] Oct 30 14:36:04.967359 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:04.967319 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvbz9\" (UniqueName: \"kubernetes.io/projected/a566e97f-4f8a-499c-9630-7e5de078910a-kube-api-access-wvbz9\") pod \"payload-pre-processing-e2e-ipp-a-d3a829-6fcfc75547-xlcmh\" (UID: \"a566e97f-4f8a-499c-9630-7e5de078910a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d3a829-6fcfc75547-xlcmh" Oct 30 14:36:05.027247 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:05.027216 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4ee9039-363a-4761-8b74-dd5ff95a4949-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d3a829-698854d99d-8f69p\" (UID: \"f4ee9039-363a-4761-8b74-dd5ff95a4949\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d3a829-698854d99d-8f69p" Oct 30 14:36:05.027675 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:05.027337 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnn86\" (UniqueName: \"kubernetes.io/projected/f4ee9039-363a-4761-8b74-dd5ff95a4949-kube-api-access-tnn86\") pod \"payload-processing-e2e-ipp-a-d3a829-698854d99d-8f69p\" (UID: \"f4ee9039-363a-4761-8b74-dd5ff95a4949\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d3a829-698854d99d-8f69p" Oct 30 14:36:05.066200 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:05.066169 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d3a829-6fcfc75547-xlcmh" Oct 30 14:36:05.129524 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:05.129129 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnn86\" (UniqueName: \"kubernetes.io/projected/f4ee9039-363a-4761-8b74-dd5ff95a4949-kube-api-access-tnn86\") pod \"payload-processing-e2e-ipp-a-d3a829-698854d99d-8f69p\" (UID: \"f4ee9039-363a-4761-8b74-dd5ff95a4949\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d3a829-698854d99d-8f69p" Oct 30 14:36:05.129524 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:05.129359 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4ee9039-363a-4761-8b74-dd5ff95a4949-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d3a829-698854d99d-8f69p\" (UID: \"f4ee9039-363a-4761-8b74-dd5ff95a4949\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d3a829-698854d99d-8f69p" Oct 30 14:36:05.130617 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:05.130590 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4ee9039-363a-4761-8b74-dd5ff95a4949-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d3a829-698854d99d-8f69p\" (UID: \"f4ee9039-363a-4761-8b74-dd5ff95a4949\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d3a829-698854d99d-8f69p" Oct 30 14:36:05.161210 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:05.161131 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnn86\" (UniqueName: \"kubernetes.io/projected/f4ee9039-363a-4761-8b74-dd5ff95a4949-kube-api-access-tnn86\") pod \"payload-processing-e2e-ipp-a-d3a829-698854d99d-8f69p\" (UID: \"f4ee9039-363a-4761-8b74-dd5ff95a4949\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d3a829-698854d99d-8f69p" Oct 30 14:36:05.200475 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:05.200445 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9346a31-qnmmw" Oct 30 14:36:05.236030 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:05.235999 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d3a829-698854d99d-8f69p" Oct 30 14:36:05.331181 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:05.331153 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4acdc09c-aeff-4e7d-a67f-b8fe04ce5736-kube-api-access-p8xv6\" (UniqueName: \"kubernetes.io/projected/4acdc09c-aeff-4e7d-a67f-b8fe04ce5736-kube-api-access-p8xv6\") pod \"4acdc09c-aeff-4e7d-a67f-b8fe04ce5736\" (UID: \"4acdc09c-aeff-4e7d-a67f-b8fe04ce5736\") " Oct 30 14:36:05.331320 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:05.331221 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4acdc09c-aeff-4e7d-a67f-b8fe04ce5736-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4acdc09c-aeff-4e7d-a67f-b8fe04ce5736-maas-api-openshift-ca\") pod \"4acdc09c-aeff-4e7d-a67f-b8fe04ce5736\" (UID: \"4acdc09c-aeff-4e7d-a67f-b8fe04ce5736\") " Oct 30 14:36:05.331320 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:05.331269 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4acdc09c-aeff-4e7d-a67f-b8fe04ce5736-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4acdc09c-aeff-4e7d-a67f-b8fe04ce5736-maas-api-xks-ca\") pod \"4acdc09c-aeff-4e7d-a67f-b8fe04ce5736\" (UID: \"4acdc09c-aeff-4e7d-a67f-b8fe04ce5736\") " Oct 30 14:36:05.331824 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:05.331781 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4acdc09c-aeff-4e7d-a67f-b8fe04ce5736-maas-api-openshift-ca" pod "4acdc09c-aeff-4e7d-a67f-b8fe04ce5736" (UID: "4acdc09c-aeff-4e7d-a67f-b8fe04ce5736"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:36:05.333365 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:05.333342 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4acdc09c-aeff-4e7d-a67f-b8fe04ce5736-kube-api-access-p8xv6" pod "4acdc09c-aeff-4e7d-a67f-b8fe04ce5736" (UID: "4acdc09c-aeff-4e7d-a67f-b8fe04ce5736"). InnerVolumeSpecName "kube-api-access-p8xv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:36:05.334292 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:05.334270 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4acdc09c-aeff-4e7d-a67f-b8fe04ce5736-maas-api-xks-ca" pod "4acdc09c-aeff-4e7d-a67f-b8fe04ce5736" (UID: "4acdc09c-aeff-4e7d-a67f-b8fe04ce5736"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:36:05.412691 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:05.412662 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d3a829-6fcfc75547-xlcmh"] Oct 30 14:36:05.413682 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:36:05.413657 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda566e97f_4f8a_499c_9630_7e5de078910a.slice/crio-e9e2271f31cbad9f2020e2a208c874d5072523c601409c9f6117e7052b982b55 WatchSource:0}: Error finding container e9e2271f31cbad9f2020e2a208c874d5072523c601409c9f6117e7052b982b55: Status 404 returned error can't find the container with id e9e2271f31cbad9f2020e2a208c874d5072523c601409c9f6117e7052b982b55 Oct 30 14:36:05.432824 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:05.432793 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8xv6\" (UniqueName: \"kubernetes.io/projected/4acdc09c-aeff-4e7d-a67f-b8fe04ce5736-kube-api-access-p8xv6\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:36:05.432939 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:05.432825 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4acdc09c-aeff-4e7d-a67f-b8fe04ce5736-maas-api-openshift-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:36:05.432939 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:05.432839 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4acdc09c-aeff-4e7d-a67f-b8fe04ce5736-maas-api-xks-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:36:05.537656 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:05.537619 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-e371c3/rate-test-model-e371c3-kserve-fb686cfb5-bn68k" Oct 30 14:36:05.552382 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:36:05.552347 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4ee9039_363a_4761_8b74_dd5ff95a4949.slice/crio-48bf60565d7a804e3a13eced665cf9ff5ab799397f34785319e2d14ae60febd8 WatchSource:0}: Error finding container 48bf60565d7a804e3a13eced665cf9ff5ab799397f34785319e2d14ae60febd8: Status 404 returned error can't find the container with id 48bf60565d7a804e3a13eced665cf9ff5ab799397f34785319e2d14ae60febd8 Oct 30 14:36:05.552527 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:05.552502 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d3a829-698854d99d-8f69p"] Oct 30 14:36:05.555865 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:05.555846 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-e371c3/rate-test-model-e371c3-kserve-fb686cfb5-bn68k" Oct 30 14:36:06.026020 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:06.025982 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d3a829-6fcfc75547-xlcmh" event={"ID":"a566e97f-4f8a-499c-9630-7e5de078910a","Type":"ContainerStarted","Data":"f307e6f67389ed1e8446f2a983167c44be53f68fe1609c2d9bc38d90ce788d7e"} Oct 30 14:36:06.026192 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:06.026047 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d3a829-6fcfc75547-xlcmh" event={"ID":"a566e97f-4f8a-499c-9630-7e5de078910a","Type":"ContainerStarted","Data":"e9e2271f31cbad9f2020e2a208c874d5072523c601409c9f6117e7052b982b55"} Oct 30 14:36:06.026192 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:06.026099 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d3a829-6fcfc75547-xlcmh" Oct 30 14:36:06.027714 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:06.027685 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9346a31-qnmmw" event={"ID":"4acdc09c-aeff-4e7d-a67f-b8fe04ce5736","Type":"ContainerDied","Data":"7ec5766ae6624f759a10a28aafed919c1fbd152a7d57b5056e96dd2e69a8a292"} Oct 30 14:36:06.028120 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:06.027735 4982 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ec5766ae6624f759a10a28aafed919c1fbd152a7d57b5056e96dd2e69a8a292" Oct 30 14:36:06.028120 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:06.027796 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9346a31-qnmmw" Oct 30 14:36:06.031709 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:06.031679 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d3a829-698854d99d-8f69p" event={"ID":"f4ee9039-363a-4761-8b74-dd5ff95a4949","Type":"ContainerStarted","Data":"e4701b29940ff327330146ff3e51c8d839a7a734761f812ce0b23d6747e0a2f4"} Oct 30 14:36:06.031836 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:06.031730 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d3a829-698854d99d-8f69p" event={"ID":"f4ee9039-363a-4761-8b74-dd5ff95a4949","Type":"ContainerStarted","Data":"48bf60565d7a804e3a13eced665cf9ff5ab799397f34785319e2d14ae60febd8"} Oct 30 14:36:06.031836 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:06.031759 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d3a829-698854d99d-8f69p" Oct 30 14:36:06.043536 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:06.043512 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-e371c3/rate-test-model-e371c3-kserve-fb686cfb5-bn68k" Oct 30 14:36:06.075203 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:06.075154 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d3a829-6fcfc75547-xlcmh" podStartSLOduration=2.075138645 podStartE2EDuration="2.075138645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:36:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:36:06.070006839 +0000 UTC m=+2835.814140881" watchObservedRunningTime="2026-10-30 14:36:06.075138645 +0000 UTC m=+2835.819272688" Oct 30 14:36:06.161835 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:06.161761 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-d3a829-698854d99d-8f69p" podStartSLOduration=2.161741702 podStartE2EDuration="2.161741702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:36:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:36:06.160674061 +0000 UTC m=+2835.904808103" watchObservedRunningTime="2026-10-30 14:36:06.161741702 +0000 UTC m=+2835.905875743" Oct 30 14:36:06.577737 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:06.577595 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:36:08.390284 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:08.390250 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:36:10.389750 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:10.389704 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:36:11.036332 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:11.036288 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18b719b9-qrvcd"] Oct 30 14:36:11.049391 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:11.049360 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18b719b9-qrvcd"] Oct 30 14:36:12.238064 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:12.238018 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:36:12.362084 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:12.362048 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6f920ce-qgqnn"] Oct 30 14:36:12.362676 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:12.362657 4982 cpu_manager.go:414] "containerMap: removing container" podUID="4acdc09c-aeff-4e7d-a67f-b8fe04ce5736" containerName="revoke-keys" Oct 30 14:36:12.362676 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:12.362673 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4acdc09c-aeff-4e7d-a67f-b8fe04ce5736" containerName="revoke-keys" Oct 30 14:36:12.362841 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:12.362765 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="4acdc09c-aeff-4e7d-a67f-b8fe04ce5736" containerName="revoke-keys" Oct 30 14:36:12.367548 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:12.367531 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6f920ce-qgqnn" Oct 30 14:36:12.387950 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:12.387922 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6f920ce-qgqnn"] Oct 30 14:36:12.396207 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:12.396180 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:36:12.396335 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:12.396247 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:36:12.404186 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:12.404162 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c88117c-d932-43c2-8a2a-9402c12ce0f7" path="/var/lib/kubelet/pods/9c88117c-d932-43c2-8a2a-9402c12ce0f7/volumes" Oct 30 14:36:12.522495 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:12.522408 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wlwd\" (UniqueName: \"kubernetes.io/projected/c013f868-ccac-4409-b5cc-ce01b6dd954a-kube-api-access-5wlwd\") pod \"maas-api-revoke-sub-c6f920ce-qgqnn\" (UID: \"c013f868-ccac-4409-b5cc-ce01b6dd954a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6f920ce-qgqnn" Oct 30 14:36:12.522726 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:12.522586 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c013f868-ccac-4409-b5cc-ce01b6dd954a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6f920ce-qgqnn\" (UID: \"c013f868-ccac-4409-b5cc-ce01b6dd954a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6f920ce-qgqnn" Oct 30 14:36:12.522726 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:12.522620 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c013f868-ccac-4409-b5cc-ce01b6dd954a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6f920ce-qgqnn\" (UID: \"c013f868-ccac-4409-b5cc-ce01b6dd954a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6f920ce-qgqnn" Oct 30 14:36:12.623963 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:12.623934 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c013f868-ccac-4409-b5cc-ce01b6dd954a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6f920ce-qgqnn\" (UID: \"c013f868-ccac-4409-b5cc-ce01b6dd954a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6f920ce-qgqnn" Oct 30 14:36:12.624116 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:12.623973 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c013f868-ccac-4409-b5cc-ce01b6dd954a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6f920ce-qgqnn\" (UID: \"c013f868-ccac-4409-b5cc-ce01b6dd954a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6f920ce-qgqnn" Oct 30 14:36:12.624116 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:12.624061 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wlwd\" (UniqueName: \"kubernetes.io/projected/c013f868-ccac-4409-b5cc-ce01b6dd954a-kube-api-access-5wlwd\") pod \"maas-api-revoke-sub-c6f920ce-qgqnn\" (UID: \"c013f868-ccac-4409-b5cc-ce01b6dd954a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6f920ce-qgqnn" Oct 30 14:36:12.624888 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:12.624864 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c013f868-ccac-4409-b5cc-ce01b6dd954a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6f920ce-qgqnn\" (UID: \"c013f868-ccac-4409-b5cc-ce01b6dd954a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6f920ce-qgqnn" Oct 30 14:36:12.626443 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:12.626417 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c013f868-ccac-4409-b5cc-ce01b6dd954a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6f920ce-qgqnn\" (UID: \"c013f868-ccac-4409-b5cc-ce01b6dd954a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6f920ce-qgqnn" Oct 30 14:36:12.649597 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:12.649573 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wlwd\" (UniqueName: \"kubernetes.io/projected/c013f868-ccac-4409-b5cc-ce01b6dd954a-kube-api-access-5wlwd\") pod \"maas-api-revoke-sub-c6f920ce-qgqnn\" (UID: \"c013f868-ccac-4409-b5cc-ce01b6dd954a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6f920ce-qgqnn" Oct 30 14:36:12.681566 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:12.681541 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6f920ce-qgqnn" Oct 30 14:36:12.846369 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:12.846333 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:36:13.000302 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:13.000272 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6f920ce-qgqnn"] Oct 30 14:36:13.002201 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:36:13.002174 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc013f868_ccac_4409_b5cc_ce01b6dd954a.slice/crio-d2a220fdc43b7e83ffec66a6a5a1e8175925fa0a02dd62c99e151fb63e5cb58f WatchSource:0}: Error finding container d2a220fdc43b7e83ffec66a6a5a1e8175925fa0a02dd62c99e151fb63e5cb58f: Status 404 returned error can't find the container with id d2a220fdc43b7e83ffec66a6a5a1e8175925fa0a02dd62c99e151fb63e5cb58f Oct 30 14:36:13.087468 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:13.087439 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6f920ce-qgqnn" event={"ID":"c013f868-ccac-4409-b5cc-ce01b6dd954a","Type":"ContainerStarted","Data":"d2a220fdc43b7e83ffec66a6a5a1e8175925fa0a02dd62c99e151fb63e5cb58f"} Oct 30 14:36:14.098007 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:14.097974 4982 generic.go:356] "Generic (PLEG): container finished" podID="c013f868-ccac-4409-b5cc-ce01b6dd954a" containerID="feb1ac0485a55d78d89a6c7629dd62d0880d12b317623b049a412d447f808928" exitCode=0 Oct 30 14:36:14.098501 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:14.098037 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6f920ce-qgqnn" event={"ID":"c013f868-ccac-4409-b5cc-ce01b6dd954a","Type":"ContainerDied","Data":"feb1ac0485a55d78d89a6c7629dd62d0880d12b317623b049a412d447f808928"} Oct 30 14:36:14.240557 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:14.240520 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:36:15.267939 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:15.267914 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6f920ce-qgqnn" Oct 30 14:36:15.362061 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:15.361991 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c013f868-ccac-4409-b5cc-ce01b6dd954a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c013f868-ccac-4409-b5cc-ce01b6dd954a-maas-api-openshift-ca\") pod \"c013f868-ccac-4409-b5cc-ce01b6dd954a\" (UID: \"c013f868-ccac-4409-b5cc-ce01b6dd954a\") " Oct 30 14:36:15.362061 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:15.362060 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c013f868-ccac-4409-b5cc-ce01b6dd954a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c013f868-ccac-4409-b5cc-ce01b6dd954a-maas-api-xks-ca\") pod \"c013f868-ccac-4409-b5cc-ce01b6dd954a\" (UID: \"c013f868-ccac-4409-b5cc-ce01b6dd954a\") " Oct 30 14:36:15.362285 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:15.362175 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c013f868-ccac-4409-b5cc-ce01b6dd954a-kube-api-access-5wlwd\" (UniqueName: \"kubernetes.io/projected/c013f868-ccac-4409-b5cc-ce01b6dd954a-kube-api-access-5wlwd\") pod \"c013f868-ccac-4409-b5cc-ce01b6dd954a\" (UID: \"c013f868-ccac-4409-b5cc-ce01b6dd954a\") " Oct 30 14:36:15.362389 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:15.362365 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c013f868-ccac-4409-b5cc-ce01b6dd954a-maas-api-openshift-ca" pod "c013f868-ccac-4409-b5cc-ce01b6dd954a" (UID: "c013f868-ccac-4409-b5cc-ce01b6dd954a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:36:15.362667 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:15.362619 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c013f868-ccac-4409-b5cc-ce01b6dd954a-maas-api-openshift-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:36:15.364106 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:15.364084 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c013f868-ccac-4409-b5cc-ce01b6dd954a-maas-api-xks-ca" pod "c013f868-ccac-4409-b5cc-ce01b6dd954a" (UID: "c013f868-ccac-4409-b5cc-ce01b6dd954a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:36:15.364106 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:15.364094 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c013f868-ccac-4409-b5cc-ce01b6dd954a-kube-api-access-5wlwd" pod "c013f868-ccac-4409-b5cc-ce01b6dd954a" (UID: "c013f868-ccac-4409-b5cc-ce01b6dd954a"). InnerVolumeSpecName "kube-api-access-5wlwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:36:15.463517 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:15.463494 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wlwd\" (UniqueName: \"kubernetes.io/projected/c013f868-ccac-4409-b5cc-ce01b6dd954a-kube-api-access-5wlwd\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:36:15.463517 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:15.463517 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c013f868-ccac-4409-b5cc-ce01b6dd954a-maas-api-xks-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:36:16.101733 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:16.101699 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:36:16.131391 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:16.131358 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6f920ce-qgqnn" event={"ID":"c013f868-ccac-4409-b5cc-ce01b6dd954a","Type":"ContainerDied","Data":"d2a220fdc43b7e83ffec66a6a5a1e8175925fa0a02dd62c99e151fb63e5cb58f"} Oct 30 14:36:16.131566 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:16.131409 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6f920ce-qgqnn" Oct 30 14:36:16.131566 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:16.131429 4982 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2a220fdc43b7e83ffec66a6a5a1e8175925fa0a02dd62c99e151fb63e5cb58f" Oct 30 14:36:16.259786 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:16.259748 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:36:16.417058 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:16.416975 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:36:17.042320 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:17.042289 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d3a829-698854d99d-8f69p" Oct 30 14:36:17.042595 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:17.042425 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d3a829-6fcfc75547-xlcmh" Oct 30 14:36:18.096795 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:18.096759 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:36:20.188785 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:20.188753 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:36:20.275130 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:20.275095 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-a59ecb/rate-test-model-a59ecb-kserve-8678fc4b5c-pr4bs"] Oct 30 14:36:20.275715 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:20.275694 4982 cpu_manager.go:414] "containerMap: removing container" podUID="c013f868-ccac-4409-b5cc-ce01b6dd954a" containerName="revoke-keys" Oct 30 14:36:20.275715 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:20.275710 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c013f868-ccac-4409-b5cc-ce01b6dd954a" containerName="revoke-keys" Oct 30 14:36:20.275920 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:20.275810 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="c013f868-ccac-4409-b5cc-ce01b6dd954a" containerName="revoke-keys" Oct 30 14:36:20.279507 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:20.279487 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-a59ecb/rate-test-model-a59ecb-kserve-8678fc4b5c-pr4bs" Oct 30 14:36:20.306062 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:20.306034 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-a59ecb/rate-test-model-a59ecb-kserve-8678fc4b5c-pr4bs"] Oct 30 14:36:20.319668 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:20.319004 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-a59ecb\"/\"kube-root-ca.crt\"" Oct 30 14:36:20.319668 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:20.319158 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-a59ecb\"/\"default-dockercfg-6mpvj\"" Oct 30 14:36:20.319668 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:20.319283 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-a59ecb\"/\"rate-test-model-a59ecb-kserve-self-signed-certs\"" Oct 30 14:36:20.319668 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:20.319378 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-a59ecb\"/\"openshift-service-ca.crt\"" Oct 30 14:36:20.331412 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:20.331376 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cb47566-b4c9-4384-a2e8-f3428ad8b9a4-model-cache\") pod \"rate-test-model-a59ecb-kserve-8678fc4b5c-pr4bs\" (UID: \"8cb47566-b4c9-4384-a2e8-f3428ad8b9a4\") " pod="ai-tenant-e2e-shared-b-w4-a59ecb/rate-test-model-a59ecb-kserve-8678fc4b5c-pr4bs" Oct 30 14:36:20.331555 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:20.331507 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cb47566-b4c9-4384-a2e8-f3428ad8b9a4-dshm\") pod \"rate-test-model-a59ecb-kserve-8678fc4b5c-pr4bs\" (UID: \"8cb47566-b4c9-4384-a2e8-f3428ad8b9a4\") " pod="ai-tenant-e2e-shared-b-w4-a59ecb/rate-test-model-a59ecb-kserve-8678fc4b5c-pr4bs" Oct 30 14:36:20.331624 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:20.331555 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z28vd\" (UniqueName: \"kubernetes.io/projected/8cb47566-b4c9-4384-a2e8-f3428ad8b9a4-kube-api-access-z28vd\") pod \"rate-test-model-a59ecb-kserve-8678fc4b5c-pr4bs\" (UID: \"8cb47566-b4c9-4384-a2e8-f3428ad8b9a4\") " pod="ai-tenant-e2e-shared-b-w4-a59ecb/rate-test-model-a59ecb-kserve-8678fc4b5c-pr4bs" Oct 30 14:36:20.331624 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:20.331598 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cb47566-b4c9-4384-a2e8-f3428ad8b9a4-tls-certs\") pod \"rate-test-model-a59ecb-kserve-8678fc4b5c-pr4bs\" (UID: \"8cb47566-b4c9-4384-a2e8-f3428ad8b9a4\") " pod="ai-tenant-e2e-shared-b-w4-a59ecb/rate-test-model-a59ecb-kserve-8678fc4b5c-pr4bs" Oct 30 14:36:20.331798 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:20.331714 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cb47566-b4c9-4384-a2e8-f3428ad8b9a4-home\") pod \"rate-test-model-a59ecb-kserve-8678fc4b5c-pr4bs\" (UID: \"8cb47566-b4c9-4384-a2e8-f3428ad8b9a4\") " pod="ai-tenant-e2e-shared-b-w4-a59ecb/rate-test-model-a59ecb-kserve-8678fc4b5c-pr4bs" Oct 30 14:36:20.331858 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:20.331798 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cb47566-b4c9-4384-a2e8-f3428ad8b9a4-tmp-dir\") pod \"rate-test-model-a59ecb-kserve-8678fc4b5c-pr4bs\" (UID: \"8cb47566-b4c9-4384-a2e8-f3428ad8b9a4\") " pod="ai-tenant-e2e-shared-b-w4-a59ecb/rate-test-model-a59ecb-kserve-8678fc4b5c-pr4bs" Oct 30 14:36:20.422198 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:20.422163 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:36:20.432463 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:20.432434 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cb47566-b4c9-4384-a2e8-f3428ad8b9a4-tmp-dir\") pod \"rate-test-model-a59ecb-kserve-8678fc4b5c-pr4bs\" (UID: \"8cb47566-b4c9-4384-a2e8-f3428ad8b9a4\") " pod="ai-tenant-e2e-shared-b-w4-a59ecb/rate-test-model-a59ecb-kserve-8678fc4b5c-pr4bs" Oct 30 14:36:20.432620 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:20.432542 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cb47566-b4c9-4384-a2e8-f3428ad8b9a4-model-cache\") pod \"rate-test-model-a59ecb-kserve-8678fc4b5c-pr4bs\" (UID: \"8cb47566-b4c9-4384-a2e8-f3428ad8b9a4\") " pod="ai-tenant-e2e-shared-b-w4-a59ecb/rate-test-model-a59ecb-kserve-8678fc4b5c-pr4bs" Oct 30 14:36:20.432854 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:20.432830 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cb47566-b4c9-4384-a2e8-f3428ad8b9a4-dshm\") pod \"rate-test-model-a59ecb-kserve-8678fc4b5c-pr4bs\" (UID: \"8cb47566-b4c9-4384-a2e8-f3428ad8b9a4\") " pod="ai-tenant-e2e-shared-b-w4-a59ecb/rate-test-model-a59ecb-kserve-8678fc4b5c-pr4bs" Oct 30 14:36:20.432985 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:20.432878 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z28vd\" (UniqueName: \"kubernetes.io/projected/8cb47566-b4c9-4384-a2e8-f3428ad8b9a4-kube-api-access-z28vd\") pod \"rate-test-model-a59ecb-kserve-8678fc4b5c-pr4bs\" (UID: \"8cb47566-b4c9-4384-a2e8-f3428ad8b9a4\") " pod="ai-tenant-e2e-shared-b-w4-a59ecb/rate-test-model-a59ecb-kserve-8678fc4b5c-pr4bs" Oct 30 14:36:20.432985 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:20.432934 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cb47566-b4c9-4384-a2e8-f3428ad8b9a4-tls-certs\") pod \"rate-test-model-a59ecb-kserve-8678fc4b5c-pr4bs\" (UID: \"8cb47566-b4c9-4384-a2e8-f3428ad8b9a4\") " pod="ai-tenant-e2e-shared-b-w4-a59ecb/rate-test-model-a59ecb-kserve-8678fc4b5c-pr4bs" Oct 30 14:36:20.432985 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:20.432939 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cb47566-b4c9-4384-a2e8-f3428ad8b9a4-tmp-dir\") pod \"rate-test-model-a59ecb-kserve-8678fc4b5c-pr4bs\" (UID: \"8cb47566-b4c9-4384-a2e8-f3428ad8b9a4\") " pod="ai-tenant-e2e-shared-b-w4-a59ecb/rate-test-model-a59ecb-kserve-8678fc4b5c-pr4bs" Oct 30 14:36:20.432985 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:20.432967 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cb47566-b4c9-4384-a2e8-f3428ad8b9a4-model-cache\") pod \"rate-test-model-a59ecb-kserve-8678fc4b5c-pr4bs\" (UID: \"8cb47566-b4c9-4384-a2e8-f3428ad8b9a4\") " pod="ai-tenant-e2e-shared-b-w4-a59ecb/rate-test-model-a59ecb-kserve-8678fc4b5c-pr4bs" Oct 30 14:36:20.433216 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:20.433006 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cb47566-b4c9-4384-a2e8-f3428ad8b9a4-home\") pod \"rate-test-model-a59ecb-kserve-8678fc4b5c-pr4bs\" (UID: \"8cb47566-b4c9-4384-a2e8-f3428ad8b9a4\") " pod="ai-tenant-e2e-shared-b-w4-a59ecb/rate-test-model-a59ecb-kserve-8678fc4b5c-pr4bs" Oct 30 14:36:20.433278 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:20.433249 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cb47566-b4c9-4384-a2e8-f3428ad8b9a4-home\") pod \"rate-test-model-a59ecb-kserve-8678fc4b5c-pr4bs\" (UID: \"8cb47566-b4c9-4384-a2e8-f3428ad8b9a4\") " pod="ai-tenant-e2e-shared-b-w4-a59ecb/rate-test-model-a59ecb-kserve-8678fc4b5c-pr4bs" Oct 30 14:36:20.435476 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:20.435446 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cb47566-b4c9-4384-a2e8-f3428ad8b9a4-tls-certs\") pod \"rate-test-model-a59ecb-kserve-8678fc4b5c-pr4bs\" (UID: \"8cb47566-b4c9-4384-a2e8-f3428ad8b9a4\") " pod="ai-tenant-e2e-shared-b-w4-a59ecb/rate-test-model-a59ecb-kserve-8678fc4b5c-pr4bs" Oct 30 14:36:20.435806 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:20.435787 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cb47566-b4c9-4384-a2e8-f3428ad8b9a4-dshm\") pod \"rate-test-model-a59ecb-kserve-8678fc4b5c-pr4bs\" (UID: \"8cb47566-b4c9-4384-a2e8-f3428ad8b9a4\") " pod="ai-tenant-e2e-shared-b-w4-a59ecb/rate-test-model-a59ecb-kserve-8678fc4b5c-pr4bs" Oct 30 14:36:20.467103 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:20.467054 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z28vd\" (UniqueName: \"kubernetes.io/projected/8cb47566-b4c9-4384-a2e8-f3428ad8b9a4-kube-api-access-z28vd\") pod \"rate-test-model-a59ecb-kserve-8678fc4b5c-pr4bs\" (UID: \"8cb47566-b4c9-4384-a2e8-f3428ad8b9a4\") " pod="ai-tenant-e2e-shared-b-w4-a59ecb/rate-test-model-a59ecb-kserve-8678fc4b5c-pr4bs" Oct 30 14:36:20.609664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:20.607113 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-a59ecb/rate-test-model-a59ecb-kserve-8678fc4b5c-pr4bs" Oct 30 14:36:21.001095 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:21.000563 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-a59ecb/rate-test-model-a59ecb-kserve-8678fc4b5c-pr4bs"] Oct 30 14:36:21.002775 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:36:21.002742 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8cb47566_b4c9_4384_a2e8_f3428ad8b9a4.slice/crio-2eb82e7e0b1f94b3f232ff6f038abda059f0f6eadc92a9e19694f222487479e4 WatchSource:0}: Error finding container 2eb82e7e0b1f94b3f232ff6f038abda059f0f6eadc92a9e19694f222487479e4: Status 404 returned error can't find the container with id 2eb82e7e0b1f94b3f232ff6f038abda059f0f6eadc92a9e19694f222487479e4 Oct 30 14:36:21.173624 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:21.173587 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-a59ecb/rate-test-model-a59ecb-kserve-8678fc4b5c-pr4bs" event={"ID":"8cb47566-b4c9-4384-a2e8-f3428ad8b9a4","Type":"ContainerStarted","Data":"d8ed49bd890a704717ac3adead502823c735eff30b6fbb2f47b0612c15c091d0"} Oct 30 14:36:21.173799 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:21.173676 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-a59ecb/rate-test-model-a59ecb-kserve-8678fc4b5c-pr4bs" event={"ID":"8cb47566-b4c9-4384-a2e8-f3428ad8b9a4","Type":"ContainerStarted","Data":"2eb82e7e0b1f94b3f232ff6f038abda059f0f6eadc92a9e19694f222487479e4"} Oct 30 14:36:21.228532 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:21.228478 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-a59ecb/rate-test-model-a59ecb-kserve-8678fc4b5c-pr4bs" podStartSLOduration=1.228460779 podStartE2EDuration="1.228460779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:36:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:36:21.228012876 +0000 UTC m=+2850.972146918" watchObservedRunningTime="2026-10-30 14:36:21.228460779 +0000 UTC m=+2850.972594823" Oct 30 14:36:21.609181 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:21.609140 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-a59ecb/rate-test-model-a59ecb-kserve-8678fc4b5c-pr4bs" Oct 30 14:36:22.123976 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:22.123912 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:36:24.009183 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:24.009141 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:36:26.239709 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:26.239674 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:36:28.012583 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:36:28.012544 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:36:28.262434 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:28.262399 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:36:30.094976 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:30.094938 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:36:30.609429 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:30.609387 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-a59ecb/rate-test-model-a59ecb-kserve-8678fc4b5c-pr4bs" Oct 30 14:36:30.621655 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:30.621614 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-a59ecb/rate-test-model-a59ecb-kserve-8678fc4b5c-pr4bs" Oct 30 14:36:31.267546 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:31.267507 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-a59ecb/rate-test-model-a59ecb-kserve-8678fc4b5c-pr4bs" Oct 30 14:36:32.088734 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:32.088700 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:36:32.529290 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:32.529203 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:36:32.672705 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:32.672669 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:36:34.208671 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:34.206691 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:36:36.333491 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:36.333460 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:36:37.636116 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:37.636082 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b4e0be4-jd99f"] Oct 30 14:36:37.641447 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:37.641423 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b4e0be4-jd99f" Oct 30 14:36:37.661948 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:37.661911 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b4e0be4-jd99f"] Oct 30 14:36:37.670630 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:37.670606 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:36:37.670630 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:37.670606 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:36:37.794084 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:37.794055 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rtmb\" (UniqueName: \"kubernetes.io/projected/3a34762f-5688-47e3-b7d4-2cc93b375358-kube-api-access-6rtmb\") pod \"maas-api-revoke-sub-8b4e0be4-jd99f\" (UID: \"3a34762f-5688-47e3-b7d4-2cc93b375358\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b4e0be4-jd99f" Oct 30 14:36:37.794242 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:37.794184 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a34762f-5688-47e3-b7d4-2cc93b375358-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b4e0be4-jd99f\" (UID: \"3a34762f-5688-47e3-b7d4-2cc93b375358\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b4e0be4-jd99f" Oct 30 14:36:37.794242 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:37.794213 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a34762f-5688-47e3-b7d4-2cc93b375358-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b4e0be4-jd99f\" (UID: \"3a34762f-5688-47e3-b7d4-2cc93b375358\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b4e0be4-jd99f" Oct 30 14:36:37.895855 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:37.895772 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a34762f-5688-47e3-b7d4-2cc93b375358-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b4e0be4-jd99f\" (UID: \"3a34762f-5688-47e3-b7d4-2cc93b375358\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b4e0be4-jd99f" Oct 30 14:36:37.895855 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:37.895818 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a34762f-5688-47e3-b7d4-2cc93b375358-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b4e0be4-jd99f\" (UID: \"3a34762f-5688-47e3-b7d4-2cc93b375358\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b4e0be4-jd99f" Oct 30 14:36:37.896064 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:37.895964 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rtmb\" (UniqueName: \"kubernetes.io/projected/3a34762f-5688-47e3-b7d4-2cc93b375358-kube-api-access-6rtmb\") pod \"maas-api-revoke-sub-8b4e0be4-jd99f\" (UID: \"3a34762f-5688-47e3-b7d4-2cc93b375358\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b4e0be4-jd99f" Oct 30 14:36:37.896525 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:37.896500 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a34762f-5688-47e3-b7d4-2cc93b375358-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b4e0be4-jd99f\" (UID: \"3a34762f-5688-47e3-b7d4-2cc93b375358\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b4e0be4-jd99f" Oct 30 14:36:37.898298 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:37.898279 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a34762f-5688-47e3-b7d4-2cc93b375358-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b4e0be4-jd99f\" (UID: \"3a34762f-5688-47e3-b7d4-2cc93b375358\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b4e0be4-jd99f" Oct 30 14:36:37.922002 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:37.921975 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rtmb\" (UniqueName: \"kubernetes.io/projected/3a34762f-5688-47e3-b7d4-2cc93b375358-kube-api-access-6rtmb\") pod \"maas-api-revoke-sub-8b4e0be4-jd99f\" (UID: \"3a34762f-5688-47e3-b7d4-2cc93b375358\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b4e0be4-jd99f" Oct 30 14:36:37.956470 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:37.956449 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b4e0be4-jd99f" Oct 30 14:36:38.268060 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:38.268032 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b4e0be4-jd99f"] Oct 30 14:36:38.269135 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:36:38.269102 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a34762f_5688_47e3_b7d4_2cc93b375358.slice/crio-ae3e595347361c388e0803fa8e624aea545367a6144a874cf999232ec5093903 WatchSource:0}: Error finding container ae3e595347361c388e0803fa8e624aea545367a6144a874cf999232ec5093903: Status 404 returned error can't find the container with id ae3e595347361c388e0803fa8e624aea545367a6144a874cf999232ec5093903 Oct 30 14:36:38.312388 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:38.312357 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b4e0be4-jd99f" event={"ID":"3a34762f-5688-47e3-b7d4-2cc93b375358","Type":"ContainerStarted","Data":"ae3e595347361c388e0803fa8e624aea545367a6144a874cf999232ec5093903"} Oct 30 14:36:38.386025 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:38.385987 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:36:39.036385 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:39.036303 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00aba928-8v68d"] Oct 30 14:36:39.051738 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:39.051704 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00aba928-8v68d"] Oct 30 14:36:39.323932 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:39.323902 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b4e0be4-jd99f" event={"ID":"3a34762f-5688-47e3-b7d4-2cc93b375358","Type":"ContainerStarted","Data":"b70bd0b1db12584542b28fd4c9ff8153ae0aef5652e98f2d66f5bcd4306c44aa"} Oct 30 14:36:40.186959 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:40.186923 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:36:40.334014 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:40.333979 4982 generic.go:356] "Generic (PLEG): container finished" podID="3a34762f-5688-47e3-b7d4-2cc93b375358" containerID="b70bd0b1db12584542b28fd4c9ff8153ae0aef5652e98f2d66f5bcd4306c44aa" exitCode=0 Oct 30 14:36:40.334194 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:40.334044 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b4e0be4-jd99f" event={"ID":"3a34762f-5688-47e3-b7d4-2cc93b375358","Type":"ContainerDied","Data":"b70bd0b1db12584542b28fd4c9ff8153ae0aef5652e98f2d66f5bcd4306c44aa"} Oct 30 14:36:40.405591 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:40.405557 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="659a82d6-b679-4925-91d7-d8064ed22314" path="/var/lib/kubelet/pods/659a82d6-b679-4925-91d7-d8064ed22314/volumes" Oct 30 14:36:40.490546 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:40.490522 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b4e0be4-jd99f" Oct 30 14:36:40.632958 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:40.632925 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a34762f-5688-47e3-b7d4-2cc93b375358-kube-api-access-6rtmb\" (UniqueName: \"kubernetes.io/projected/3a34762f-5688-47e3-b7d4-2cc93b375358-kube-api-access-6rtmb\") pod \"3a34762f-5688-47e3-b7d4-2cc93b375358\" (UID: \"3a34762f-5688-47e3-b7d4-2cc93b375358\") " Oct 30 14:36:40.633140 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:40.633019 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a34762f-5688-47e3-b7d4-2cc93b375358-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a34762f-5688-47e3-b7d4-2cc93b375358-maas-api-openshift-ca\") pod \"3a34762f-5688-47e3-b7d4-2cc93b375358\" (UID: \"3a34762f-5688-47e3-b7d4-2cc93b375358\") " Oct 30 14:36:40.633140 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:40.633073 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a34762f-5688-47e3-b7d4-2cc93b375358-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a34762f-5688-47e3-b7d4-2cc93b375358-maas-api-xks-ca\") pod \"3a34762f-5688-47e3-b7d4-2cc93b375358\" (UID: \"3a34762f-5688-47e3-b7d4-2cc93b375358\") " Oct 30 14:36:40.633373 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:40.633345 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a34762f-5688-47e3-b7d4-2cc93b375358-maas-api-openshift-ca" pod "3a34762f-5688-47e3-b7d4-2cc93b375358" (UID: "3a34762f-5688-47e3-b7d4-2cc93b375358"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:36:40.633652 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:40.633626 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a34762f-5688-47e3-b7d4-2cc93b375358-maas-api-openshift-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:36:40.635001 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:40.634980 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a34762f-5688-47e3-b7d4-2cc93b375358-maas-api-xks-ca" pod "3a34762f-5688-47e3-b7d4-2cc93b375358" (UID: "3a34762f-5688-47e3-b7d4-2cc93b375358"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:36:40.635059 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:40.634983 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a34762f-5688-47e3-b7d4-2cc93b375358-kube-api-access-6rtmb" pod "3a34762f-5688-47e3-b7d4-2cc93b375358" (UID: "3a34762f-5688-47e3-b7d4-2cc93b375358"). InnerVolumeSpecName "kube-api-access-6rtmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:36:40.735339 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:40.735258 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a34762f-5688-47e3-b7d4-2cc93b375358-maas-api-xks-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:36:40.735339 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:40.735298 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rtmb\" (UniqueName: \"kubernetes.io/projected/3a34762f-5688-47e3-b7d4-2cc93b375358-kube-api-access-6rtmb\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:36:41.343418 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:41.343383 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b4e0be4-jd99f" Oct 30 14:36:41.343418 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:41.343405 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b4e0be4-jd99f" event={"ID":"3a34762f-5688-47e3-b7d4-2cc93b375358","Type":"ContainerDied","Data":"ae3e595347361c388e0803fa8e624aea545367a6144a874cf999232ec5093903"} Oct 30 14:36:41.343903 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:41.343462 4982 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae3e595347361c388e0803fa8e624aea545367a6144a874cf999232ec5093903" Oct 30 14:36:42.085801 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:42.085769 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:36:44.185523 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:44.185488 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:36:44.358441 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:44.358402 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:36:44.508120 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:44.508036 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:36:46.130748 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:46.130710 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:36:46.278698 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:46.278666 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:36:47.973951 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:47.973916 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-318d85fc-22c2x"] Oct 30 14:36:47.974509 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:47.974483 4982 cpu_manager.go:414] "containerMap: removing container" podUID="3a34762f-5688-47e3-b7d4-2cc93b375358" containerName="revoke-keys" Oct 30 14:36:47.974509 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:47.974499 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a34762f-5688-47e3-b7d4-2cc93b375358" containerName="revoke-keys" Oct 30 14:36:47.974729 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:47.974607 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a34762f-5688-47e3-b7d4-2cc93b375358" containerName="revoke-keys" Oct 30 14:36:47.979251 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:47.979231 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-318d85fc-22c2x" Oct 30 14:36:47.999610 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:47.999590 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-318d85fc-22c2x"] Oct 30 14:36:48.008537 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:48.008513 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:36:48.008700 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:48.008559 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:36:48.033722 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:48.033685 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78flp\" (UniqueName: \"kubernetes.io/projected/b7a27530-b0f4-446b-91c3-cd597557e224-kube-api-access-78flp\") pod \"maas-api-revoke-sub-318d85fc-22c2x\" (UID: \"b7a27530-b0f4-446b-91c3-cd597557e224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-318d85fc-22c2x" Oct 30 14:36:48.033871 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:48.033850 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7a27530-b0f4-446b-91c3-cd597557e224-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-318d85fc-22c2x\" (UID: \"b7a27530-b0f4-446b-91c3-cd597557e224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-318d85fc-22c2x" Oct 30 14:36:48.033941 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:48.033888 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7a27530-b0f4-446b-91c3-cd597557e224-maas-api-xks-ca\") pod \"maas-api-revoke-sub-318d85fc-22c2x\" (UID: \"b7a27530-b0f4-446b-91c3-cd597557e224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-318d85fc-22c2x" Oct 30 14:36:48.066935 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:48.066903 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:36:48.135386 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:48.135351 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78flp\" (UniqueName: \"kubernetes.io/projected/b7a27530-b0f4-446b-91c3-cd597557e224-kube-api-access-78flp\") pod \"maas-api-revoke-sub-318d85fc-22c2x\" (UID: \"b7a27530-b0f4-446b-91c3-cd597557e224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-318d85fc-22c2x" Oct 30 14:36:48.135603 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:48.135464 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7a27530-b0f4-446b-91c3-cd597557e224-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-318d85fc-22c2x\" (UID: \"b7a27530-b0f4-446b-91c3-cd597557e224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-318d85fc-22c2x" Oct 30 14:36:48.135603 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:48.135497 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7a27530-b0f4-446b-91c3-cd597557e224-maas-api-xks-ca\") pod \"maas-api-revoke-sub-318d85fc-22c2x\" (UID: \"b7a27530-b0f4-446b-91c3-cd597557e224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-318d85fc-22c2x" Oct 30 14:36:48.136287 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:48.136263 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7a27530-b0f4-446b-91c3-cd597557e224-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-318d85fc-22c2x\" (UID: \"b7a27530-b0f4-446b-91c3-cd597557e224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-318d85fc-22c2x" Oct 30 14:36:48.137985 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:48.137961 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7a27530-b0f4-446b-91c3-cd597557e224-maas-api-xks-ca\") pod \"maas-api-revoke-sub-318d85fc-22c2x\" (UID: \"b7a27530-b0f4-446b-91c3-cd597557e224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-318d85fc-22c2x" Oct 30 14:36:48.161251 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:48.161231 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78flp\" (UniqueName: \"kubernetes.io/projected/b7a27530-b0f4-446b-91c3-cd597557e224-kube-api-access-78flp\") pod \"maas-api-revoke-sub-318d85fc-22c2x\" (UID: \"b7a27530-b0f4-446b-91c3-cd597557e224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-318d85fc-22c2x" Oct 30 14:36:48.292939 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:48.292868 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-318d85fc-22c2x" Oct 30 14:36:48.612168 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:48.612134 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-318d85fc-22c2x"] Oct 30 14:36:48.613409 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:36:48.613380 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7a27530_b0f4_446b_91c3_cd597557e224.slice/crio-d0546e5eef897620cf3bbebf9c60a50e5d46b4c80823ed5f835e6dc32e0d2583 WatchSource:0}: Error finding container d0546e5eef897620cf3bbebf9c60a50e5d46b4c80823ed5f835e6dc32e0d2583: Status 404 returned error can't find the container with id d0546e5eef897620cf3bbebf9c60a50e5d46b4c80823ed5f835e6dc32e0d2583 Oct 30 14:36:49.423479 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:49.423431 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-318d85fc-22c2x" event={"ID":"b7a27530-b0f4-446b-91c3-cd597557e224","Type":"ContainerStarted","Data":"16998eb0ea67daf7a464871f494ed3dffc64f50884d77c06e41f350380c3583c"} Oct 30 14:36:49.423990 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:49.423544 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-318d85fc-22c2x" event={"ID":"b7a27530-b0f4-446b-91c3-cd597557e224","Type":"ContainerStarted","Data":"d0546e5eef897620cf3bbebf9c60a50e5d46b4c80823ed5f835e6dc32e0d2583"} Oct 30 14:36:49.470294 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:49.470246 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-318d85fc-22c2x" podStartSLOduration=2.470232794 podStartE2EDuration="2.470232794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:36:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:36:49.467955106 +0000 UTC m=+2879.212089147" watchObservedRunningTime="2026-10-30 14:36:49.470232794 +0000 UTC m=+2879.214366834" Oct 30 14:36:50.144218 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:50.144170 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:36:50.431916 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:50.431837 4982 generic.go:356] "Generic (PLEG): container finished" podID="b7a27530-b0f4-446b-91c3-cd597557e224" containerID="16998eb0ea67daf7a464871f494ed3dffc64f50884d77c06e41f350380c3583c" exitCode=0 Oct 30 14:36:50.432298 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:50.431921 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-318d85fc-22c2x" event={"ID":"b7a27530-b0f4-446b-91c3-cd597557e224","Type":"ContainerDied","Data":"16998eb0ea67daf7a464871f494ed3dffc64f50884d77c06e41f350380c3583c"} Oct 30 14:36:51.603832 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:51.603803 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-318d85fc-22c2x" Oct 30 14:36:51.680479 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:51.680449 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7a27530-b0f4-446b-91c3-cd597557e224-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7a27530-b0f4-446b-91c3-cd597557e224-maas-api-openshift-ca\") pod \"b7a27530-b0f4-446b-91c3-cd597557e224\" (UID: \"b7a27530-b0f4-446b-91c3-cd597557e224\") " Oct 30 14:36:51.680666 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:51.680492 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7a27530-b0f4-446b-91c3-cd597557e224-kube-api-access-78flp\" (UniqueName: \"kubernetes.io/projected/b7a27530-b0f4-446b-91c3-cd597557e224-kube-api-access-78flp\") pod \"b7a27530-b0f4-446b-91c3-cd597557e224\" (UID: \"b7a27530-b0f4-446b-91c3-cd597557e224\") " Oct 30 14:36:51.680666 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:51.680538 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7a27530-b0f4-446b-91c3-cd597557e224-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7a27530-b0f4-446b-91c3-cd597557e224-maas-api-xks-ca\") pod \"b7a27530-b0f4-446b-91c3-cd597557e224\" (UID: \"b7a27530-b0f4-446b-91c3-cd597557e224\") " Oct 30 14:36:51.681873 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:51.681842 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7a27530-b0f4-446b-91c3-cd597557e224-maas-api-openshift-ca" pod "b7a27530-b0f4-446b-91c3-cd597557e224" (UID: "b7a27530-b0f4-446b-91c3-cd597557e224"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:36:51.683958 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:51.683929 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7a27530-b0f4-446b-91c3-cd597557e224-maas-api-xks-ca" pod "b7a27530-b0f4-446b-91c3-cd597557e224" (UID: "b7a27530-b0f4-446b-91c3-cd597557e224"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:36:51.684675 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:51.684581 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7a27530-b0f4-446b-91c3-cd597557e224-kube-api-access-78flp" pod "b7a27530-b0f4-446b-91c3-cd597557e224" (UID: "b7a27530-b0f4-446b-91c3-cd597557e224"). InnerVolumeSpecName "kube-api-access-78flp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:36:51.760006 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:51.759979 4982 scope.go:122] "RemoveContainer" containerID="3cda605315726171249c02fb3573e294d7fed2d6afc8a03e58b8bc94bfbce9e1" Oct 30 14:36:51.770665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:51.770608 4982 scope.go:122] "RemoveContainer" containerID="8009de666b1b7c9c254d4317566cb4c88fed78e6847cfa6ec31dcd480c15b6aa" Oct 30 14:36:51.782020 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:51.781995 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7a27530-b0f4-446b-91c3-cd597557e224-maas-api-openshift-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:36:51.782113 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:51.782022 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78flp\" (UniqueName: \"kubernetes.io/projected/b7a27530-b0f4-446b-91c3-cd597557e224-kube-api-access-78flp\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:36:51.782113 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:51.782037 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7a27530-b0f4-446b-91c3-cd597557e224-maas-api-xks-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:36:52.040623 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:52.040545 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:36:52.366195 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:52.366155 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:36:52.512036 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:52.511995 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-318d85fc-22c2x" event={"ID":"b7a27530-b0f4-446b-91c3-cd597557e224","Type":"ContainerDied","Data":"d0546e5eef897620cf3bbebf9c60a50e5d46b4c80823ed5f835e6dc32e0d2583"} Oct 30 14:36:52.512144 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:52.512068 4982 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0546e5eef897620cf3bbebf9c60a50e5d46b4c80823ed5f835e6dc32e0d2583" Oct 30 14:36:52.512202 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:52.512130 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-318d85fc-22c2x" Oct 30 14:36:54.017942 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:54.017902 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:36:54.177762 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:54.177731 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:36:56.134746 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:56.134707 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:36:58.014784 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:36:58.014743 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:36:58.049436 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:58.049396 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:36:59.034929 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:59.034894 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd6b9bd9-xljkb"] Oct 30 14:36:59.047797 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:59.047767 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd6b9bd9-xljkb"] Oct 30 14:36:59.832147 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:36:59.832115 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:37:00.402166 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:00.402129 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64fe4a3b-0e15-47e6-a017-7f43db151a0d" path="/var/lib/kubelet/pods/64fe4a3b-0e15-47e6-a017-7f43db151a0d/volumes" Oct 30 14:37:01.862841 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:01.862807 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:37:03.847901 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:03.847865 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:37:04.002785 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:04.002753 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:37:04.466804 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:04.466770 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:37:05.051203 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:05.051091 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:37:05.205851 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:05.205807 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:37:05.704366 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:05.704325 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:37:07.642389 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:07.642355 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:37:09.703722 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:09.703684 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:37:11.461461 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:11.461431 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:37:12.368376 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:12.368331 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd12b3e9-qblsv"] Oct 30 14:37:12.368990 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:12.368975 4982 cpu_manager.go:414] "containerMap: removing container" podUID="b7a27530-b0f4-446b-91c3-cd597557e224" containerName="revoke-keys" Oct 30 14:37:12.369049 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:12.368991 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7a27530-b0f4-446b-91c3-cd597557e224" containerName="revoke-keys" Oct 30 14:37:12.369097 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:12.369088 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7a27530-b0f4-446b-91c3-cd597557e224" containerName="revoke-keys" Oct 30 14:37:12.374370 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:12.374350 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd12b3e9-qblsv" Oct 30 14:37:12.402336 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:12.402314 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:37:12.402615 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:12.402599 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:37:12.408763 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:12.408735 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd12b3e9-qblsv"] Oct 30 14:37:12.412502 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:12.412469 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkt7t\" (UniqueName: \"kubernetes.io/projected/84bd46f0-7339-4f38-83cb-d728845bda61-kube-api-access-zkt7t\") pod \"maas-api-revoke-sub-bd12b3e9-qblsv\" (UID: \"84bd46f0-7339-4f38-83cb-d728845bda61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd12b3e9-qblsv" Oct 30 14:37:12.412625 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:12.412608 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84bd46f0-7339-4f38-83cb-d728845bda61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd12b3e9-qblsv\" (UID: \"84bd46f0-7339-4f38-83cb-d728845bda61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd12b3e9-qblsv" Oct 30 14:37:12.412705 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:12.412657 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84bd46f0-7339-4f38-83cb-d728845bda61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd12b3e9-qblsv\" (UID: \"84bd46f0-7339-4f38-83cb-d728845bda61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd12b3e9-qblsv" Oct 30 14:37:12.514220 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:12.514190 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkt7t\" (UniqueName: \"kubernetes.io/projected/84bd46f0-7339-4f38-83cb-d728845bda61-kube-api-access-zkt7t\") pod \"maas-api-revoke-sub-bd12b3e9-qblsv\" (UID: \"84bd46f0-7339-4f38-83cb-d728845bda61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd12b3e9-qblsv" Oct 30 14:37:12.514716 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:12.514298 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84bd46f0-7339-4f38-83cb-d728845bda61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd12b3e9-qblsv\" (UID: \"84bd46f0-7339-4f38-83cb-d728845bda61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd12b3e9-qblsv" Oct 30 14:37:12.514716 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:12.514319 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84bd46f0-7339-4f38-83cb-d728845bda61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd12b3e9-qblsv\" (UID: \"84bd46f0-7339-4f38-83cb-d728845bda61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd12b3e9-qblsv" Oct 30 14:37:12.515190 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:12.515163 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84bd46f0-7339-4f38-83cb-d728845bda61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd12b3e9-qblsv\" (UID: \"84bd46f0-7339-4f38-83cb-d728845bda61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd12b3e9-qblsv" Oct 30 14:37:12.516970 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:12.516943 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84bd46f0-7339-4f38-83cb-d728845bda61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd12b3e9-qblsv\" (UID: \"84bd46f0-7339-4f38-83cb-d728845bda61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd12b3e9-qblsv" Oct 30 14:37:12.539921 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:12.539894 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkt7t\" (UniqueName: \"kubernetes.io/projected/84bd46f0-7339-4f38-83cb-d728845bda61-kube-api-access-zkt7t\") pod \"maas-api-revoke-sub-bd12b3e9-qblsv\" (UID: \"84bd46f0-7339-4f38-83cb-d728845bda61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd12b3e9-qblsv" Oct 30 14:37:12.690133 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:12.690063 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd12b3e9-qblsv" Oct 30 14:37:13.011984 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:13.011954 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd12b3e9-qblsv"] Oct 30 14:37:13.012289 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:37:13.012255 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84bd46f0_7339_4f38_83cb_d728845bda61.slice/crio-e77bdb1d87e0da61beae4f7c9107f8998e7b10a67e545378ab6ef825b40fb7ef WatchSource:0}: Error finding container e77bdb1d87e0da61beae4f7c9107f8998e7b10a67e545378ab6ef825b40fb7ef: Status 404 returned error can't find the container with id e77bdb1d87e0da61beae4f7c9107f8998e7b10a67e545378ab6ef825b40fb7ef Oct 30 14:37:13.471403 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:13.471371 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:37:13.746853 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:13.746773 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd12b3e9-qblsv" event={"ID":"84bd46f0-7339-4f38-83cb-d728845bda61","Type":"ContainerStarted","Data":"02ce3148e67b35adf05397ec7c2328594b68197f5ffd1da04a50980432d5e59a"} Oct 30 14:37:13.746853 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:13.746825 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd12b3e9-qblsv" event={"ID":"84bd46f0-7339-4f38-83cb-d728845bda61","Type":"ContainerStarted","Data":"e77bdb1d87e0da61beae4f7c9107f8998e7b10a67e545378ab6ef825b40fb7ef"} Oct 30 14:37:13.793144 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:13.793094 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd12b3e9-qblsv" podStartSLOduration=1.793080035 podStartE2EDuration="1.793080035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:37:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:37:13.789818476 +0000 UTC m=+2903.533952519" watchObservedRunningTime="2026-10-30 14:37:13.793080035 +0000 UTC m=+2903.537214075" Oct 30 14:37:14.755866 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:14.755834 4982 generic.go:356] "Generic (PLEG): container finished" podID="84bd46f0-7339-4f38-83cb-d728845bda61" containerID="02ce3148e67b35adf05397ec7c2328594b68197f5ffd1da04a50980432d5e59a" exitCode=0 Oct 30 14:37:14.756251 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:14.755911 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd12b3e9-qblsv" event={"ID":"84bd46f0-7339-4f38-83cb-d728845bda61","Type":"ContainerDied","Data":"02ce3148e67b35adf05397ec7c2328594b68197f5ffd1da04a50980432d5e59a"} Oct 30 14:37:15.330574 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:15.330542 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:37:15.937235 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:15.937211 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd12b3e9-qblsv" Oct 30 14:37:16.063298 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:16.063267 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84bd46f0-7339-4f38-83cb-d728845bda61-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84bd46f0-7339-4f38-83cb-d728845bda61-maas-api-xks-ca\") pod \"84bd46f0-7339-4f38-83cb-d728845bda61\" (UID: \"84bd46f0-7339-4f38-83cb-d728845bda61\") " Oct 30 14:37:16.063478 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:16.063307 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84bd46f0-7339-4f38-83cb-d728845bda61-kube-api-access-zkt7t\" (UniqueName: \"kubernetes.io/projected/84bd46f0-7339-4f38-83cb-d728845bda61-kube-api-access-zkt7t\") pod \"84bd46f0-7339-4f38-83cb-d728845bda61\" (UID: \"84bd46f0-7339-4f38-83cb-d728845bda61\") " Oct 30 14:37:16.063478 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:16.063434 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84bd46f0-7339-4f38-83cb-d728845bda61-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84bd46f0-7339-4f38-83cb-d728845bda61-maas-api-openshift-ca\") pod \"84bd46f0-7339-4f38-83cb-d728845bda61\" (UID: \"84bd46f0-7339-4f38-83cb-d728845bda61\") " Oct 30 14:37:16.063797 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:16.063774 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84bd46f0-7339-4f38-83cb-d728845bda61-maas-api-openshift-ca" pod "84bd46f0-7339-4f38-83cb-d728845bda61" (UID: "84bd46f0-7339-4f38-83cb-d728845bda61"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:37:16.064183 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:16.064156 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84bd46f0-7339-4f38-83cb-d728845bda61-maas-api-openshift-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:37:16.065449 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:16.065419 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84bd46f0-7339-4f38-83cb-d728845bda61-maas-api-xks-ca" pod "84bd46f0-7339-4f38-83cb-d728845bda61" (UID: "84bd46f0-7339-4f38-83cb-d728845bda61"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:37:16.065531 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:16.065501 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84bd46f0-7339-4f38-83cb-d728845bda61-kube-api-access-zkt7t" pod "84bd46f0-7339-4f38-83cb-d728845bda61" (UID: "84bd46f0-7339-4f38-83cb-d728845bda61"). InnerVolumeSpecName "kube-api-access-zkt7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:37:16.165127 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:16.165099 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84bd46f0-7339-4f38-83cb-d728845bda61-maas-api-xks-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:37:16.165127 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:16.165126 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkt7t\" (UniqueName: \"kubernetes.io/projected/84bd46f0-7339-4f38-83cb-d728845bda61-kube-api-access-zkt7t\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:37:16.773490 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:16.773413 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd12b3e9-qblsv" Oct 30 14:37:16.773490 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:16.773450 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd12b3e9-qblsv" event={"ID":"84bd46f0-7339-4f38-83cb-d728845bda61","Type":"ContainerDied","Data":"e77bdb1d87e0da61beae4f7c9107f8998e7b10a67e545378ab6ef825b40fb7ef"} Oct 30 14:37:16.773799 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:16.773500 4982 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e77bdb1d87e0da61beae4f7c9107f8998e7b10a67e545378ab6ef825b40fb7ef" Oct 30 14:37:17.160212 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:17.160176 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:37:18.704198 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:18.704163 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:37:18.860234 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:18.860199 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:37:19.362073 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:19.362030 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:37:19.630009 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:19.629927 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:37:20.721363 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:20.721325 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:37:21.863126 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:21.863055 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:37:22.317387 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:22.317350 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:37:22.489036 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:22.489000 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:37:22.720734 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:22.719781 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c152ab3d-wpzgh"] Oct 30 14:37:22.720914 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:22.720790 4982 cpu_manager.go:414] "containerMap: removing container" podUID="84bd46f0-7339-4f38-83cb-d728845bda61" containerName="revoke-keys" Oct 30 14:37:22.720914 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:22.720813 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84bd46f0-7339-4f38-83cb-d728845bda61" containerName="revoke-keys" Oct 30 14:37:22.721043 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:22.720964 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="84bd46f0-7339-4f38-83cb-d728845bda61" containerName="revoke-keys" Oct 30 14:37:22.727110 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:22.727084 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c152ab3d-wpzgh" Oct 30 14:37:22.740331 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:22.740305 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c152ab3d-wpzgh"] Oct 30 14:37:22.745880 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:22.745859 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:37:22.746154 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:22.746138 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:37:22.802039 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:22.802006 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3225123b-b2e3-4ba0-a1b7-f60d2509cd97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c152ab3d-wpzgh\" (UID: \"3225123b-b2e3-4ba0-a1b7-f60d2509cd97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c152ab3d-wpzgh" Oct 30 14:37:22.802204 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:22.802079 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjtnv\" (UniqueName: \"kubernetes.io/projected/3225123b-b2e3-4ba0-a1b7-f60d2509cd97-kube-api-access-fjtnv\") pod \"maas-api-revoke-sub-c152ab3d-wpzgh\" (UID: \"3225123b-b2e3-4ba0-a1b7-f60d2509cd97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c152ab3d-wpzgh" Oct 30 14:37:22.802304 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:22.802280 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3225123b-b2e3-4ba0-a1b7-f60d2509cd97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c152ab3d-wpzgh\" (UID: \"3225123b-b2e3-4ba0-a1b7-f60d2509cd97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c152ab3d-wpzgh" Oct 30 14:37:22.905534 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:22.905473 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3225123b-b2e3-4ba0-a1b7-f60d2509cd97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c152ab3d-wpzgh\" (UID: \"3225123b-b2e3-4ba0-a1b7-f60d2509cd97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c152ab3d-wpzgh" Oct 30 14:37:22.906155 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:22.905656 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3225123b-b2e3-4ba0-a1b7-f60d2509cd97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c152ab3d-wpzgh\" (UID: \"3225123b-b2e3-4ba0-a1b7-f60d2509cd97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c152ab3d-wpzgh" Oct 30 14:37:22.906155 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:22.905709 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjtnv\" (UniqueName: \"kubernetes.io/projected/3225123b-b2e3-4ba0-a1b7-f60d2509cd97-kube-api-access-fjtnv\") pod \"maas-api-revoke-sub-c152ab3d-wpzgh\" (UID: \"3225123b-b2e3-4ba0-a1b7-f60d2509cd97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c152ab3d-wpzgh" Oct 30 14:37:22.906450 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:22.906368 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3225123b-b2e3-4ba0-a1b7-f60d2509cd97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c152ab3d-wpzgh\" (UID: \"3225123b-b2e3-4ba0-a1b7-f60d2509cd97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c152ab3d-wpzgh" Oct 30 14:37:22.912400 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:22.912371 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3225123b-b2e3-4ba0-a1b7-f60d2509cd97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c152ab3d-wpzgh\" (UID: \"3225123b-b2e3-4ba0-a1b7-f60d2509cd97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c152ab3d-wpzgh" Oct 30 14:37:22.933249 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:22.933218 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjtnv\" (UniqueName: \"kubernetes.io/projected/3225123b-b2e3-4ba0-a1b7-f60d2509cd97-kube-api-access-fjtnv\") pod \"maas-api-revoke-sub-c152ab3d-wpzgh\" (UID: \"3225123b-b2e3-4ba0-a1b7-f60d2509cd97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c152ab3d-wpzgh" Oct 30 14:37:23.053740 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:23.053669 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c152ab3d-wpzgh" Oct 30 14:37:23.364073 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:23.364040 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c152ab3d-wpzgh"] Oct 30 14:37:23.364823 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:37:23.364796 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3225123b_b2e3_4ba0_a1b7_f60d2509cd97.slice/crio-195a4841c55712b6acc9bc087ab8bcf542eb3db32abff47e810ab34f3af9e141 WatchSource:0}: Error finding container 195a4841c55712b6acc9bc087ab8bcf542eb3db32abff47e810ab34f3af9e141: Status 404 returned error can't find the container with id 195a4841c55712b6acc9bc087ab8bcf542eb3db32abff47e810ab34f3af9e141 Oct 30 14:37:23.833627 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:23.833590 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c152ab3d-wpzgh" event={"ID":"3225123b-b2e3-4ba0-a1b7-f60d2509cd97","Type":"ContainerStarted","Data":"36b695902601745a61bea495b17f69f85e7a71b748a9e26ca07e48bb8ffdc134"} Oct 30 14:37:23.833791 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:23.833674 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c152ab3d-wpzgh" event={"ID":"3225123b-b2e3-4ba0-a1b7-f60d2509cd97","Type":"ContainerStarted","Data":"195a4841c55712b6acc9bc087ab8bcf542eb3db32abff47e810ab34f3af9e141"} Oct 30 14:37:23.872991 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:23.872962 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:37:23.878562 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:23.878521 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c152ab3d-wpzgh" podStartSLOduration=1.878508689 podStartE2EDuration="1.878508689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:37:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:37:23.876160591 +0000 UTC m=+2913.620294632" watchObservedRunningTime="2026-10-30 14:37:23.878508689 +0000 UTC m=+2913.622642729" Oct 30 14:37:24.842172 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:24.842139 4982 generic.go:356] "Generic (PLEG): container finished" podID="3225123b-b2e3-4ba0-a1b7-f60d2509cd97" containerID="36b695902601745a61bea495b17f69f85e7a71b748a9e26ca07e48bb8ffdc134" exitCode=0 Oct 30 14:37:24.842572 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:24.842224 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c152ab3d-wpzgh" event={"ID":"3225123b-b2e3-4ba0-a1b7-f60d2509cd97","Type":"ContainerDied","Data":"36b695902601745a61bea495b17f69f85e7a71b748a9e26ca07e48bb8ffdc134"} Oct 30 14:37:25.783891 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:25.783854 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:37:26.017426 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:26.017397 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c152ab3d-wpzgh" Oct 30 14:37:26.047515 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:26.047447 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3225123b-b2e3-4ba0-a1b7-f60d2509cd97-kube-api-access-fjtnv\" (UniqueName: \"kubernetes.io/projected/3225123b-b2e3-4ba0-a1b7-f60d2509cd97-kube-api-access-fjtnv\") pod \"3225123b-b2e3-4ba0-a1b7-f60d2509cd97\" (UID: \"3225123b-b2e3-4ba0-a1b7-f60d2509cd97\") " Oct 30 14:37:26.047678 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:26.047623 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3225123b-b2e3-4ba0-a1b7-f60d2509cd97-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3225123b-b2e3-4ba0-a1b7-f60d2509cd97-maas-api-xks-ca\") pod \"3225123b-b2e3-4ba0-a1b7-f60d2509cd97\" (UID: \"3225123b-b2e3-4ba0-a1b7-f60d2509cd97\") " Oct 30 14:37:26.047765 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:26.047746 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3225123b-b2e3-4ba0-a1b7-f60d2509cd97-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3225123b-b2e3-4ba0-a1b7-f60d2509cd97-maas-api-openshift-ca\") pod \"3225123b-b2e3-4ba0-a1b7-f60d2509cd97\" (UID: \"3225123b-b2e3-4ba0-a1b7-f60d2509cd97\") " Oct 30 14:37:26.048141 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:26.048114 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3225123b-b2e3-4ba0-a1b7-f60d2509cd97-maas-api-openshift-ca" pod "3225123b-b2e3-4ba0-a1b7-f60d2509cd97" (UID: "3225123b-b2e3-4ba0-a1b7-f60d2509cd97"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:37:26.048908 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:26.048874 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3225123b-b2e3-4ba0-a1b7-f60d2509cd97-maas-api-openshift-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:37:26.049829 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:26.049801 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3225123b-b2e3-4ba0-a1b7-f60d2509cd97-kube-api-access-fjtnv" pod "3225123b-b2e3-4ba0-a1b7-f60d2509cd97" (UID: "3225123b-b2e3-4ba0-a1b7-f60d2509cd97"). InnerVolumeSpecName "kube-api-access-fjtnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:37:26.050120 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:26.050096 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3225123b-b2e3-4ba0-a1b7-f60d2509cd97-maas-api-xks-ca" pod "3225123b-b2e3-4ba0-a1b7-f60d2509cd97" (UID: "3225123b-b2e3-4ba0-a1b7-f60d2509cd97"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:37:26.150375 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:26.150339 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjtnv\" (UniqueName: \"kubernetes.io/projected/3225123b-b2e3-4ba0-a1b7-f60d2509cd97-kube-api-access-fjtnv\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:37:26.150375 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:26.150372 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3225123b-b2e3-4ba0-a1b7-f60d2509cd97-maas-api-xks-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:37:26.860383 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:26.860350 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c152ab3d-wpzgh" Oct 30 14:37:26.860552 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:26.860381 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c152ab3d-wpzgh" event={"ID":"3225123b-b2e3-4ba0-a1b7-f60d2509cd97","Type":"ContainerDied","Data":"195a4841c55712b6acc9bc087ab8bcf542eb3db32abff47e810ab34f3af9e141"} Oct 30 14:37:26.860552 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:26.860430 4982 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="195a4841c55712b6acc9bc087ab8bcf542eb3db32abff47e810ab34f3af9e141" Oct 30 14:37:28.024445 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:28.024365 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:37:28.050709 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:37:28.050663 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:37:30.217051 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:30.217018 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:37:31.515172 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:31.515136 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:37:32.356615 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:32.356568 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:37:33.601329 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:33.601244 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6c013df-vvrs6"] Oct 30 14:37:33.601875 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:33.601847 4982 cpu_manager.go:414] "containerMap: removing container" podUID="3225123b-b2e3-4ba0-a1b7-f60d2509cd97" containerName="revoke-keys" Oct 30 14:37:33.601875 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:33.601861 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3225123b-b2e3-4ba0-a1b7-f60d2509cd97" containerName="revoke-keys" Oct 30 14:37:33.601999 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:33.601995 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="3225123b-b2e3-4ba0-a1b7-f60d2509cd97" containerName="revoke-keys" Oct 30 14:37:33.606941 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:33.606918 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6c013df-vvrs6" Oct 30 14:37:33.626248 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:33.626212 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6c013df-vvrs6"] Oct 30 14:37:33.634745 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:33.634720 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:37:33.635042 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:33.635021 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:37:33.652988 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:33.652957 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f307466f-7382-4469-b6cd-b19e87a6fa81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6c013df-vvrs6\" (UID: \"f307466f-7382-4469-b6cd-b19e87a6fa81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6c013df-vvrs6" Oct 30 14:37:33.653087 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:33.653010 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kssz7\" (UniqueName: \"kubernetes.io/projected/f307466f-7382-4469-b6cd-b19e87a6fa81-kube-api-access-kssz7\") pod \"maas-api-revoke-sub-f6c013df-vvrs6\" (UID: \"f307466f-7382-4469-b6cd-b19e87a6fa81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6c013df-vvrs6" Oct 30 14:37:33.653182 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:33.653160 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f307466f-7382-4469-b6cd-b19e87a6fa81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6c013df-vvrs6\" (UID: \"f307466f-7382-4469-b6cd-b19e87a6fa81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6c013df-vvrs6" Oct 30 14:37:33.754717 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:33.754675 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f307466f-7382-4469-b6cd-b19e87a6fa81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6c013df-vvrs6\" (UID: \"f307466f-7382-4469-b6cd-b19e87a6fa81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6c013df-vvrs6" Oct 30 14:37:33.754887 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:33.754841 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f307466f-7382-4469-b6cd-b19e87a6fa81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6c013df-vvrs6\" (UID: \"f307466f-7382-4469-b6cd-b19e87a6fa81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6c013df-vvrs6" Oct 30 14:37:33.754887 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:33.754873 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kssz7\" (UniqueName: \"kubernetes.io/projected/f307466f-7382-4469-b6cd-b19e87a6fa81-kube-api-access-kssz7\") pod \"maas-api-revoke-sub-f6c013df-vvrs6\" (UID: \"f307466f-7382-4469-b6cd-b19e87a6fa81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6c013df-vvrs6" Oct 30 14:37:33.755377 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:33.755349 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f307466f-7382-4469-b6cd-b19e87a6fa81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6c013df-vvrs6\" (UID: \"f307466f-7382-4469-b6cd-b19e87a6fa81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6c013df-vvrs6" Oct 30 14:37:33.757188 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:33.757163 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f307466f-7382-4469-b6cd-b19e87a6fa81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6c013df-vvrs6\" (UID: \"f307466f-7382-4469-b6cd-b19e87a6fa81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6c013df-vvrs6" Oct 30 14:37:33.780324 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:33.780300 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kssz7\" (UniqueName: \"kubernetes.io/projected/f307466f-7382-4469-b6cd-b19e87a6fa81-kube-api-access-kssz7\") pod \"maas-api-revoke-sub-f6c013df-vvrs6\" (UID: \"f307466f-7382-4469-b6cd-b19e87a6fa81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6c013df-vvrs6" Oct 30 14:37:33.921347 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:33.921263 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6c013df-vvrs6" Oct 30 14:37:34.232662 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:34.232619 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6c013df-vvrs6"] Oct 30 14:37:34.235433 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:37:34.235406 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf307466f_7382_4469_b6cd_b19e87a6fa81.slice/crio-60c101abd3bfd102e05232da846c187ddc6f0dcafe27092e8bda597e5c4f34e3 WatchSource:0}: Error finding container 60c101abd3bfd102e05232da846c187ddc6f0dcafe27092e8bda597e5c4f34e3: Status 404 returned error can't find the container with id 60c101abd3bfd102e05232da846c187ddc6f0dcafe27092e8bda597e5c4f34e3 Oct 30 14:37:34.408119 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:34.408085 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:37:34.928290 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:34.928253 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6c013df-vvrs6" event={"ID":"f307466f-7382-4469-b6cd-b19e87a6fa81","Type":"ContainerStarted","Data":"c1d25184c03263c6bfd74313f78bfd5f540f44884faea71a0955dfeb9e01651e"} Oct 30 14:37:34.928770 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:34.928311 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6c013df-vvrs6" event={"ID":"f307466f-7382-4469-b6cd-b19e87a6fa81","Type":"ContainerStarted","Data":"60c101abd3bfd102e05232da846c187ddc6f0dcafe27092e8bda597e5c4f34e3"} Oct 30 14:37:34.972923 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:34.971970 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6c013df-vvrs6" podStartSLOduration=1.971956829 podStartE2EDuration="1.971956829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:37:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:37:34.971185686 +0000 UTC m=+2924.715319729" watchObservedRunningTime="2026-10-30 14:37:34.971956829 +0000 UTC m=+2924.716090870" Oct 30 14:37:35.022196 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:35.022165 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d3a829-698854d99d-8f69p"] Oct 30 14:37:35.054213 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:35.054186 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d3a829-6fcfc75547-xlcmh"] Oct 30 14:37:35.187245 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:35.187211 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:37:35.337164 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:35.337133 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:37:35.482035 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:35.481959 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:37:35.937853 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:35.937816 4982 generic.go:356] "Generic (PLEG): container finished" podID="f307466f-7382-4469-b6cd-b19e87a6fa81" containerID="c1d25184c03263c6bfd74313f78bfd5f540f44884faea71a0955dfeb9e01651e" exitCode=0 Oct 30 14:37:35.938235 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:35.937928 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6c013df-vvrs6" event={"ID":"f307466f-7382-4469-b6cd-b19e87a6fa81","Type":"ContainerDied","Data":"c1d25184c03263c6bfd74313f78bfd5f540f44884faea71a0955dfeb9e01651e"} Oct 30 14:37:36.306931 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:36.306901 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:37:37.144358 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:37.144319 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6c013df-vvrs6" Oct 30 14:37:37.202382 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:37.202301 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f307466f-7382-4469-b6cd-b19e87a6fa81-kube-api-access-kssz7\" (UniqueName: \"kubernetes.io/projected/f307466f-7382-4469-b6cd-b19e87a6fa81-kube-api-access-kssz7\") pod \"f307466f-7382-4469-b6cd-b19e87a6fa81\" (UID: \"f307466f-7382-4469-b6cd-b19e87a6fa81\") " Oct 30 14:37:37.202561 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:37.202476 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f307466f-7382-4469-b6cd-b19e87a6fa81-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f307466f-7382-4469-b6cd-b19e87a6fa81-maas-api-xks-ca\") pod \"f307466f-7382-4469-b6cd-b19e87a6fa81\" (UID: \"f307466f-7382-4469-b6cd-b19e87a6fa81\") " Oct 30 14:37:37.202561 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:37.202543 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f307466f-7382-4469-b6cd-b19e87a6fa81-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f307466f-7382-4469-b6cd-b19e87a6fa81-maas-api-openshift-ca\") pod \"f307466f-7382-4469-b6cd-b19e87a6fa81\" (UID: \"f307466f-7382-4469-b6cd-b19e87a6fa81\") " Oct 30 14:37:37.203139 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:37.203112 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f307466f-7382-4469-b6cd-b19e87a6fa81-maas-api-openshift-ca" pod "f307466f-7382-4469-b6cd-b19e87a6fa81" (UID: "f307466f-7382-4469-b6cd-b19e87a6fa81"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:37:37.218659 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:37.204921 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f307466f-7382-4469-b6cd-b19e87a6fa81-kube-api-access-kssz7" pod "f307466f-7382-4469-b6cd-b19e87a6fa81" (UID: "f307466f-7382-4469-b6cd-b19e87a6fa81"). InnerVolumeSpecName "kube-api-access-kssz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:37:37.218659 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:37.205149 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f307466f-7382-4469-b6cd-b19e87a6fa81-maas-api-xks-ca" pod "f307466f-7382-4469-b6cd-b19e87a6fa81" (UID: "f307466f-7382-4469-b6cd-b19e87a6fa81"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:37:37.304560 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:37.304514 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f307466f-7382-4469-b6cd-b19e87a6fa81-maas-api-xks-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:37:37.304560 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:37.304554 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f307466f-7382-4469-b6cd-b19e87a6fa81-maas-api-openshift-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:37:37.304560 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:37.304567 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kssz7\" (UniqueName: \"kubernetes.io/projected/f307466f-7382-4469-b6cd-b19e87a6fa81-kube-api-access-kssz7\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:37:37.955401 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:37.955373 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6c013df-vvrs6" Oct 30 14:37:37.955401 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:37.955396 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6c013df-vvrs6" event={"ID":"f307466f-7382-4469-b6cd-b19e87a6fa81","Type":"ContainerDied","Data":"60c101abd3bfd102e05232da846c187ddc6f0dcafe27092e8bda597e5c4f34e3"} Oct 30 14:37:37.955618 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:37.955451 4982 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60c101abd3bfd102e05232da846c187ddc6f0dcafe27092e8bda597e5c4f34e3" Oct 30 14:37:38.258698 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:38.258595 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:37:40.023246 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:40.023181 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-d3a829-698854d99d-8f69p" podUID="f4ee9039-363a-4761-8b74-dd5ff95a4949" containerName="payload-processing" containerID="cri-o://e4701b29940ff327330146ff3e51c8d839a7a734761f812ce0b23d6747e0a2f4" gracePeriod=25 Oct 30 14:37:40.055201 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:40.055166 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d3a829-6fcfc75547-xlcmh" podUID="a566e97f-4f8a-499c-9630-7e5de078910a" containerName="payload-pre-processing" containerID="cri-o://f307e6f67389ed1e8446f2a983167c44be53f68fe1609c2d9bc38d90ce788d7e" gracePeriod=25 Oct 30 14:37:40.155005 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:40.154944 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:37:40.363130 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:40.363101 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d3a829-698854d99d-8f69p" Oct 30 14:37:40.397216 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:40.397190 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d3a829-6fcfc75547-xlcmh" Oct 30 14:37:40.449739 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:40.449704 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4ee9039-363a-4761-8b74-dd5ff95a4949-kube-api-access-tnn86\" (UniqueName: \"kubernetes.io/projected/f4ee9039-363a-4761-8b74-dd5ff95a4949-kube-api-access-tnn86\") pod \"f4ee9039-363a-4761-8b74-dd5ff95a4949\" (UID: \"f4ee9039-363a-4761-8b74-dd5ff95a4949\") " Oct 30 14:37:40.449920 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:40.449903 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4ee9039-363a-4761-8b74-dd5ff95a4949-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4ee9039-363a-4761-8b74-dd5ff95a4949-plugins-config-volume\") pod \"f4ee9039-363a-4761-8b74-dd5ff95a4949\" (UID: \"f4ee9039-363a-4761-8b74-dd5ff95a4949\") " Oct 30 14:37:40.450342 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:40.450311 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4ee9039-363a-4761-8b74-dd5ff95a4949-plugins-config-volume" pod "f4ee9039-363a-4761-8b74-dd5ff95a4949" (UID: "f4ee9039-363a-4761-8b74-dd5ff95a4949"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:37:40.450452 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:40.450426 4982 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4ee9039-363a-4761-8b74-dd5ff95a4949-plugins-config-volume\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:37:40.456161 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:40.456132 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4ee9039-363a-4761-8b74-dd5ff95a4949-kube-api-access-tnn86" pod "f4ee9039-363a-4761-8b74-dd5ff95a4949" (UID: "f4ee9039-363a-4761-8b74-dd5ff95a4949"). InnerVolumeSpecName "kube-api-access-tnn86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:37:40.551553 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:40.551485 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a566e97f-4f8a-499c-9630-7e5de078910a-kube-api-access-wvbz9\" (UniqueName: \"kubernetes.io/projected/a566e97f-4f8a-499c-9630-7e5de078910a-kube-api-access-wvbz9\") pod \"a566e97f-4f8a-499c-9630-7e5de078910a\" (UID: \"a566e97f-4f8a-499c-9630-7e5de078910a\") " Oct 30 14:37:40.551553 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:40.551560 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a566e97f-4f8a-499c-9630-7e5de078910a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a566e97f-4f8a-499c-9630-7e5de078910a-plugins-config-volume\") pod \"a566e97f-4f8a-499c-9630-7e5de078910a\" (UID: \"a566e97f-4f8a-499c-9630-7e5de078910a\") " Oct 30 14:37:40.551961 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:40.551935 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a566e97f-4f8a-499c-9630-7e5de078910a-plugins-config-volume" pod "a566e97f-4f8a-499c-9630-7e5de078910a" (UID: "a566e97f-4f8a-499c-9630-7e5de078910a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:37:40.552050 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:40.551963 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnn86\" (UniqueName: \"kubernetes.io/projected/f4ee9039-363a-4761-8b74-dd5ff95a4949-kube-api-access-tnn86\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:37:40.553668 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:40.553630 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a566e97f-4f8a-499c-9630-7e5de078910a-kube-api-access-wvbz9" pod "a566e97f-4f8a-499c-9630-7e5de078910a" (UID: "a566e97f-4f8a-499c-9630-7e5de078910a"). InnerVolumeSpecName "kube-api-access-wvbz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:37:40.653431 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:40.653402 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvbz9\" (UniqueName: \"kubernetes.io/projected/a566e97f-4f8a-499c-9630-7e5de078910a-kube-api-access-wvbz9\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:37:40.653431 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:40.653429 4982 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a566e97f-4f8a-499c-9630-7e5de078910a-plugins-config-volume\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:37:40.980101 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:40.980066 4982 generic.go:356] "Generic (PLEG): container finished" podID="a566e97f-4f8a-499c-9630-7e5de078910a" containerID="f307e6f67389ed1e8446f2a983167c44be53f68fe1609c2d9bc38d90ce788d7e" exitCode=0 Oct 30 14:37:40.980308 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:40.980122 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d3a829-6fcfc75547-xlcmh" Oct 30 14:37:40.980308 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:40.980152 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d3a829-6fcfc75547-xlcmh" event={"ID":"a566e97f-4f8a-499c-9630-7e5de078910a","Type":"ContainerDied","Data":"f307e6f67389ed1e8446f2a983167c44be53f68fe1609c2d9bc38d90ce788d7e"} Oct 30 14:37:40.980308 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:40.980203 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d3a829-6fcfc75547-xlcmh" event={"ID":"a566e97f-4f8a-499c-9630-7e5de078910a","Type":"ContainerDied","Data":"e9e2271f31cbad9f2020e2a208c874d5072523c601409c9f6117e7052b982b55"} Oct 30 14:37:40.980308 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:40.980226 4982 scope.go:122] "RemoveContainer" containerID="f307e6f67389ed1e8446f2a983167c44be53f68fe1609c2d9bc38d90ce788d7e" Oct 30 14:37:40.982093 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:40.982066 4982 generic.go:356] "Generic (PLEG): container finished" podID="f4ee9039-363a-4761-8b74-dd5ff95a4949" containerID="e4701b29940ff327330146ff3e51c8d839a7a734761f812ce0b23d6747e0a2f4" exitCode=0 Oct 30 14:37:40.982245 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:40.982101 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d3a829-698854d99d-8f69p" Oct 30 14:37:40.982245 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:40.982179 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d3a829-698854d99d-8f69p" event={"ID":"f4ee9039-363a-4761-8b74-dd5ff95a4949","Type":"ContainerDied","Data":"e4701b29940ff327330146ff3e51c8d839a7a734761f812ce0b23d6747e0a2f4"} Oct 30 14:37:40.982245 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:40.982222 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d3a829-698854d99d-8f69p" event={"ID":"f4ee9039-363a-4761-8b74-dd5ff95a4949","Type":"ContainerDied","Data":"48bf60565d7a804e3a13eced665cf9ff5ab799397f34785319e2d14ae60febd8"} Oct 30 14:37:40.993316 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:40.993299 4982 scope.go:122] "RemoveContainer" containerID="f307e6f67389ed1e8446f2a983167c44be53f68fe1609c2d9bc38d90ce788d7e" Oct 30 14:37:40.993594 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:37:40.993575 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f307e6f67389ed1e8446f2a983167c44be53f68fe1609c2d9bc38d90ce788d7e\": container with ID starting with f307e6f67389ed1e8446f2a983167c44be53f68fe1609c2d9bc38d90ce788d7e not found: ID does not exist" containerID="f307e6f67389ed1e8446f2a983167c44be53f68fe1609c2d9bc38d90ce788d7e" Oct 30 14:37:40.993697 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:40.993600 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f307e6f67389ed1e8446f2a983167c44be53f68fe1609c2d9bc38d90ce788d7e"} err="failed to get container status \"f307e6f67389ed1e8446f2a983167c44be53f68fe1609c2d9bc38d90ce788d7e\": rpc error: code = NotFound desc = could not find container \"f307e6f67389ed1e8446f2a983167c44be53f68fe1609c2d9bc38d90ce788d7e\": container with ID starting with f307e6f67389ed1e8446f2a983167c44be53f68fe1609c2d9bc38d90ce788d7e not found: ID does not exist" Oct 30 14:37:40.993697 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:40.993615 4982 scope.go:122] "RemoveContainer" containerID="e4701b29940ff327330146ff3e51c8d839a7a734761f812ce0b23d6747e0a2f4" Oct 30 14:37:41.006582 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:41.006559 4982 scope.go:122] "RemoveContainer" containerID="e4701b29940ff327330146ff3e51c8d839a7a734761f812ce0b23d6747e0a2f4" Oct 30 14:37:41.007049 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:37:41.006990 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4701b29940ff327330146ff3e51c8d839a7a734761f812ce0b23d6747e0a2f4\": container with ID starting with e4701b29940ff327330146ff3e51c8d839a7a734761f812ce0b23d6747e0a2f4 not found: ID does not exist" containerID="e4701b29940ff327330146ff3e51c8d839a7a734761f812ce0b23d6747e0a2f4" Oct 30 14:37:41.007049 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:41.007025 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4701b29940ff327330146ff3e51c8d839a7a734761f812ce0b23d6747e0a2f4"} err="failed to get container status \"e4701b29940ff327330146ff3e51c8d839a7a734761f812ce0b23d6747e0a2f4\": rpc error: code = NotFound desc = could not find container \"e4701b29940ff327330146ff3e51c8d839a7a734761f812ce0b23d6747e0a2f4\": container with ID starting with e4701b29940ff327330146ff3e51c8d839a7a734761f812ce0b23d6747e0a2f4 not found: ID does not exist" Oct 30 14:37:41.042000 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:41.041958 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d3a829-6fcfc75547-xlcmh"] Oct 30 14:37:41.051068 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:41.051044 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d3a829-6fcfc75547-xlcmh"] Oct 30 14:37:41.088909 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:41.088874 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d3a829-698854d99d-8f69p"] Oct 30 14:37:41.097580 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:41.097559 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d3a829-698854d99d-8f69p"] Oct 30 14:37:41.952376 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:41.952341 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:37:42.403622 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:42.403586 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a566e97f-4f8a-499c-9630-7e5de078910a" path="/var/lib/kubelet/pods/a566e97f-4f8a-499c-9630-7e5de078910a/volumes" Oct 30 14:37:42.404214 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:42.404190 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4ee9039-363a-4761-8b74-dd5ff95a4949" path="/var/lib/kubelet/pods/f4ee9039-363a-4761-8b74-dd5ff95a4949/volumes" Oct 30 14:37:43.755591 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:43.755561 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:37:44.162245 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:44.162211 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf36da92-5cf7n"] Oct 30 14:37:44.162785 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:44.162771 4982 cpu_manager.go:414] "containerMap: removing container" podUID="f307466f-7382-4469-b6cd-b19e87a6fa81" containerName="revoke-keys" Oct 30 14:37:44.162848 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:44.162786 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f307466f-7382-4469-b6cd-b19e87a6fa81" containerName="revoke-keys" Oct 30 14:37:44.162848 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:44.162805 4982 cpu_manager.go:414] "containerMap: removing container" podUID="f4ee9039-363a-4761-8b74-dd5ff95a4949" containerName="payload-processing" Oct 30 14:37:44.162848 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:44.162810 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4ee9039-363a-4761-8b74-dd5ff95a4949" containerName="payload-processing" Oct 30 14:37:44.162848 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:44.162817 4982 cpu_manager.go:414] "containerMap: removing container" podUID="a566e97f-4f8a-499c-9630-7e5de078910a" containerName="payload-pre-processing" Oct 30 14:37:44.162848 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:44.162823 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a566e97f-4f8a-499c-9630-7e5de078910a" containerName="payload-pre-processing" Oct 30 14:37:44.163057 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:44.162942 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4ee9039-363a-4761-8b74-dd5ff95a4949" containerName="payload-processing" Oct 30 14:37:44.163057 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:44.162951 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="a566e97f-4f8a-499c-9630-7e5de078910a" containerName="payload-pre-processing" Oct 30 14:37:44.163057 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:44.162967 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="f307466f-7382-4469-b6cd-b19e87a6fa81" containerName="revoke-keys" Oct 30 14:37:44.168046 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:44.168025 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf36da92-5cf7n" Oct 30 14:37:44.189877 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:44.188960 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf36da92-5cf7n"] Oct 30 14:37:44.197531 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:44.197506 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:37:44.197717 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:44.197556 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:37:44.302540 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:44.302512 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2214368-f584-4b7f-b208-ea49e90e7829-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf36da92-5cf7n\" (UID: \"d2214368-f584-4b7f-b208-ea49e90e7829\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf36da92-5cf7n" Oct 30 14:37:44.302540 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:44.302543 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfxtm\" (UniqueName: \"kubernetes.io/projected/d2214368-f584-4b7f-b208-ea49e90e7829-kube-api-access-jfxtm\") pod \"maas-api-revoke-sub-bf36da92-5cf7n\" (UID: \"d2214368-f584-4b7f-b208-ea49e90e7829\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf36da92-5cf7n" Oct 30 14:37:44.302819 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:44.302667 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2214368-f584-4b7f-b208-ea49e90e7829-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf36da92-5cf7n\" (UID: \"d2214368-f584-4b7f-b208-ea49e90e7829\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf36da92-5cf7n" Oct 30 14:37:44.404202 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:44.404169 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2214368-f584-4b7f-b208-ea49e90e7829-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf36da92-5cf7n\" (UID: \"d2214368-f584-4b7f-b208-ea49e90e7829\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf36da92-5cf7n" Oct 30 14:37:44.404386 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:44.404214 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfxtm\" (UniqueName: \"kubernetes.io/projected/d2214368-f584-4b7f-b208-ea49e90e7829-kube-api-access-jfxtm\") pod \"maas-api-revoke-sub-bf36da92-5cf7n\" (UID: \"d2214368-f584-4b7f-b208-ea49e90e7829\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf36da92-5cf7n" Oct 30 14:37:44.404386 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:44.404310 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2214368-f584-4b7f-b208-ea49e90e7829-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf36da92-5cf7n\" (UID: \"d2214368-f584-4b7f-b208-ea49e90e7829\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf36da92-5cf7n" Oct 30 14:37:44.405182 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:44.405148 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2214368-f584-4b7f-b208-ea49e90e7829-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf36da92-5cf7n\" (UID: \"d2214368-f584-4b7f-b208-ea49e90e7829\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf36da92-5cf7n" Oct 30 14:37:44.409078 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:44.409052 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2214368-f584-4b7f-b208-ea49e90e7829-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf36da92-5cf7n\" (UID: \"d2214368-f584-4b7f-b208-ea49e90e7829\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf36da92-5cf7n" Oct 30 14:37:44.430315 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:44.430259 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfxtm\" (UniqueName: \"kubernetes.io/projected/d2214368-f584-4b7f-b208-ea49e90e7829-kube-api-access-jfxtm\") pod \"maas-api-revoke-sub-bf36da92-5cf7n\" (UID: \"d2214368-f584-4b7f-b208-ea49e90e7829\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf36da92-5cf7n" Oct 30 14:37:44.482878 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:44.482852 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf36da92-5cf7n" Oct 30 14:37:44.797872 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:44.797836 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf36da92-5cf7n"] Oct 30 14:37:44.799273 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:37:44.799240 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2214368_f584_4b7f_b208_ea49e90e7829.slice/crio-72888e3090a929905a9fe452cb2ef7e9c11b9fee32f7f98fe616015d2f9f8473 WatchSource:0}: Error finding container 72888e3090a929905a9fe452cb2ef7e9c11b9fee32f7f98fe616015d2f9f8473: Status 404 returned error can't find the container with id 72888e3090a929905a9fe452cb2ef7e9c11b9fee32f7f98fe616015d2f9f8473 Oct 30 14:37:45.027468 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:45.027425 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf36da92-5cf7n" event={"ID":"d2214368-f584-4b7f-b208-ea49e90e7829","Type":"ContainerStarted","Data":"72888e3090a929905a9fe452cb2ef7e9c11b9fee32f7f98fe616015d2f9f8473"} Oct 30 14:37:45.291742 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:45.291704 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4"] Oct 30 14:37:45.291946 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:45.291915 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4" podUID="e1633499-c0e2-4c59-b509-8c8937bc8b49" containerName="istio-proxy" containerID="cri-o://26af3da49249009817c3c279a1cbd0b5cf6e428d2c4a74d387c03df192203a19" gracePeriod=30 Oct 30 14:37:45.761542 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:45.761496 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:37:46.048776 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:46.048689 4982 generic.go:356] "Generic (PLEG): container finished" podID="d2214368-f584-4b7f-b208-ea49e90e7829" containerID="90a2b3b3168f8a4af54a68946309dea29dc6dc7118c34e3d187c2abc7ef8c340" exitCode=0 Oct 30 14:37:46.049128 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:46.048773 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf36da92-5cf7n" event={"ID":"d2214368-f584-4b7f-b208-ea49e90e7829","Type":"ContainerDied","Data":"90a2b3b3168f8a4af54a68946309dea29dc6dc7118c34e3d187c2abc7ef8c340"} Oct 30 14:37:46.286818 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:46.286777 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:37:46.521627 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:46.521578 4982 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 14:37:46.521814 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:46.521663 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4" podUID="e1633499-c0e2-4c59-b509-8c8937bc8b49" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 14:37:47.220963 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:47.220935 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf36da92-5cf7n" Oct 30 14:37:47.257532 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:47.257506 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2214368-f584-4b7f-b208-ea49e90e7829-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2214368-f584-4b7f-b208-ea49e90e7829-maas-api-openshift-ca\") pod \"d2214368-f584-4b7f-b208-ea49e90e7829\" (UID: \"d2214368-f584-4b7f-b208-ea49e90e7829\") " Oct 30 14:37:47.257693 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:47.257576 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2214368-f584-4b7f-b208-ea49e90e7829-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2214368-f584-4b7f-b208-ea49e90e7829-maas-api-xks-ca\") pod \"d2214368-f584-4b7f-b208-ea49e90e7829\" (UID: \"d2214368-f584-4b7f-b208-ea49e90e7829\") " Oct 30 14:37:47.257826 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:47.257804 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2214368-f584-4b7f-b208-ea49e90e7829-kube-api-access-jfxtm\" (UniqueName: \"kubernetes.io/projected/d2214368-f584-4b7f-b208-ea49e90e7829-kube-api-access-jfxtm\") pod \"d2214368-f584-4b7f-b208-ea49e90e7829\" (UID: \"d2214368-f584-4b7f-b208-ea49e90e7829\") " Oct 30 14:37:47.257967 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:47.257907 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2214368-f584-4b7f-b208-ea49e90e7829-maas-api-openshift-ca" pod "d2214368-f584-4b7f-b208-ea49e90e7829" (UID: "d2214368-f584-4b7f-b208-ea49e90e7829"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:37:47.258175 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:47.258156 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2214368-f584-4b7f-b208-ea49e90e7829-maas-api-openshift-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:37:47.259703 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:47.259679 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2214368-f584-4b7f-b208-ea49e90e7829-maas-api-xks-ca" pod "d2214368-f584-4b7f-b208-ea49e90e7829" (UID: "d2214368-f584-4b7f-b208-ea49e90e7829"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:37:47.259912 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:47.259893 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2214368-f584-4b7f-b208-ea49e90e7829-kube-api-access-jfxtm" pod "d2214368-f584-4b7f-b208-ea49e90e7829" (UID: "d2214368-f584-4b7f-b208-ea49e90e7829"). InnerVolumeSpecName "kube-api-access-jfxtm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:37:47.359579 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:47.359515 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfxtm\" (UniqueName: \"kubernetes.io/projected/d2214368-f584-4b7f-b208-ea49e90e7829-kube-api-access-jfxtm\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:37:47.359579 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:47.359544 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2214368-f584-4b7f-b208-ea49e90e7829-maas-api-xks-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:37:47.623255 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:47.623128 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:37:48.064806 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:48.064777 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf36da92-5cf7n" Oct 30 14:37:48.064968 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:48.064819 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf36da92-5cf7n" event={"ID":"d2214368-f584-4b7f-b208-ea49e90e7829","Type":"ContainerDied","Data":"72888e3090a929905a9fe452cb2ef7e9c11b9fee32f7f98fe616015d2f9f8473"} Oct 30 14:37:48.064968 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:48.064869 4982 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72888e3090a929905a9fe452cb2ef7e9c11b9fee32f7f98fe616015d2f9f8473" Oct 30 14:37:48.084364 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:48.084322 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w6-75443e/rate-test-model-75443e-kserve-6b64d597fc-74z2f"] Oct 30 14:37:48.085241 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:48.085214 4982 cpu_manager.go:414] "containerMap: removing container" podUID="d2214368-f584-4b7f-b208-ea49e90e7829" containerName="revoke-keys" Oct 30 14:37:48.085352 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:48.085241 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2214368-f584-4b7f-b208-ea49e90e7829" containerName="revoke-keys" Oct 30 14:37:48.085427 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:48.085415 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2214368-f584-4b7f-b208-ea49e90e7829" containerName="revoke-keys" Oct 30 14:37:48.089778 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:48.089752 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-75443e/rate-test-model-75443e-kserve-6b64d597fc-74z2f" Oct 30 14:37:48.121124 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:48.121096 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-75443e\"/\"kube-root-ca.crt\"" Oct 30 14:37:48.121381 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:48.121355 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-75443e\"/\"openshift-service-ca.crt\"" Oct 30 14:37:48.123150 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:48.123128 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-75443e/rate-test-model-75443e-kserve-6b64d597fc-74z2f"] Oct 30 14:37:48.132502 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:48.132473 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-75443e\"/\"default-dockercfg-smxvj\"" Oct 30 14:37:48.132793 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:48.132749 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-75443e\"/\"rate-test-model-75443e-kserve-self-signed-certs\"" Oct 30 14:37:48.169858 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:48.169826 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48dd08f0-9733-48b1-bb45-cd93e4bb0564-model-cache\") pod \"rate-test-model-75443e-kserve-6b64d597fc-74z2f\" (UID: \"48dd08f0-9733-48b1-bb45-cd93e4bb0564\") " pod="ai-tenant-e2e-shared-a-w6-75443e/rate-test-model-75443e-kserve-6b64d597fc-74z2f" Oct 30 14:37:48.170056 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:48.170021 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48dd08f0-9733-48b1-bb45-cd93e4bb0564-tmp-dir\") pod \"rate-test-model-75443e-kserve-6b64d597fc-74z2f\" (UID: \"48dd08f0-9733-48b1-bb45-cd93e4bb0564\") " pod="ai-tenant-e2e-shared-a-w6-75443e/rate-test-model-75443e-kserve-6b64d597fc-74z2f" Oct 30 14:37:48.170146 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:48.170086 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48dd08f0-9733-48b1-bb45-cd93e4bb0564-home\") pod \"rate-test-model-75443e-kserve-6b64d597fc-74z2f\" (UID: \"48dd08f0-9733-48b1-bb45-cd93e4bb0564\") " pod="ai-tenant-e2e-shared-a-w6-75443e/rate-test-model-75443e-kserve-6b64d597fc-74z2f" Oct 30 14:37:48.170211 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:48.170176 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb8jb\" (UniqueName: \"kubernetes.io/projected/48dd08f0-9733-48b1-bb45-cd93e4bb0564-kube-api-access-rb8jb\") pod \"rate-test-model-75443e-kserve-6b64d597fc-74z2f\" (UID: \"48dd08f0-9733-48b1-bb45-cd93e4bb0564\") " pod="ai-tenant-e2e-shared-a-w6-75443e/rate-test-model-75443e-kserve-6b64d597fc-74z2f" Oct 30 14:37:48.170272 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:48.170217 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48dd08f0-9733-48b1-bb45-cd93e4bb0564-tls-certs\") pod \"rate-test-model-75443e-kserve-6b64d597fc-74z2f\" (UID: \"48dd08f0-9733-48b1-bb45-cd93e4bb0564\") " pod="ai-tenant-e2e-shared-a-w6-75443e/rate-test-model-75443e-kserve-6b64d597fc-74z2f" Oct 30 14:37:48.170272 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:48.170259 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48dd08f0-9733-48b1-bb45-cd93e4bb0564-dshm\") pod \"rate-test-model-75443e-kserve-6b64d597fc-74z2f\" (UID: \"48dd08f0-9733-48b1-bb45-cd93e4bb0564\") " pod="ai-tenant-e2e-shared-a-w6-75443e/rate-test-model-75443e-kserve-6b64d597fc-74z2f" Oct 30 14:37:48.188199 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:48.188170 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:37:48.271028 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:48.270989 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48dd08f0-9733-48b1-bb45-cd93e4bb0564-tmp-dir\") pod \"rate-test-model-75443e-kserve-6b64d597fc-74z2f\" (UID: \"48dd08f0-9733-48b1-bb45-cd93e4bb0564\") " pod="ai-tenant-e2e-shared-a-w6-75443e/rate-test-model-75443e-kserve-6b64d597fc-74z2f" Oct 30 14:37:48.271476 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:48.271039 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48dd08f0-9733-48b1-bb45-cd93e4bb0564-home\") pod \"rate-test-model-75443e-kserve-6b64d597fc-74z2f\" (UID: \"48dd08f0-9733-48b1-bb45-cd93e4bb0564\") " pod="ai-tenant-e2e-shared-a-w6-75443e/rate-test-model-75443e-kserve-6b64d597fc-74z2f" Oct 30 14:37:48.271476 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:48.271068 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb8jb\" (UniqueName: \"kubernetes.io/projected/48dd08f0-9733-48b1-bb45-cd93e4bb0564-kube-api-access-rb8jb\") pod \"rate-test-model-75443e-kserve-6b64d597fc-74z2f\" (UID: \"48dd08f0-9733-48b1-bb45-cd93e4bb0564\") " pod="ai-tenant-e2e-shared-a-w6-75443e/rate-test-model-75443e-kserve-6b64d597fc-74z2f" Oct 30 14:37:48.271476 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:48.271086 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48dd08f0-9733-48b1-bb45-cd93e4bb0564-tls-certs\") pod \"rate-test-model-75443e-kserve-6b64d597fc-74z2f\" (UID: \"48dd08f0-9733-48b1-bb45-cd93e4bb0564\") " pod="ai-tenant-e2e-shared-a-w6-75443e/rate-test-model-75443e-kserve-6b64d597fc-74z2f" Oct 30 14:37:48.271476 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:48.271110 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48dd08f0-9733-48b1-bb45-cd93e4bb0564-dshm\") pod \"rate-test-model-75443e-kserve-6b64d597fc-74z2f\" (UID: \"48dd08f0-9733-48b1-bb45-cd93e4bb0564\") " pod="ai-tenant-e2e-shared-a-w6-75443e/rate-test-model-75443e-kserve-6b64d597fc-74z2f" Oct 30 14:37:48.271476 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:48.271195 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48dd08f0-9733-48b1-bb45-cd93e4bb0564-model-cache\") pod \"rate-test-model-75443e-kserve-6b64d597fc-74z2f\" (UID: \"48dd08f0-9733-48b1-bb45-cd93e4bb0564\") " pod="ai-tenant-e2e-shared-a-w6-75443e/rate-test-model-75443e-kserve-6b64d597fc-74z2f" Oct 30 14:37:48.271476 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:48.271444 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48dd08f0-9733-48b1-bb45-cd93e4bb0564-tmp-dir\") pod \"rate-test-model-75443e-kserve-6b64d597fc-74z2f\" (UID: \"48dd08f0-9733-48b1-bb45-cd93e4bb0564\") " pod="ai-tenant-e2e-shared-a-w6-75443e/rate-test-model-75443e-kserve-6b64d597fc-74z2f" Oct 30 14:37:48.272104 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:48.271483 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48dd08f0-9733-48b1-bb45-cd93e4bb0564-home\") pod \"rate-test-model-75443e-kserve-6b64d597fc-74z2f\" (UID: \"48dd08f0-9733-48b1-bb45-cd93e4bb0564\") " pod="ai-tenant-e2e-shared-a-w6-75443e/rate-test-model-75443e-kserve-6b64d597fc-74z2f" Oct 30 14:37:48.272104 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:48.271622 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48dd08f0-9733-48b1-bb45-cd93e4bb0564-model-cache\") pod \"rate-test-model-75443e-kserve-6b64d597fc-74z2f\" (UID: \"48dd08f0-9733-48b1-bb45-cd93e4bb0564\") " pod="ai-tenant-e2e-shared-a-w6-75443e/rate-test-model-75443e-kserve-6b64d597fc-74z2f" Oct 30 14:37:48.273598 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:48.273560 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48dd08f0-9733-48b1-bb45-cd93e4bb0564-dshm\") pod \"rate-test-model-75443e-kserve-6b64d597fc-74z2f\" (UID: \"48dd08f0-9733-48b1-bb45-cd93e4bb0564\") " pod="ai-tenant-e2e-shared-a-w6-75443e/rate-test-model-75443e-kserve-6b64d597fc-74z2f" Oct 30 14:37:48.273920 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:48.273901 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48dd08f0-9733-48b1-bb45-cd93e4bb0564-tls-certs\") pod \"rate-test-model-75443e-kserve-6b64d597fc-74z2f\" (UID: \"48dd08f0-9733-48b1-bb45-cd93e4bb0564\") " pod="ai-tenant-e2e-shared-a-w6-75443e/rate-test-model-75443e-kserve-6b64d597fc-74z2f" Oct 30 14:37:48.302473 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:48.302445 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb8jb\" (UniqueName: \"kubernetes.io/projected/48dd08f0-9733-48b1-bb45-cd93e4bb0564-kube-api-access-rb8jb\") pod \"rate-test-model-75443e-kserve-6b64d597fc-74z2f\" (UID: \"48dd08f0-9733-48b1-bb45-cd93e4bb0564\") " pod="ai-tenant-e2e-shared-a-w6-75443e/rate-test-model-75443e-kserve-6b64d597fc-74z2f" Oct 30 14:37:48.413911 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:48.413811 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-75443e/rate-test-model-75443e-kserve-6b64d597fc-74z2f" Oct 30 14:37:48.769728 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:48.769689 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-75443e/rate-test-model-75443e-kserve-6b64d597fc-74z2f"] Oct 30 14:37:48.770047 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:37:48.770027 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod48dd08f0_9733_48b1_bb45_cd93e4bb0564.slice/crio-9f489910d087911c7cdc09b25f5d991058af6764be72892ab8f407ea1f2b38b6 WatchSource:0}: Error finding container 9f489910d087911c7cdc09b25f5d991058af6764be72892ab8f407ea1f2b38b6: Status 404 returned error can't find the container with id 9f489910d087911c7cdc09b25f5d991058af6764be72892ab8f407ea1f2b38b6 Oct 30 14:37:49.073296 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:49.073262 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-75443e/rate-test-model-75443e-kserve-6b64d597fc-74z2f" event={"ID":"48dd08f0-9733-48b1-bb45-cd93e4bb0564","Type":"ContainerStarted","Data":"b431916a3c55afd0dbe6bb6e169a8f98091f38581a2b1918e8c3c97d6c3b7a56"} Oct 30 14:37:49.073454 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:49.073313 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-75443e/rate-test-model-75443e-kserve-6b64d597fc-74z2f" event={"ID":"48dd08f0-9733-48b1-bb45-cd93e4bb0564","Type":"ContainerStarted","Data":"9f489910d087911c7cdc09b25f5d991058af6764be72892ab8f407ea1f2b38b6"} Oct 30 14:37:49.119910 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:49.119851 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w6-75443e/rate-test-model-75443e-kserve-6b64d597fc-74z2f" podStartSLOduration=1.119833677 podStartE2EDuration="1.119833677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:37:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:37:49.118327659 +0000 UTC m=+2938.862461700" watchObservedRunningTime="2026-10-30 14:37:49.119833677 +0000 UTC m=+2938.863967718" Oct 30 14:37:49.414846 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:49.414754 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w6-75443e/rate-test-model-75443e-kserve-6b64d597fc-74z2f" Oct 30 14:37:49.532631 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:49.532575 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:37:50.977110 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:50.977072 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4" Oct 30 14:37:51.012830 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:51.012801 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e1633499-c0e2-4c59-b509-8c8937bc8b49-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1633499-c0e2-4c59-b509-8c8937bc8b49-istio-podinfo\") pod \"e1633499-c0e2-4c59-b509-8c8937bc8b49\" (UID: \"e1633499-c0e2-4c59-b509-8c8937bc8b49\") " Oct 30 14:37:51.012957 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:51.012844 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1633499-c0e2-4c59-b509-8c8937bc8b49-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1633499-c0e2-4c59-b509-8c8937bc8b49-credential-socket\") pod \"e1633499-c0e2-4c59-b509-8c8937bc8b49\" (UID: \"e1633499-c0e2-4c59-b509-8c8937bc8b49\") " Oct 30 14:37:51.012957 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:51.012871 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1633499-c0e2-4c59-b509-8c8937bc8b49-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1633499-c0e2-4c59-b509-8c8937bc8b49-istio-envoy\") pod \"e1633499-c0e2-4c59-b509-8c8937bc8b49\" (UID: \"e1633499-c0e2-4c59-b509-8c8937bc8b49\") " Oct 30 14:37:51.012957 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:51.012921 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1633499-c0e2-4c59-b509-8c8937bc8b49-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1633499-c0e2-4c59-b509-8c8937bc8b49-istio-data\") pod \"e1633499-c0e2-4c59-b509-8c8937bc8b49\" (UID: \"e1633499-c0e2-4c59-b509-8c8937bc8b49\") " Oct 30 14:37:51.013064 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:51.012977 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1633499-c0e2-4c59-b509-8c8937bc8b49-kube-api-access-bns85\" (UniqueName: \"kubernetes.io/projected/e1633499-c0e2-4c59-b509-8c8937bc8b49-kube-api-access-bns85\") pod \"e1633499-c0e2-4c59-b509-8c8937bc8b49\" (UID: \"e1633499-c0e2-4c59-b509-8c8937bc8b49\") " Oct 30 14:37:51.013064 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:51.013013 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1633499-c0e2-4c59-b509-8c8937bc8b49-istio-token\" (UniqueName: \"kubernetes.io/projected/e1633499-c0e2-4c59-b509-8c8937bc8b49-istio-token\") pod \"e1633499-c0e2-4c59-b509-8c8937bc8b49\" (UID: \"e1633499-c0e2-4c59-b509-8c8937bc8b49\") " Oct 30 14:37:51.013064 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:51.013054 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1633499-c0e2-4c59-b509-8c8937bc8b49-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1633499-c0e2-4c59-b509-8c8937bc8b49-workload-socket\") pod \"e1633499-c0e2-4c59-b509-8c8937bc8b49\" (UID: \"e1633499-c0e2-4c59-b509-8c8937bc8b49\") " Oct 30 14:37:51.013150 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:51.013127 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1633499-c0e2-4c59-b509-8c8937bc8b49-credential-socket" pod "e1633499-c0e2-4c59-b509-8c8937bc8b49" (UID: "e1633499-c0e2-4c59-b509-8c8937bc8b49"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:37:51.013258 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:51.013238 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1633499-c0e2-4c59-b509-8c8937bc8b49-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1633499-c0e2-4c59-b509-8c8937bc8b49-workload-certs\") pod \"e1633499-c0e2-4c59-b509-8c8937bc8b49\" (UID: \"e1633499-c0e2-4c59-b509-8c8937bc8b49\") " Oct 30 14:37:51.013437 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:51.013422 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1633499-c0e2-4c59-b509-8c8937bc8b49-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1633499-c0e2-4c59-b509-8c8937bc8b49-istiod-ca-cert\") pod \"e1633499-c0e2-4c59-b509-8c8937bc8b49\" (UID: \"e1633499-c0e2-4c59-b509-8c8937bc8b49\") " Oct 30 14:37:51.013794 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:51.013431 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1633499-c0e2-4c59-b509-8c8937bc8b49-istio-data" pod "e1633499-c0e2-4c59-b509-8c8937bc8b49" (UID: "e1633499-c0e2-4c59-b509-8c8937bc8b49"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:37:51.013794 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:51.013628 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1633499-c0e2-4c59-b509-8c8937bc8b49-workload-socket" pod "e1633499-c0e2-4c59-b509-8c8937bc8b49" (UID: "e1633499-c0e2-4c59-b509-8c8937bc8b49"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:37:51.013794 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:51.013681 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1633499-c0e2-4c59-b509-8c8937bc8b49-workload-certs" pod "e1633499-c0e2-4c59-b509-8c8937bc8b49" (UID: "e1633499-c0e2-4c59-b509-8c8937bc8b49"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:37:51.014071 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:51.013850 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1633499-c0e2-4c59-b509-8c8937bc8b49-istiod-ca-cert" pod "e1633499-c0e2-4c59-b509-8c8937bc8b49" (UID: "e1633499-c0e2-4c59-b509-8c8937bc8b49"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:37:51.014539 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:51.014514 4982 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1633499-c0e2-4c59-b509-8c8937bc8b49-istio-data\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:37:51.014671 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:51.014542 4982 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1633499-c0e2-4c59-b509-8c8937bc8b49-workload-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:37:51.014671 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:51.014555 4982 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1633499-c0e2-4c59-b509-8c8937bc8b49-workload-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:37:51.014671 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:51.014568 4982 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1633499-c0e2-4c59-b509-8c8937bc8b49-istiod-ca-cert\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:37:51.014671 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:51.014583 4982 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1633499-c0e2-4c59-b509-8c8937bc8b49-credential-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:37:51.015426 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:51.015401 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e1633499-c0e2-4c59-b509-8c8937bc8b49-istio-podinfo" pod "e1633499-c0e2-4c59-b509-8c8937bc8b49" (UID: "e1633499-c0e2-4c59-b509-8c8937bc8b49"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:37:51.015631 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:51.015609 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1633499-c0e2-4c59-b509-8c8937bc8b49-kube-api-access-bns85" pod "e1633499-c0e2-4c59-b509-8c8937bc8b49" (UID: "e1633499-c0e2-4c59-b509-8c8937bc8b49"). InnerVolumeSpecName "kube-api-access-bns85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:37:51.015829 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:51.015807 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1633499-c0e2-4c59-b509-8c8937bc8b49-istio-token" pod "e1633499-c0e2-4c59-b509-8c8937bc8b49" (UID: "e1633499-c0e2-4c59-b509-8c8937bc8b49"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:37:51.015893 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:51.015841 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1633499-c0e2-4c59-b509-8c8937bc8b49-istio-envoy" pod "e1633499-c0e2-4c59-b509-8c8937bc8b49" (UID: "e1633499-c0e2-4c59-b509-8c8937bc8b49"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:37:51.093203 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:51.093123 4982 generic.go:356] "Generic (PLEG): container finished" podID="e1633499-c0e2-4c59-b509-8c8937bc8b49" containerID="26af3da49249009817c3c279a1cbd0b5cf6e428d2c4a74d387c03df192203a19" exitCode=0 Oct 30 14:37:51.093203 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:51.093178 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4" Oct 30 14:37:51.093422 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:51.093212 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4" event={"ID":"e1633499-c0e2-4c59-b509-8c8937bc8b49","Type":"ContainerDied","Data":"26af3da49249009817c3c279a1cbd0b5cf6e428d2c4a74d387c03df192203a19"} Oct 30 14:37:51.093422 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:51.093272 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4" event={"ID":"e1633499-c0e2-4c59-b509-8c8937bc8b49","Type":"ContainerDied","Data":"f6fd85fca1f20e35124c070b2721ab8b60bffeffe5ed27e15818e5e8cb86cd14"} Oct 30 14:37:51.093422 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:51.093302 4982 scope.go:122] "RemoveContainer" containerID="26af3da49249009817c3c279a1cbd0b5cf6e428d2c4a74d387c03df192203a19" Oct 30 14:37:51.106365 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:51.106347 4982 scope.go:122] "RemoveContainer" containerID="26af3da49249009817c3c279a1cbd0b5cf6e428d2c4a74d387c03df192203a19" Oct 30 14:37:51.106693 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:37:51.106667 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26af3da49249009817c3c279a1cbd0b5cf6e428d2c4a74d387c03df192203a19\": container with ID starting with 26af3da49249009817c3c279a1cbd0b5cf6e428d2c4a74d387c03df192203a19 not found: ID does not exist" containerID="26af3da49249009817c3c279a1cbd0b5cf6e428d2c4a74d387c03df192203a19" Oct 30 14:37:51.106798 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:51.106696 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26af3da49249009817c3c279a1cbd0b5cf6e428d2c4a74d387c03df192203a19"} err="failed to get container status \"26af3da49249009817c3c279a1cbd0b5cf6e428d2c4a74d387c03df192203a19\": rpc error: code = NotFound desc = could not find container \"26af3da49249009817c3c279a1cbd0b5cf6e428d2c4a74d387c03df192203a19\": container with ID starting with 26af3da49249009817c3c279a1cbd0b5cf6e428d2c4a74d387c03df192203a19 not found: ID does not exist" Oct 30 14:37:51.115617 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:51.115593 4982 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1633499-c0e2-4c59-b509-8c8937bc8b49-istio-podinfo\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:37:51.115716 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:51.115622 4982 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1633499-c0e2-4c59-b509-8c8937bc8b49-istio-envoy\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:37:51.115716 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:51.115652 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bns85\" (UniqueName: \"kubernetes.io/projected/e1633499-c0e2-4c59-b509-8c8937bc8b49-kube-api-access-bns85\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:37:51.115716 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:51.115669 4982 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1633499-c0e2-4c59-b509-8c8937bc8b49-istio-token\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:37:51.168146 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:51.168115 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4"] Oct 30 14:37:51.173822 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:51.173794 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-mqmb4"] Oct 30 14:37:51.542274 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:51.542199 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:37:51.808325 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:51.808295 4982 scope.go:122] "RemoveContainer" containerID="9ee010f1c7d06772fd92c3ae361f625711546b32cc3cecc41faf851b2fd9aac2" Oct 30 14:37:51.822238 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:51.822212 4982 scope.go:122] "RemoveContainer" containerID="f5fc1d0829dd04a8322f20dc60838701fc37d7e2049329febf466a994eb8f141" Oct 30 14:37:51.833543 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:51.833520 4982 scope.go:122] "RemoveContainer" containerID="c0e20d9281483b5d4f8e912d9aeeeb652cf41ab81912e710b876babe4e789f8e" Oct 30 14:37:51.842164 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:51.842148 4982 scope.go:122] "RemoveContainer" containerID="a12fcf491f1e9ef73ea04451090d06b541442570767df0445a074a65c06fa9f4" Oct 30 14:37:51.850473 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:51.850452 4982 scope.go:122] "RemoveContainer" containerID="18caf6b5fb83db8f935ce4ed7cc69fe8b3ddeb0f56fba3279c28fd5515a0566e" Oct 30 14:37:52.401230 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:52.401198 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1633499-c0e2-4c59-b509-8c8937bc8b49" path="/var/lib/kubelet/pods/e1633499-c0e2-4c59-b509-8c8937bc8b49/volumes" Oct 30 14:37:53.615666 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:53.612679 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:37:55.567842 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:55.567805 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:37:57.547414 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:57.547384 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:37:58.015367 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:37:58.015328 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:37:58.415452 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:58.415409 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w6-75443e/rate-test-model-75443e-kserve-6b64d597fc-74z2f" Oct 30 14:37:58.428461 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:58.428435 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w6-75443e/rate-test-model-75443e-kserve-6b64d597fc-74z2f" Oct 30 14:37:59.163538 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:59.163507 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w6-75443e/rate-test-model-75443e-kserve-6b64d597fc-74z2f" Oct 30 14:37:59.350464 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:37:59.350426 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:38:00.036559 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:00.036525 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80180acb-ww7mh"] Oct 30 14:38:00.049225 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:00.049199 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80180acb-ww7mh"] Oct 30 14:38:00.402257 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:00.402217 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="260f52cf-e8e0-4122-9818-f9e8a149cb55" path="/var/lib/kubelet/pods/260f52cf-e8e0-4122-9818-f9e8a149cb55/volumes" Oct 30 14:38:00.571255 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:00.571216 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-f1f206/auth-test-model-f1f206-kserve-74c48d6ddf-wx5tn"] Oct 30 14:38:00.572009 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:00.571981 4982 cpu_manager.go:414] "containerMap: removing container" podUID="e1633499-c0e2-4c59-b509-8c8937bc8b49" containerName="istio-proxy" Oct 30 14:38:00.572009 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:00.572007 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1633499-c0e2-4c59-b509-8c8937bc8b49" containerName="istio-proxy" Oct 30 14:38:00.572251 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:00.572235 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1633499-c0e2-4c59-b509-8c8937bc8b49" containerName="istio-proxy" Oct 30 14:38:00.578187 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:00.578165 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-f1f206/auth-test-model-f1f206-kserve-74c48d6ddf-wx5tn" Oct 30 14:38:00.598873 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:00.598846 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-f1f206/auth-test-model-f1f206-kserve-74c48d6ddf-wx5tn"] Oct 30 14:38:00.605734 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:00.605708 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-f1f206\"/\"auth-test-model-f1f206-kserve-self-signed-certs\"" Oct 30 14:38:00.605873 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:00.605735 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-f1f206\"/\"default-dockercfg-ktr5f\"" Oct 30 14:38:00.607554 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:00.607535 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-f1f206\"/\"openshift-service-ca.crt\"" Oct 30 14:38:00.607683 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:00.607578 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-f1f206\"/\"kube-root-ca.crt\"" Oct 30 14:38:00.663881 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:00.663806 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:38:00.671271 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:00.671240 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8483d44c-7904-49bc-8f55-87870acc8e58-dshm\") pod \"auth-test-model-f1f206-kserve-74c48d6ddf-wx5tn\" (UID: \"8483d44c-7904-49bc-8f55-87870acc8e58\") " pod="ai-tenant-e2e-shared-b-w0-f1f206/auth-test-model-f1f206-kserve-74c48d6ddf-wx5tn" Oct 30 14:38:00.671374 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:00.671307 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9t4x\" (UniqueName: \"kubernetes.io/projected/8483d44c-7904-49bc-8f55-87870acc8e58-kube-api-access-n9t4x\") pod \"auth-test-model-f1f206-kserve-74c48d6ddf-wx5tn\" (UID: \"8483d44c-7904-49bc-8f55-87870acc8e58\") " pod="ai-tenant-e2e-shared-b-w0-f1f206/auth-test-model-f1f206-kserve-74c48d6ddf-wx5tn" Oct 30 14:38:00.671374 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:00.671355 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8483d44c-7904-49bc-8f55-87870acc8e58-home\") pod \"auth-test-model-f1f206-kserve-74c48d6ddf-wx5tn\" (UID: \"8483d44c-7904-49bc-8f55-87870acc8e58\") " pod="ai-tenant-e2e-shared-b-w0-f1f206/auth-test-model-f1f206-kserve-74c48d6ddf-wx5tn" Oct 30 14:38:00.671505 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:00.671427 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8483d44c-7904-49bc-8f55-87870acc8e58-model-cache\") pod \"auth-test-model-f1f206-kserve-74c48d6ddf-wx5tn\" (UID: \"8483d44c-7904-49bc-8f55-87870acc8e58\") " pod="ai-tenant-e2e-shared-b-w0-f1f206/auth-test-model-f1f206-kserve-74c48d6ddf-wx5tn" Oct 30 14:38:00.671672 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:00.671648 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8483d44c-7904-49bc-8f55-87870acc8e58-tls-certs\") pod \"auth-test-model-f1f206-kserve-74c48d6ddf-wx5tn\" (UID: \"8483d44c-7904-49bc-8f55-87870acc8e58\") " pod="ai-tenant-e2e-shared-b-w0-f1f206/auth-test-model-f1f206-kserve-74c48d6ddf-wx5tn" Oct 30 14:38:00.671892 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:00.671867 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8483d44c-7904-49bc-8f55-87870acc8e58-tmp-dir\") pod \"auth-test-model-f1f206-kserve-74c48d6ddf-wx5tn\" (UID: \"8483d44c-7904-49bc-8f55-87870acc8e58\") " pod="ai-tenant-e2e-shared-b-w0-f1f206/auth-test-model-f1f206-kserve-74c48d6ddf-wx5tn" Oct 30 14:38:00.772862 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:00.772824 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8483d44c-7904-49bc-8f55-87870acc8e58-tmp-dir\") pod \"auth-test-model-f1f206-kserve-74c48d6ddf-wx5tn\" (UID: \"8483d44c-7904-49bc-8f55-87870acc8e58\") " pod="ai-tenant-e2e-shared-b-w0-f1f206/auth-test-model-f1f206-kserve-74c48d6ddf-wx5tn" Oct 30 14:38:00.773061 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:00.772911 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8483d44c-7904-49bc-8f55-87870acc8e58-dshm\") pod \"auth-test-model-f1f206-kserve-74c48d6ddf-wx5tn\" (UID: \"8483d44c-7904-49bc-8f55-87870acc8e58\") " pod="ai-tenant-e2e-shared-b-w0-f1f206/auth-test-model-f1f206-kserve-74c48d6ddf-wx5tn" Oct 30 14:38:00.773061 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:00.772945 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9t4x\" (UniqueName: \"kubernetes.io/projected/8483d44c-7904-49bc-8f55-87870acc8e58-kube-api-access-n9t4x\") pod \"auth-test-model-f1f206-kserve-74c48d6ddf-wx5tn\" (UID: \"8483d44c-7904-49bc-8f55-87870acc8e58\") " pod="ai-tenant-e2e-shared-b-w0-f1f206/auth-test-model-f1f206-kserve-74c48d6ddf-wx5tn" Oct 30 14:38:00.773061 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:00.772971 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8483d44c-7904-49bc-8f55-87870acc8e58-home\") pod \"auth-test-model-f1f206-kserve-74c48d6ddf-wx5tn\" (UID: \"8483d44c-7904-49bc-8f55-87870acc8e58\") " pod="ai-tenant-e2e-shared-b-w0-f1f206/auth-test-model-f1f206-kserve-74c48d6ddf-wx5tn" Oct 30 14:38:00.773061 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:00.772992 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8483d44c-7904-49bc-8f55-87870acc8e58-model-cache\") pod \"auth-test-model-f1f206-kserve-74c48d6ddf-wx5tn\" (UID: \"8483d44c-7904-49bc-8f55-87870acc8e58\") " pod="ai-tenant-e2e-shared-b-w0-f1f206/auth-test-model-f1f206-kserve-74c48d6ddf-wx5tn" Oct 30 14:38:00.773331 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:00.773112 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8483d44c-7904-49bc-8f55-87870acc8e58-tls-certs\") pod \"auth-test-model-f1f206-kserve-74c48d6ddf-wx5tn\" (UID: \"8483d44c-7904-49bc-8f55-87870acc8e58\") " pod="ai-tenant-e2e-shared-b-w0-f1f206/auth-test-model-f1f206-kserve-74c48d6ddf-wx5tn" Oct 30 14:38:00.773331 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:00.773245 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8483d44c-7904-49bc-8f55-87870acc8e58-tmp-dir\") pod \"auth-test-model-f1f206-kserve-74c48d6ddf-wx5tn\" (UID: \"8483d44c-7904-49bc-8f55-87870acc8e58\") " pod="ai-tenant-e2e-shared-b-w0-f1f206/auth-test-model-f1f206-kserve-74c48d6ddf-wx5tn" Oct 30 14:38:00.773459 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:00.773348 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8483d44c-7904-49bc-8f55-87870acc8e58-home\") pod \"auth-test-model-f1f206-kserve-74c48d6ddf-wx5tn\" (UID: \"8483d44c-7904-49bc-8f55-87870acc8e58\") " pod="ai-tenant-e2e-shared-b-w0-f1f206/auth-test-model-f1f206-kserve-74c48d6ddf-wx5tn" Oct 30 14:38:00.773560 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:00.773532 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8483d44c-7904-49bc-8f55-87870acc8e58-model-cache\") pod \"auth-test-model-f1f206-kserve-74c48d6ddf-wx5tn\" (UID: \"8483d44c-7904-49bc-8f55-87870acc8e58\") " pod="ai-tenant-e2e-shared-b-w0-f1f206/auth-test-model-f1f206-kserve-74c48d6ddf-wx5tn" Oct 30 14:38:00.775588 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:00.775561 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8483d44c-7904-49bc-8f55-87870acc8e58-dshm\") pod \"auth-test-model-f1f206-kserve-74c48d6ddf-wx5tn\" (UID: \"8483d44c-7904-49bc-8f55-87870acc8e58\") " pod="ai-tenant-e2e-shared-b-w0-f1f206/auth-test-model-f1f206-kserve-74c48d6ddf-wx5tn" Oct 30 14:38:00.775888 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:00.775864 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8483d44c-7904-49bc-8f55-87870acc8e58-tls-certs\") pod \"auth-test-model-f1f206-kserve-74c48d6ddf-wx5tn\" (UID: \"8483d44c-7904-49bc-8f55-87870acc8e58\") " pod="ai-tenant-e2e-shared-b-w0-f1f206/auth-test-model-f1f206-kserve-74c48d6ddf-wx5tn" Oct 30 14:38:00.799750 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:00.799722 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9t4x\" (UniqueName: \"kubernetes.io/projected/8483d44c-7904-49bc-8f55-87870acc8e58-kube-api-access-n9t4x\") pod \"auth-test-model-f1f206-kserve-74c48d6ddf-wx5tn\" (UID: \"8483d44c-7904-49bc-8f55-87870acc8e58\") " pod="ai-tenant-e2e-shared-b-w0-f1f206/auth-test-model-f1f206-kserve-74c48d6ddf-wx5tn" Oct 30 14:38:00.903933 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:00.903896 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-f1f206/auth-test-model-f1f206-kserve-74c48d6ddf-wx5tn" Oct 30 14:38:01.240722 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:01.240694 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-f1f206/auth-test-model-f1f206-kserve-74c48d6ddf-wx5tn"] Oct 30 14:38:01.242112 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:38:01.242088 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8483d44c_7904_49bc_8f55_87870acc8e58.slice/crio-1541c740dac658f565fe209e5a1ed1b5479da3736d97ff579ad7cde93b666e6c WatchSource:0}: Error finding container 1541c740dac658f565fe209e5a1ed1b5479da3736d97ff579ad7cde93b666e6c: Status 404 returned error can't find the container with id 1541c740dac658f565fe209e5a1ed1b5479da3736d97ff579ad7cde93b666e6c Oct 30 14:38:01.265761 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:01.265730 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:38:02.178125 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:02.178087 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-f1f206/auth-test-model-f1f206-kserve-74c48d6ddf-wx5tn" event={"ID":"8483d44c-7904-49bc-8f55-87870acc8e58","Type":"ContainerStarted","Data":"8a74e3bf8d6104c6b16060e485e7738bb577b5bbbe12a7a680cc2bb2fafc5139"} Oct 30 14:38:02.178618 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:02.178137 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-f1f206/auth-test-model-f1f206-kserve-74c48d6ddf-wx5tn" event={"ID":"8483d44c-7904-49bc-8f55-87870acc8e58","Type":"ContainerStarted","Data":"1541c740dac658f565fe209e5a1ed1b5479da3736d97ff579ad7cde93b666e6c"} Oct 30 14:38:02.227146 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:02.227085 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-f1f206/auth-test-model-f1f206-kserve-74c48d6ddf-wx5tn" podStartSLOduration=2.227068648 podStartE2EDuration="2.227068648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:38:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:38:02.224075879 +0000 UTC m=+2951.968209919" watchObservedRunningTime="2026-10-30 14:38:02.227068648 +0000 UTC m=+2951.971202691" Oct 30 14:38:02.435893 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:02.435786 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:38:02.905043 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:02.905007 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-f1f206/auth-test-model-f1f206-kserve-74c48d6ddf-wx5tn" Oct 30 14:38:03.018213 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:03.018177 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:38:03.326127 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:03.326075 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:38:03.486966 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:03.486927 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:38:03.648556 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:03.648469 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:38:03.806455 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:03.806417 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:38:05.244790 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:05.244734 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:38:07.193089 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:07.193049 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:38:09.053936 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:09.053903 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:38:10.904498 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:10.904462 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-f1f206/auth-test-model-f1f206-kserve-74c48d6ddf-wx5tn" Oct 30 14:38:10.922993 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:10.922966 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-f1f206/auth-test-model-f1f206-kserve-74c48d6ddf-wx5tn" Oct 30 14:38:11.046061 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:11.046028 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e25b4aa-mg4n9"] Oct 30 14:38:11.057718 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:11.057689 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:38:11.059235 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:11.059199 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e25b4aa-mg4n9"] Oct 30 14:38:11.275078 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:11.275003 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-f1f206/auth-test-model-f1f206-kserve-74c48d6ddf-wx5tn" Oct 30 14:38:12.401252 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:12.401214 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4be53bee-e002-4021-b073-f1609e19a1fd" path="/var/lib/kubelet/pods/4be53bee-e002-4021-b073-f1609e19a1fd/volumes" Oct 30 14:38:13.011783 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:13.011745 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:38:14.444451 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:14.444394 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:38:14.622994 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:14.622954 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:38:14.854444 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:14.854404 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d3a829-64f6b74c79-6wf64"] Oct 30 14:38:14.869997 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:14.869968 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d3a829-64f6b74c79-6wf64" Oct 30 14:38:14.875838 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:14.875790 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d3a829-64f6b74c79-6wf64"] Oct 30 14:38:14.890735 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:14.890702 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-d3a829\"" Oct 30 14:38:14.895926 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:14.895900 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-d3a829-dockercfg-jd4tt\"" Oct 30 14:38:14.996377 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:14.996331 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baeb0f86-9313-4cb6-a075-8d3cdd5e9e53-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d3a829-64f6b74c79-6wf64\" (UID: \"baeb0f86-9313-4cb6-a075-8d3cdd5e9e53\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d3a829-64f6b74c79-6wf64" Oct 30 14:38:14.996573 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:14.996470 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll9hs\" (UniqueName: \"kubernetes.io/projected/baeb0f86-9313-4cb6-a075-8d3cdd5e9e53-kube-api-access-ll9hs\") pod \"payload-pre-processing-e2e-ipp-a-d3a829-64f6b74c79-6wf64\" (UID: \"baeb0f86-9313-4cb6-a075-8d3cdd5e9e53\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d3a829-64f6b74c79-6wf64" Oct 30 14:38:15.097570 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:15.097527 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baeb0f86-9313-4cb6-a075-8d3cdd5e9e53-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d3a829-64f6b74c79-6wf64\" (UID: \"baeb0f86-9313-4cb6-a075-8d3cdd5e9e53\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d3a829-64f6b74c79-6wf64" Oct 30 14:38:15.097797 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:15.097683 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll9hs\" (UniqueName: \"kubernetes.io/projected/baeb0f86-9313-4cb6-a075-8d3cdd5e9e53-kube-api-access-ll9hs\") pod \"payload-pre-processing-e2e-ipp-a-d3a829-64f6b74c79-6wf64\" (UID: \"baeb0f86-9313-4cb6-a075-8d3cdd5e9e53\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d3a829-64f6b74c79-6wf64" Oct 30 14:38:15.098322 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:15.098296 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baeb0f86-9313-4cb6-a075-8d3cdd5e9e53-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d3a829-64f6b74c79-6wf64\" (UID: \"baeb0f86-9313-4cb6-a075-8d3cdd5e9e53\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d3a829-64f6b74c79-6wf64" Oct 30 14:38:15.124629 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:15.124558 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll9hs\" (UniqueName: \"kubernetes.io/projected/baeb0f86-9313-4cb6-a075-8d3cdd5e9e53-kube-api-access-ll9hs\") pod \"payload-pre-processing-e2e-ipp-a-d3a829-64f6b74c79-6wf64\" (UID: \"baeb0f86-9313-4cb6-a075-8d3cdd5e9e53\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d3a829-64f6b74c79-6wf64" Oct 30 14:38:15.200810 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:15.200767 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d3a829-64f6b74c79-6wf64" Oct 30 14:38:15.407719 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:15.407612 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84"] Oct 30 14:38:15.407936 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:15.407897 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84" podUID="e0991b98-e891-4582-a9ce-88846fcaad92" containerName="istio-proxy" containerID="cri-o://f50889a5471bfa929aca7f9a76117b150a51172237b53872b17585b70b801543" gracePeriod=30 Oct 30 14:38:15.518534 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:38:15.518498 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaeb0f86_9313_4cb6_a075_8d3cdd5e9e53.slice/crio-1876bd1a4db40e11e9c2e87ae3b5df48981798bdc825b8cd9df8ce7dc1224b12 WatchSource:0}: Error finding container 1876bd1a4db40e11e9c2e87ae3b5df48981798bdc825b8cd9df8ce7dc1224b12: Status 404 returned error can't find the container with id 1876bd1a4db40e11e9c2e87ae3b5df48981798bdc825b8cd9df8ce7dc1224b12 Oct 30 14:38:15.519748 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:15.519715 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d3a829-64f6b74c79-6wf64"] Oct 30 14:38:15.520938 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:15.520893 4982 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 14:38:15.600892 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:15.600859 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:38:15.777609 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:15.777532 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w"] Oct 30 14:38:15.777814 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:15.777742 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w" podUID="7db57691-2261-4a03-abe3-a0fa7bb45429" containerName="istio-proxy" containerID="cri-o://2a2859a9d0d837ecc3a1e65a024a68b3cc7954ef8cce3bc1b7fb288607668961" gracePeriod=30 Oct 30 14:38:16.308628 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:16.308587 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d3a829-64f6b74c79-6wf64" event={"ID":"baeb0f86-9313-4cb6-a075-8d3cdd5e9e53","Type":"ContainerStarted","Data":"1876bd1a4db40e11e9c2e87ae3b5df48981798bdc825b8cd9df8ce7dc1224b12"} Oct 30 14:38:17.320957 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:17.320918 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d3a829-64f6b74c79-6wf64" event={"ID":"baeb0f86-9313-4cb6-a075-8d3cdd5e9e53","Type":"ContainerStarted","Data":"c19558b4446c8e2437d863eae5610e60b2c444ba16663583fa24c8c5550a47cc"} Oct 30 14:38:17.321446 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:17.321052 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d3a829-64f6b74c79-6wf64" Oct 30 14:38:17.366515 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:17.366460 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d3a829-64f6b74c79-6wf64" podStartSLOduration=2.049681123 podStartE2EDuration="3.366446091s" totalImagesPullingTime="1.316764968s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:38:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:38:17.365057243 +0000 UTC m=+2967.109191283" watchObservedRunningTime="2026-10-30 14:38:17.366446091 +0000 UTC m=+2967.110580132" Oct 30 14:38:17.632154 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:17.632072 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:38:17.813471 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:17.811996 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:38:18.071219 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:18.071186 4982 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 14:38:18.071393 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:18.071237 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84" podUID="e0991b98-e891-4582-a9ce-88846fcaad92" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 14:38:18.234921 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:18.234881 4982 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 14:38:18.235079 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:18.234943 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w" podUID="7db57691-2261-4a03-abe3-a0fa7bb45429" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 14:38:19.335668 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:19.335618 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:38:19.479389 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:19.479359 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:38:19.932918 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:19.932867 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:38:20.701508 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:20.701449 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84" Oct 30 14:38:20.787695 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:20.787659 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0991b98-e891-4582-a9ce-88846fcaad92-kube-api-access-wnmtb\" (UniqueName: \"kubernetes.io/projected/e0991b98-e891-4582-a9ce-88846fcaad92-kube-api-access-wnmtb\") pod \"e0991b98-e891-4582-a9ce-88846fcaad92\" (UID: \"e0991b98-e891-4582-a9ce-88846fcaad92\") " Oct 30 14:38:20.787890 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:20.787748 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0991b98-e891-4582-a9ce-88846fcaad92-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0991b98-e891-4582-a9ce-88846fcaad92-istio-data\") pod \"e0991b98-e891-4582-a9ce-88846fcaad92\" (UID: \"e0991b98-e891-4582-a9ce-88846fcaad92\") " Oct 30 14:38:20.787890 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:20.787782 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0991b98-e891-4582-a9ce-88846fcaad92-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0991b98-e891-4582-a9ce-88846fcaad92-workload-certs\") pod \"e0991b98-e891-4582-a9ce-88846fcaad92\" (UID: \"e0991b98-e891-4582-a9ce-88846fcaad92\") " Oct 30 14:38:20.787890 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:20.787817 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0991b98-e891-4582-a9ce-88846fcaad92-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0991b98-e891-4582-a9ce-88846fcaad92-istio-envoy\") pod \"e0991b98-e891-4582-a9ce-88846fcaad92\" (UID: \"e0991b98-e891-4582-a9ce-88846fcaad92\") " Oct 30 14:38:20.787890 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:20.787869 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0991b98-e891-4582-a9ce-88846fcaad92-istio-token\" (UniqueName: \"kubernetes.io/projected/e0991b98-e891-4582-a9ce-88846fcaad92-istio-token\") pod \"e0991b98-e891-4582-a9ce-88846fcaad92\" (UID: \"e0991b98-e891-4582-a9ce-88846fcaad92\") " Oct 30 14:38:20.788140 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:20.788004 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0991b98-e891-4582-a9ce-88846fcaad92-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0991b98-e891-4582-a9ce-88846fcaad92-workload-socket\") pod \"e0991b98-e891-4582-a9ce-88846fcaad92\" (UID: \"e0991b98-e891-4582-a9ce-88846fcaad92\") " Oct 30 14:38:20.788140 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:20.788118 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e0991b98-e891-4582-a9ce-88846fcaad92-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0991b98-e891-4582-a9ce-88846fcaad92-istio-podinfo\") pod \"e0991b98-e891-4582-a9ce-88846fcaad92\" (UID: \"e0991b98-e891-4582-a9ce-88846fcaad92\") " Oct 30 14:38:20.788256 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:20.788179 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0991b98-e891-4582-a9ce-88846fcaad92-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0991b98-e891-4582-a9ce-88846fcaad92-credential-socket\") pod \"e0991b98-e891-4582-a9ce-88846fcaad92\" (UID: \"e0991b98-e891-4582-a9ce-88846fcaad92\") " Oct 30 14:38:20.788343 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:20.788315 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0991b98-e891-4582-a9ce-88846fcaad92-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0991b98-e891-4582-a9ce-88846fcaad92-istiod-ca-cert\") pod \"e0991b98-e891-4582-a9ce-88846fcaad92\" (UID: \"e0991b98-e891-4582-a9ce-88846fcaad92\") " Oct 30 14:38:20.789226 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:20.789200 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0991b98-e891-4582-a9ce-88846fcaad92-istiod-ca-cert" pod "e0991b98-e891-4582-a9ce-88846fcaad92" (UID: "e0991b98-e891-4582-a9ce-88846fcaad92"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:38:20.790791 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:20.790682 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0991b98-e891-4582-a9ce-88846fcaad92-workload-certs" pod "e0991b98-e891-4582-a9ce-88846fcaad92" (UID: "e0991b98-e891-4582-a9ce-88846fcaad92"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:38:20.791455 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:20.791200 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0991b98-e891-4582-a9ce-88846fcaad92-istio-data" pod "e0991b98-e891-4582-a9ce-88846fcaad92" (UID: "e0991b98-e891-4582-a9ce-88846fcaad92"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:38:20.791895 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:20.791872 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0991b98-e891-4582-a9ce-88846fcaad92-workload-socket" pod "e0991b98-e891-4582-a9ce-88846fcaad92" (UID: "e0991b98-e891-4582-a9ce-88846fcaad92"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:38:20.792283 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:20.792260 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0991b98-e891-4582-a9ce-88846fcaad92-credential-socket" pod "e0991b98-e891-4582-a9ce-88846fcaad92" (UID: "e0991b98-e891-4582-a9ce-88846fcaad92"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:38:20.792526 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:20.792494 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0991b98-e891-4582-a9ce-88846fcaad92-kube-api-access-wnmtb" pod "e0991b98-e891-4582-a9ce-88846fcaad92" (UID: "e0991b98-e891-4582-a9ce-88846fcaad92"). InnerVolumeSpecName "kube-api-access-wnmtb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:38:20.792868 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:20.792845 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0991b98-e891-4582-a9ce-88846fcaad92-istio-token" pod "e0991b98-e891-4582-a9ce-88846fcaad92" (UID: "e0991b98-e891-4582-a9ce-88846fcaad92"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:38:20.793113 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:20.793078 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e0991b98-e891-4582-a9ce-88846fcaad92-istio-podinfo" pod "e0991b98-e891-4582-a9ce-88846fcaad92" (UID: "e0991b98-e891-4582-a9ce-88846fcaad92"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:38:20.794114 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:20.794082 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0991b98-e891-4582-a9ce-88846fcaad92-istio-envoy" pod "e0991b98-e891-4582-a9ce-88846fcaad92" (UID: "e0991b98-e891-4582-a9ce-88846fcaad92"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:38:20.890331 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:20.890299 4982 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0991b98-e891-4582-a9ce-88846fcaad92-workload-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:38:20.890331 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:20.890332 4982 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0991b98-e891-4582-a9ce-88846fcaad92-istio-podinfo\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:38:20.890602 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:20.890348 4982 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0991b98-e891-4582-a9ce-88846fcaad92-credential-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:38:20.890602 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:20.890361 4982 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0991b98-e891-4582-a9ce-88846fcaad92-istiod-ca-cert\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:38:20.890602 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:20.890374 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnmtb\" (UniqueName: \"kubernetes.io/projected/e0991b98-e891-4582-a9ce-88846fcaad92-kube-api-access-wnmtb\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:38:20.890602 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:20.890387 4982 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0991b98-e891-4582-a9ce-88846fcaad92-istio-data\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:38:20.890602 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:20.890401 4982 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0991b98-e891-4582-a9ce-88846fcaad92-workload-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:38:20.890602 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:20.890412 4982 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0991b98-e891-4582-a9ce-88846fcaad92-istio-envoy\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:38:20.890602 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:20.890422 4982 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0991b98-e891-4582-a9ce-88846fcaad92-istio-token\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:38:21.043279 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:21.043250 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w" Oct 30 14:38:21.094541 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:21.094512 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7db57691-2261-4a03-abe3-a0fa7bb45429-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7db57691-2261-4a03-abe3-a0fa7bb45429-istio-envoy\") pod \"7db57691-2261-4a03-abe3-a0fa7bb45429\" (UID: \"7db57691-2261-4a03-abe3-a0fa7bb45429\") " Oct 30 14:38:21.094732 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:21.094554 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7db57691-2261-4a03-abe3-a0fa7bb45429-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7db57691-2261-4a03-abe3-a0fa7bb45429-istio-data\") pod \"7db57691-2261-4a03-abe3-a0fa7bb45429\" (UID: \"7db57691-2261-4a03-abe3-a0fa7bb45429\") " Oct 30 14:38:21.094732 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:21.094601 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7db57691-2261-4a03-abe3-a0fa7bb45429-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7db57691-2261-4a03-abe3-a0fa7bb45429-credential-socket\") pod \"7db57691-2261-4a03-abe3-a0fa7bb45429\" (UID: \"7db57691-2261-4a03-abe3-a0fa7bb45429\") " Oct 30 14:38:21.094732 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:21.094624 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7db57691-2261-4a03-abe3-a0fa7bb45429-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7db57691-2261-4a03-abe3-a0fa7bb45429-workload-socket\") pod \"7db57691-2261-4a03-abe3-a0fa7bb45429\" (UID: \"7db57691-2261-4a03-abe3-a0fa7bb45429\") " Oct 30 14:38:21.094732 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:21.094681 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7db57691-2261-4a03-abe3-a0fa7bb45429-istio-token\" (UniqueName: \"kubernetes.io/projected/7db57691-2261-4a03-abe3-a0fa7bb45429-istio-token\") pod \"7db57691-2261-4a03-abe3-a0fa7bb45429\" (UID: \"7db57691-2261-4a03-abe3-a0fa7bb45429\") " Oct 30 14:38:21.094963 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:21.094751 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7db57691-2261-4a03-abe3-a0fa7bb45429-kube-api-access-kw65m\" (UniqueName: \"kubernetes.io/projected/7db57691-2261-4a03-abe3-a0fa7bb45429-kube-api-access-kw65m\") pod \"7db57691-2261-4a03-abe3-a0fa7bb45429\" (UID: \"7db57691-2261-4a03-abe3-a0fa7bb45429\") " Oct 30 14:38:21.094963 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:21.094840 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7db57691-2261-4a03-abe3-a0fa7bb45429-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7db57691-2261-4a03-abe3-a0fa7bb45429-workload-certs\") pod \"7db57691-2261-4a03-abe3-a0fa7bb45429\" (UID: \"7db57691-2261-4a03-abe3-a0fa7bb45429\") " Oct 30 14:38:21.094963 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:21.094846 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7db57691-2261-4a03-abe3-a0fa7bb45429-istio-data" pod "7db57691-2261-4a03-abe3-a0fa7bb45429" (UID: "7db57691-2261-4a03-abe3-a0fa7bb45429"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:38:21.094963 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:21.094875 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7db57691-2261-4a03-abe3-a0fa7bb45429-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7db57691-2261-4a03-abe3-a0fa7bb45429-istiod-ca-cert\") pod \"7db57691-2261-4a03-abe3-a0fa7bb45429\" (UID: \"7db57691-2261-4a03-abe3-a0fa7bb45429\") " Oct 30 14:38:21.094963 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:21.094933 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7db57691-2261-4a03-abe3-a0fa7bb45429-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7db57691-2261-4a03-abe3-a0fa7bb45429-istio-podinfo\") pod \"7db57691-2261-4a03-abe3-a0fa7bb45429\" (UID: \"7db57691-2261-4a03-abe3-a0fa7bb45429\") " Oct 30 14:38:21.095232 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:21.095126 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7db57691-2261-4a03-abe3-a0fa7bb45429-workload-certs" pod "7db57691-2261-4a03-abe3-a0fa7bb45429" (UID: "7db57691-2261-4a03-abe3-a0fa7bb45429"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:38:21.095799 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:21.095513 4982 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7db57691-2261-4a03-abe3-a0fa7bb45429-workload-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:38:21.095799 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:21.095530 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7db57691-2261-4a03-abe3-a0fa7bb45429-credential-socket" pod "7db57691-2261-4a03-abe3-a0fa7bb45429" (UID: "7db57691-2261-4a03-abe3-a0fa7bb45429"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:38:21.095799 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:21.095542 4982 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7db57691-2261-4a03-abe3-a0fa7bb45429-istio-data\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:38:21.095799 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:21.095687 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7db57691-2261-4a03-abe3-a0fa7bb45429-workload-socket" pod "7db57691-2261-4a03-abe3-a0fa7bb45429" (UID: "7db57691-2261-4a03-abe3-a0fa7bb45429"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:38:21.096254 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:21.096220 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7db57691-2261-4a03-abe3-a0fa7bb45429-istiod-ca-cert" pod "7db57691-2261-4a03-abe3-a0fa7bb45429" (UID: "7db57691-2261-4a03-abe3-a0fa7bb45429"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:38:21.098001 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:21.097965 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7db57691-2261-4a03-abe3-a0fa7bb45429-istio-token" pod "7db57691-2261-4a03-abe3-a0fa7bb45429" (UID: "7db57691-2261-4a03-abe3-a0fa7bb45429"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:38:21.098940 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:21.098915 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7db57691-2261-4a03-abe3-a0fa7bb45429-istio-envoy" pod "7db57691-2261-4a03-abe3-a0fa7bb45429" (UID: "7db57691-2261-4a03-abe3-a0fa7bb45429"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:38:21.099078 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:21.099055 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7db57691-2261-4a03-abe3-a0fa7bb45429-istio-podinfo" pod "7db57691-2261-4a03-abe3-a0fa7bb45429" (UID: "7db57691-2261-4a03-abe3-a0fa7bb45429"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:38:21.099211 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:21.099193 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7db57691-2261-4a03-abe3-a0fa7bb45429-kube-api-access-kw65m" pod "7db57691-2261-4a03-abe3-a0fa7bb45429" (UID: "7db57691-2261-4a03-abe3-a0fa7bb45429"). InnerVolumeSpecName "kube-api-access-kw65m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:38:21.197060 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:21.197026 4982 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7db57691-2261-4a03-abe3-a0fa7bb45429-istio-podinfo\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:38:21.197060 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:21.197053 4982 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7db57691-2261-4a03-abe3-a0fa7bb45429-istio-envoy\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:38:21.197060 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:21.197062 4982 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7db57691-2261-4a03-abe3-a0fa7bb45429-credential-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:38:21.197359 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:21.197070 4982 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7db57691-2261-4a03-abe3-a0fa7bb45429-workload-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:38:21.197359 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:21.197081 4982 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7db57691-2261-4a03-abe3-a0fa7bb45429-istio-token\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:38:21.197359 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:21.197089 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kw65m\" (UniqueName: \"kubernetes.io/projected/7db57691-2261-4a03-abe3-a0fa7bb45429-kube-api-access-kw65m\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:38:21.197359 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:21.197097 4982 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7db57691-2261-4a03-abe3-a0fa7bb45429-istiod-ca-cert\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:38:21.355860 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:21.355823 4982 generic.go:356] "Generic (PLEG): container finished" podID="7db57691-2261-4a03-abe3-a0fa7bb45429" containerID="2a2859a9d0d837ecc3a1e65a024a68b3cc7954ef8cce3bc1b7fb288607668961" exitCode=0 Oct 30 14:38:21.356055 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:21.355869 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w" Oct 30 14:38:21.356055 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:21.355893 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w" event={"ID":"7db57691-2261-4a03-abe3-a0fa7bb45429","Type":"ContainerDied","Data":"2a2859a9d0d837ecc3a1e65a024a68b3cc7954ef8cce3bc1b7fb288607668961"} Oct 30 14:38:21.356055 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:21.355952 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w" event={"ID":"7db57691-2261-4a03-abe3-a0fa7bb45429","Type":"ContainerDied","Data":"aabd41ca0b34aa9ab038e58a9e1238e15df738b4bf34e16910e95056576a5392"} Oct 30 14:38:21.356055 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:21.355978 4982 scope.go:122] "RemoveContainer" containerID="2a2859a9d0d837ecc3a1e65a024a68b3cc7954ef8cce3bc1b7fb288607668961" Oct 30 14:38:21.357712 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:21.357682 4982 generic.go:356] "Generic (PLEG): container finished" podID="e0991b98-e891-4582-a9ce-88846fcaad92" containerID="f50889a5471bfa929aca7f9a76117b150a51172237b53872b17585b70b801543" exitCode=0 Oct 30 14:38:21.357821 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:21.357759 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84" Oct 30 14:38:21.357821 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:21.357770 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84" event={"ID":"e0991b98-e891-4582-a9ce-88846fcaad92","Type":"ContainerDied","Data":"f50889a5471bfa929aca7f9a76117b150a51172237b53872b17585b70b801543"} Oct 30 14:38:21.357909 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:21.357826 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84" event={"ID":"e0991b98-e891-4582-a9ce-88846fcaad92","Type":"ContainerDied","Data":"388eec4e88ceff7d7e268019b0547558f5091644c60dc23897d6420fafe35db8"} Oct 30 14:38:21.368384 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:21.368206 4982 scope.go:122] "RemoveContainer" containerID="2a2859a9d0d837ecc3a1e65a024a68b3cc7954ef8cce3bc1b7fb288607668961" Oct 30 14:38:21.368572 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:38:21.368540 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a2859a9d0d837ecc3a1e65a024a68b3cc7954ef8cce3bc1b7fb288607668961\": container with ID starting with 2a2859a9d0d837ecc3a1e65a024a68b3cc7954ef8cce3bc1b7fb288607668961 not found: ID does not exist" containerID="2a2859a9d0d837ecc3a1e65a024a68b3cc7954ef8cce3bc1b7fb288607668961" Oct 30 14:38:21.368720 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:21.368575 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a2859a9d0d837ecc3a1e65a024a68b3cc7954ef8cce3bc1b7fb288607668961"} err="failed to get container status \"2a2859a9d0d837ecc3a1e65a024a68b3cc7954ef8cce3bc1b7fb288607668961\": rpc error: code = NotFound desc = could not find container \"2a2859a9d0d837ecc3a1e65a024a68b3cc7954ef8cce3bc1b7fb288607668961\": container with ID starting with 2a2859a9d0d837ecc3a1e65a024a68b3cc7954ef8cce3bc1b7fb288607668961 not found: ID does not exist" Oct 30 14:38:21.368720 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:21.368593 4982 scope.go:122] "RemoveContainer" containerID="f50889a5471bfa929aca7f9a76117b150a51172237b53872b17585b70b801543" Oct 30 14:38:21.378373 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:21.378349 4982 scope.go:122] "RemoveContainer" containerID="f50889a5471bfa929aca7f9a76117b150a51172237b53872b17585b70b801543" Oct 30 14:38:21.378750 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:38:21.378717 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f50889a5471bfa929aca7f9a76117b150a51172237b53872b17585b70b801543\": container with ID starting with f50889a5471bfa929aca7f9a76117b150a51172237b53872b17585b70b801543 not found: ID does not exist" containerID="f50889a5471bfa929aca7f9a76117b150a51172237b53872b17585b70b801543" Oct 30 14:38:21.378847 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:21.378760 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f50889a5471bfa929aca7f9a76117b150a51172237b53872b17585b70b801543"} err="failed to get container status \"f50889a5471bfa929aca7f9a76117b150a51172237b53872b17585b70b801543\": rpc error: code = NotFound desc = could not find container \"f50889a5471bfa929aca7f9a76117b150a51172237b53872b17585b70b801543\": container with ID starting with f50889a5471bfa929aca7f9a76117b150a51172237b53872b17585b70b801543 not found: ID does not exist" Oct 30 14:38:21.417392 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:21.417364 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w"] Oct 30 14:38:21.429592 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:21.429561 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-g9h4w"] Oct 30 14:38:21.468474 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:21.468442 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84"] Oct 30 14:38:21.475046 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:21.475017 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-9sr84"] Oct 30 14:38:21.955110 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:21.955079 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:38:22.403037 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:22.403000 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7db57691-2261-4a03-abe3-a0fa7bb45429" path="/var/lib/kubelet/pods/7db57691-2261-4a03-abe3-a0fa7bb45429/volumes" Oct 30 14:38:22.403578 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:22.403559 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0991b98-e891-4582-a9ce-88846fcaad92" path="/var/lib/kubelet/pods/e0991b98-e891-4582-a9ce-88846fcaad92/volumes" Oct 30 14:38:24.014716 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:24.014670 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:38:25.904579 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:25.904541 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:38:27.935968 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:27.935937 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:38:28.012658 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:38:28.012607 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:38:28.332673 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:28.332634 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d3a829-64f6b74c79-6wf64" Oct 30 14:38:29.798255 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:29.798219 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:38:31.833889 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:31.833849 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:38:33.188741 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:33.188702 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:38:33.655720 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:33.655683 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js"] Oct 30 14:38:33.656601 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:33.656572 4982 cpu_manager.go:414] "containerMap: removing container" podUID="7db57691-2261-4a03-abe3-a0fa7bb45429" containerName="istio-proxy" Oct 30 14:38:33.656807 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:33.656789 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7db57691-2261-4a03-abe3-a0fa7bb45429" containerName="istio-proxy" Oct 30 14:38:33.657001 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:33.656987 4982 cpu_manager.go:414] "containerMap: removing container" podUID="e0991b98-e891-4582-a9ce-88846fcaad92" containerName="istio-proxy" Oct 30 14:38:33.657099 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:33.657087 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0991b98-e891-4582-a9ce-88846fcaad92" containerName="istio-proxy" Oct 30 14:38:33.657384 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:33.657365 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0991b98-e891-4582-a9ce-88846fcaad92" containerName="istio-proxy" Oct 30 14:38:33.657501 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:33.657490 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="7db57691-2261-4a03-abe3-a0fa7bb45429" containerName="istio-proxy" Oct 30 14:38:33.663550 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:33.663530 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js" Oct 30 14:38:33.714005 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:33.713974 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-460b0b-openshift-default-dockercfg-2wcbm\"" Oct 30 14:38:33.720784 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:33.720751 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js"] Oct 30 14:38:33.745626 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:33.745600 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:38:33.805257 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:33.805232 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jfc5\" (UniqueName: \"kubernetes.io/projected/463ccf6c-67b5-45b6-970c-ee89fc36a12b-kube-api-access-6jfc5\") pod \"e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js\" (UID: \"463ccf6c-67b5-45b6-970c-ee89fc36a12b\") " pod="openshift-ingress/e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js" Oct 30 14:38:33.805444 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:33.805345 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/463ccf6c-67b5-45b6-970c-ee89fc36a12b-workload-certs\") pod \"e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js\" (UID: \"463ccf6c-67b5-45b6-970c-ee89fc36a12b\") " pod="openshift-ingress/e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js" Oct 30 14:38:33.805444 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:33.805394 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/463ccf6c-67b5-45b6-970c-ee89fc36a12b-istio-envoy\") pod \"e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js\" (UID: \"463ccf6c-67b5-45b6-970c-ee89fc36a12b\") " pod="openshift-ingress/e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js" Oct 30 14:38:33.805444 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:33.805423 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/463ccf6c-67b5-45b6-970c-ee89fc36a12b-istio-data\") pod \"e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js\" (UID: \"463ccf6c-67b5-45b6-970c-ee89fc36a12b\") " pod="openshift-ingress/e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js" Oct 30 14:38:33.805612 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:33.805467 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/463ccf6c-67b5-45b6-970c-ee89fc36a12b-credential-socket\") pod \"e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js\" (UID: \"463ccf6c-67b5-45b6-970c-ee89fc36a12b\") " pod="openshift-ingress/e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js" Oct 30 14:38:33.805612 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:33.805543 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/463ccf6c-67b5-45b6-970c-ee89fc36a12b-istio-token\") pod \"e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js\" (UID: \"463ccf6c-67b5-45b6-970c-ee89fc36a12b\") " pod="openshift-ingress/e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js" Oct 30 14:38:33.805612 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:33.805579 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/463ccf6c-67b5-45b6-970c-ee89fc36a12b-istio-podinfo\") pod \"e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js\" (UID: \"463ccf6c-67b5-45b6-970c-ee89fc36a12b\") " pod="openshift-ingress/e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js" Oct 30 14:38:33.805799 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:33.805613 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/463ccf6c-67b5-45b6-970c-ee89fc36a12b-workload-socket\") pod \"e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js\" (UID: \"463ccf6c-67b5-45b6-970c-ee89fc36a12b\") " pod="openshift-ingress/e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js" Oct 30 14:38:33.805799 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:33.805783 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/463ccf6c-67b5-45b6-970c-ee89fc36a12b-istiod-ca-cert\") pod \"e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js\" (UID: \"463ccf6c-67b5-45b6-970c-ee89fc36a12b\") " pod="openshift-ingress/e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js" Oct 30 14:38:33.896365 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:33.896325 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:38:33.907039 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:33.906953 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/463ccf6c-67b5-45b6-970c-ee89fc36a12b-workload-certs\") pod \"e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js\" (UID: \"463ccf6c-67b5-45b6-970c-ee89fc36a12b\") " pod="openshift-ingress/e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js" Oct 30 14:38:33.907039 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:33.907014 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/463ccf6c-67b5-45b6-970c-ee89fc36a12b-istio-envoy\") pod \"e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js\" (UID: \"463ccf6c-67b5-45b6-970c-ee89fc36a12b\") " pod="openshift-ingress/e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js" Oct 30 14:38:33.907270 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:33.907051 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/463ccf6c-67b5-45b6-970c-ee89fc36a12b-istio-data\") pod \"e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js\" (UID: \"463ccf6c-67b5-45b6-970c-ee89fc36a12b\") " pod="openshift-ingress/e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js" Oct 30 14:38:33.907270 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:33.907078 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/463ccf6c-67b5-45b6-970c-ee89fc36a12b-credential-socket\") pod \"e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js\" (UID: \"463ccf6c-67b5-45b6-970c-ee89fc36a12b\") " pod="openshift-ingress/e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js" Oct 30 14:38:33.907270 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:33.907114 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/463ccf6c-67b5-45b6-970c-ee89fc36a12b-istio-token\") pod \"e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js\" (UID: \"463ccf6c-67b5-45b6-970c-ee89fc36a12b\") " pod="openshift-ingress/e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js" Oct 30 14:38:33.907270 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:33.907157 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/463ccf6c-67b5-45b6-970c-ee89fc36a12b-istio-podinfo\") pod \"e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js\" (UID: \"463ccf6c-67b5-45b6-970c-ee89fc36a12b\") " pod="openshift-ingress/e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js" Oct 30 14:38:33.907270 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:33.907233 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/463ccf6c-67b5-45b6-970c-ee89fc36a12b-workload-socket\") pod \"e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js\" (UID: \"463ccf6c-67b5-45b6-970c-ee89fc36a12b\") " pod="openshift-ingress/e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js" Oct 30 14:38:33.907544 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:33.907341 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/463ccf6c-67b5-45b6-970c-ee89fc36a12b-istiod-ca-cert\") pod \"e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js\" (UID: \"463ccf6c-67b5-45b6-970c-ee89fc36a12b\") " pod="openshift-ingress/e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js" Oct 30 14:38:33.907544 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:33.907417 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/463ccf6c-67b5-45b6-970c-ee89fc36a12b-workload-certs\") pod \"e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js\" (UID: \"463ccf6c-67b5-45b6-970c-ee89fc36a12b\") " pod="openshift-ingress/e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js" Oct 30 14:38:33.907544 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:33.907443 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jfc5\" (UniqueName: \"kubernetes.io/projected/463ccf6c-67b5-45b6-970c-ee89fc36a12b-kube-api-access-6jfc5\") pod \"e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js\" (UID: \"463ccf6c-67b5-45b6-970c-ee89fc36a12b\") " pod="openshift-ingress/e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js" Oct 30 14:38:33.907544 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:33.907481 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/463ccf6c-67b5-45b6-970c-ee89fc36a12b-credential-socket\") pod \"e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js\" (UID: \"463ccf6c-67b5-45b6-970c-ee89fc36a12b\") " pod="openshift-ingress/e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js" Oct 30 14:38:33.907787 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:33.907683 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/463ccf6c-67b5-45b6-970c-ee89fc36a12b-workload-socket\") pod \"e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js\" (UID: \"463ccf6c-67b5-45b6-970c-ee89fc36a12b\") " pod="openshift-ingress/e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js" Oct 30 14:38:33.907905 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:33.907860 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/463ccf6c-67b5-45b6-970c-ee89fc36a12b-istio-data\") pod \"e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js\" (UID: \"463ccf6c-67b5-45b6-970c-ee89fc36a12b\") " pod="openshift-ingress/e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js" Oct 30 14:38:33.908308 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:33.908274 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/463ccf6c-67b5-45b6-970c-ee89fc36a12b-istiod-ca-cert\") pod \"e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js\" (UID: \"463ccf6c-67b5-45b6-970c-ee89fc36a12b\") " pod="openshift-ingress/e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js" Oct 30 14:38:33.910289 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:33.910256 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/463ccf6c-67b5-45b6-970c-ee89fc36a12b-istio-envoy\") pod \"e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js\" (UID: \"463ccf6c-67b5-45b6-970c-ee89fc36a12b\") " pod="openshift-ingress/e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js" Oct 30 14:38:33.911338 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:33.911316 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/463ccf6c-67b5-45b6-970c-ee89fc36a12b-istio-podinfo\") pod \"e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js\" (UID: \"463ccf6c-67b5-45b6-970c-ee89fc36a12b\") " pod="openshift-ingress/e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js" Oct 30 14:38:33.955661 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:33.954365 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/463ccf6c-67b5-45b6-970c-ee89fc36a12b-istio-token\") pod \"e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js\" (UID: \"463ccf6c-67b5-45b6-970c-ee89fc36a12b\") " pod="openshift-ingress/e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js" Oct 30 14:38:33.967663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:33.956270 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jfc5\" (UniqueName: \"kubernetes.io/projected/463ccf6c-67b5-45b6-970c-ee89fc36a12b-kube-api-access-6jfc5\") pod \"e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js\" (UID: \"463ccf6c-67b5-45b6-970c-ee89fc36a12b\") " pod="openshift-ingress/e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js" Oct 30 14:38:34.014711 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:34.013860 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js" Oct 30 14:38:34.053599 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:34.053435 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:38:34.404625 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:34.404569 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js"] Oct 30 14:38:34.405974 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:38:34.405944 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod463ccf6c_67b5_45b6_970c_ee89fc36a12b.slice/crio-1c701fdec2057027418749a81821d572a160a78556de43709cb1d3f0cdf38ae3 WatchSource:0}: Error finding container 1c701fdec2057027418749a81821d572a160a78556de43709cb1d3f0cdf38ae3: Status 404 returned error can't find the container with id 1c701fdec2057027418749a81821d572a160a78556de43709cb1d3f0cdf38ae3 Oct 30 14:38:34.408881 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:34.408844 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:38:34.408986 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:34.408925 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:38:34.408986 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:34.408968 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:38:34.507388 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:34.507350 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js" event={"ID":"463ccf6c-67b5-45b6-970c-ee89fc36a12b","Type":"ContainerStarted","Data":"1c701fdec2057027418749a81821d572a160a78556de43709cb1d3f0cdf38ae3"} Oct 30 14:38:35.385943 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:35.385907 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w6-0efe43/rate-test-model-0efe43-kserve-6f8954844f-d5kvb"] Oct 30 14:38:35.392489 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:35.391280 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-0efe43/rate-test-model-0efe43-kserve-6f8954844f-d5kvb" Oct 30 14:38:35.411437 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:35.411402 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-0efe43/rate-test-model-0efe43-kserve-6f8954844f-d5kvb"] Oct 30 14:38:35.434048 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:35.434016 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-0efe43\"/\"default-dockercfg-sb4ng\"" Oct 30 14:38:35.434701 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:35.434666 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-0efe43\"/\"openshift-service-ca.crt\"" Oct 30 14:38:35.434824 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:35.434770 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-0efe43\"/\"kube-root-ca.crt\"" Oct 30 14:38:35.434934 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:35.434886 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-0efe43\"/\"rate-test-model-0efe43-kserve-self-signed-certs\"" Oct 30 14:38:35.473517 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:35.473484 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:38:35.520048 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:35.520016 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js" event={"ID":"463ccf6c-67b5-45b6-970c-ee89fc36a12b","Type":"ContainerStarted","Data":"a99ebb3920b7a686f4de743c079efb06dc7326eb8abf9ca22b6c033fae601b79"} Oct 30 14:38:35.537610 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:35.537362 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8da898db-f45c-4c84-a91b-15129bfeffc9-home\") pod \"rate-test-model-0efe43-kserve-6f8954844f-d5kvb\" (UID: \"8da898db-f45c-4c84-a91b-15129bfeffc9\") " pod="ai-tenant-e2e-shared-b-w6-0efe43/rate-test-model-0efe43-kserve-6f8954844f-d5kvb" Oct 30 14:38:35.537610 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:35.537456 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cltsx\" (UniqueName: \"kubernetes.io/projected/8da898db-f45c-4c84-a91b-15129bfeffc9-kube-api-access-cltsx\") pod \"rate-test-model-0efe43-kserve-6f8954844f-d5kvb\" (UID: \"8da898db-f45c-4c84-a91b-15129bfeffc9\") " pod="ai-tenant-e2e-shared-b-w6-0efe43/rate-test-model-0efe43-kserve-6f8954844f-d5kvb" Oct 30 14:38:35.537610 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:35.537507 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8da898db-f45c-4c84-a91b-15129bfeffc9-tls-certs\") pod \"rate-test-model-0efe43-kserve-6f8954844f-d5kvb\" (UID: \"8da898db-f45c-4c84-a91b-15129bfeffc9\") " pod="ai-tenant-e2e-shared-b-w6-0efe43/rate-test-model-0efe43-kserve-6f8954844f-d5kvb" Oct 30 14:38:35.538013 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:35.537992 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8da898db-f45c-4c84-a91b-15129bfeffc9-dshm\") pod \"rate-test-model-0efe43-kserve-6f8954844f-d5kvb\" (UID: \"8da898db-f45c-4c84-a91b-15129bfeffc9\") " pod="ai-tenant-e2e-shared-b-w6-0efe43/rate-test-model-0efe43-kserve-6f8954844f-d5kvb" Oct 30 14:38:35.538092 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:35.538047 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8da898db-f45c-4c84-a91b-15129bfeffc9-model-cache\") pod \"rate-test-model-0efe43-kserve-6f8954844f-d5kvb\" (UID: \"8da898db-f45c-4c84-a91b-15129bfeffc9\") " pod="ai-tenant-e2e-shared-b-w6-0efe43/rate-test-model-0efe43-kserve-6f8954844f-d5kvb" Oct 30 14:38:35.538299 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:35.538166 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8da898db-f45c-4c84-a91b-15129bfeffc9-tmp-dir\") pod \"rate-test-model-0efe43-kserve-6f8954844f-d5kvb\" (UID: \"8da898db-f45c-4c84-a91b-15129bfeffc9\") " pod="ai-tenant-e2e-shared-b-w6-0efe43/rate-test-model-0efe43-kserve-6f8954844f-d5kvb" Oct 30 14:38:35.573409 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:35.573365 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js" podStartSLOduration=2.573350574 podStartE2EDuration="2.573350574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:38:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:38:35.568533302 +0000 UTC m=+2985.312667344" watchObservedRunningTime="2026-10-30 14:38:35.573350574 +0000 UTC m=+2985.317484614" Oct 30 14:38:35.639863 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:35.639781 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8da898db-f45c-4c84-a91b-15129bfeffc9-home\") pod \"rate-test-model-0efe43-kserve-6f8954844f-d5kvb\" (UID: \"8da898db-f45c-4c84-a91b-15129bfeffc9\") " pod="ai-tenant-e2e-shared-b-w6-0efe43/rate-test-model-0efe43-kserve-6f8954844f-d5kvb" Oct 30 14:38:35.639863 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:35.639843 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cltsx\" (UniqueName: \"kubernetes.io/projected/8da898db-f45c-4c84-a91b-15129bfeffc9-kube-api-access-cltsx\") pod \"rate-test-model-0efe43-kserve-6f8954844f-d5kvb\" (UID: \"8da898db-f45c-4c84-a91b-15129bfeffc9\") " pod="ai-tenant-e2e-shared-b-w6-0efe43/rate-test-model-0efe43-kserve-6f8954844f-d5kvb" Oct 30 14:38:35.640064 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:35.639887 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8da898db-f45c-4c84-a91b-15129bfeffc9-tls-certs\") pod \"rate-test-model-0efe43-kserve-6f8954844f-d5kvb\" (UID: \"8da898db-f45c-4c84-a91b-15129bfeffc9\") " pod="ai-tenant-e2e-shared-b-w6-0efe43/rate-test-model-0efe43-kserve-6f8954844f-d5kvb" Oct 30 14:38:35.640064 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:35.639952 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8da898db-f45c-4c84-a91b-15129bfeffc9-dshm\") pod \"rate-test-model-0efe43-kserve-6f8954844f-d5kvb\" (UID: \"8da898db-f45c-4c84-a91b-15129bfeffc9\") " pod="ai-tenant-e2e-shared-b-w6-0efe43/rate-test-model-0efe43-kserve-6f8954844f-d5kvb" Oct 30 14:38:35.640064 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:35.639996 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8da898db-f45c-4c84-a91b-15129bfeffc9-model-cache\") pod \"rate-test-model-0efe43-kserve-6f8954844f-d5kvb\" (UID: \"8da898db-f45c-4c84-a91b-15129bfeffc9\") " pod="ai-tenant-e2e-shared-b-w6-0efe43/rate-test-model-0efe43-kserve-6f8954844f-d5kvb" Oct 30 14:38:35.640221 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:35.640079 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8da898db-f45c-4c84-a91b-15129bfeffc9-tmp-dir\") pod \"rate-test-model-0efe43-kserve-6f8954844f-d5kvb\" (UID: \"8da898db-f45c-4c84-a91b-15129bfeffc9\") " pod="ai-tenant-e2e-shared-b-w6-0efe43/rate-test-model-0efe43-kserve-6f8954844f-d5kvb" Oct 30 14:38:35.640272 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:35.640222 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8da898db-f45c-4c84-a91b-15129bfeffc9-home\") pod \"rate-test-model-0efe43-kserve-6f8954844f-d5kvb\" (UID: \"8da898db-f45c-4c84-a91b-15129bfeffc9\") " pod="ai-tenant-e2e-shared-b-w6-0efe43/rate-test-model-0efe43-kserve-6f8954844f-d5kvb" Oct 30 14:38:35.640545 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:35.640518 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8da898db-f45c-4c84-a91b-15129bfeffc9-tmp-dir\") pod \"rate-test-model-0efe43-kserve-6f8954844f-d5kvb\" (UID: \"8da898db-f45c-4c84-a91b-15129bfeffc9\") " pod="ai-tenant-e2e-shared-b-w6-0efe43/rate-test-model-0efe43-kserve-6f8954844f-d5kvb" Oct 30 14:38:35.640738 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:35.640713 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8da898db-f45c-4c84-a91b-15129bfeffc9-model-cache\") pod \"rate-test-model-0efe43-kserve-6f8954844f-d5kvb\" (UID: \"8da898db-f45c-4c84-a91b-15129bfeffc9\") " pod="ai-tenant-e2e-shared-b-w6-0efe43/rate-test-model-0efe43-kserve-6f8954844f-d5kvb" Oct 30 14:38:35.642731 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:35.642707 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8da898db-f45c-4c84-a91b-15129bfeffc9-dshm\") pod \"rate-test-model-0efe43-kserve-6f8954844f-d5kvb\" (UID: \"8da898db-f45c-4c84-a91b-15129bfeffc9\") " pod="ai-tenant-e2e-shared-b-w6-0efe43/rate-test-model-0efe43-kserve-6f8954844f-d5kvb" Oct 30 14:38:35.643485 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:35.643455 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8da898db-f45c-4c84-a91b-15129bfeffc9-tls-certs\") pod \"rate-test-model-0efe43-kserve-6f8954844f-d5kvb\" (UID: \"8da898db-f45c-4c84-a91b-15129bfeffc9\") " pod="ai-tenant-e2e-shared-b-w6-0efe43/rate-test-model-0efe43-kserve-6f8954844f-d5kvb" Oct 30 14:38:35.666602 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:35.666577 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cltsx\" (UniqueName: \"kubernetes.io/projected/8da898db-f45c-4c84-a91b-15129bfeffc9-kube-api-access-cltsx\") pod \"rate-test-model-0efe43-kserve-6f8954844f-d5kvb\" (UID: \"8da898db-f45c-4c84-a91b-15129bfeffc9\") " pod="ai-tenant-e2e-shared-b-w6-0efe43/rate-test-model-0efe43-kserve-6f8954844f-d5kvb" Oct 30 14:38:35.685041 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:35.684927 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:38:35.717663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:35.717124 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-0efe43/rate-test-model-0efe43-kserve-6f8954844f-d5kvb" Oct 30 14:38:36.017716 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:36.017597 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js" Oct 30 14:38:36.024767 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:36.024738 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js" Oct 30 14:38:36.088829 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:36.088798 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-0efe43/rate-test-model-0efe43-kserve-6f8954844f-d5kvb"] Oct 30 14:38:36.103095 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:38:36.103061 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8da898db_f45c_4c84_a91b_15129bfeffc9.slice/crio-fb7b7e36c2a03aafb1f7727eb4d774d7f56c2db4fdad1adf822c71e7efebb333 WatchSource:0}: Error finding container fb7b7e36c2a03aafb1f7727eb4d774d7f56c2db4fdad1adf822c71e7efebb333: Status 404 returned error can't find the container with id fb7b7e36c2a03aafb1f7727eb4d774d7f56c2db4fdad1adf822c71e7efebb333 Oct 30 14:38:36.529336 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:36.529297 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-0efe43/rate-test-model-0efe43-kserve-6f8954844f-d5kvb" event={"ID":"8da898db-f45c-4c84-a91b-15129bfeffc9","Type":"ContainerStarted","Data":"63ea54cacf799cc9c64d7ada08e7660834365319cd484bc26b4d27a11e321d18"} Oct 30 14:38:36.529336 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:36.529348 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-0efe43/rate-test-model-0efe43-kserve-6f8954844f-d5kvb" event={"ID":"8da898db-f45c-4c84-a91b-15129bfeffc9","Type":"ContainerStarted","Data":"fb7b7e36c2a03aafb1f7727eb4d774d7f56c2db4fdad1adf822c71e7efebb333"} Oct 30 14:38:36.529886 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:36.529849 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js" Oct 30 14:38:36.530722 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:36.530696 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js" Oct 30 14:38:36.575630 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:36.575584 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w6-0efe43/rate-test-model-0efe43-kserve-6f8954844f-d5kvb" podStartSLOduration=1.575571879 podStartE2EDuration="1.575571879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:38:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:38:36.573786445 +0000 UTC m=+2986.317920486" watchObservedRunningTime="2026-10-30 14:38:36.575571879 +0000 UTC m=+2986.319705920" Oct 30 14:38:36.718562 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:36.718532 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w6-0efe43/rate-test-model-0efe43-kserve-6f8954844f-d5kvb" Oct 30 14:38:37.647994 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:37.647956 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:38:38.184986 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:38.184922 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:38:38.335891 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:38.335856 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:38:39.667895 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:39.667863 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:38:39.944764 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:39.944673 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:38:40.121998 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:40.121964 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:38:41.653023 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:41.652986 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:38:43.927088 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:43.927049 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:38:45.718239 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:45.718204 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w6-0efe43/rate-test-model-0efe43-kserve-6f8954844f-d5kvb" Oct 30 14:38:45.737943 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:45.737917 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w6-0efe43/rate-test-model-0efe43-kserve-6f8954844f-d5kvb" Oct 30 14:38:45.921614 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:45.921584 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:38:46.405425 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:46.405384 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:38:46.627182 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:46.627148 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w6-0efe43/rate-test-model-0efe43-kserve-6f8954844f-d5kvb" Oct 30 14:38:47.942877 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:47.942832 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:38:49.887427 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:49.887386 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:38:50.636916 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:50.636830 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:38:50.884083 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:50.884053 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ntlqq_dc967875-1546-40bb-8376-fde09416c204/console-operator/2.log" Oct 30 14:38:50.893473 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:50.893390 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ntlqq_dc967875-1546-40bb-8376-fde09416c204/console-operator/2.log" Oct 30 14:38:50.895528 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:50.895504 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhxjf_aca47fe8-7068-4743-b6ab-283ee0874f66/ovn-acl-logging/0.log" Oct 30 14:38:50.900057 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:50.900036 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhxjf_aca47fe8-7068-4743-b6ab-283ee0874f66/ovn-acl-logging/0.log" Oct 30 14:38:51.911563 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:51.911526 4982 scope.go:122] "RemoveContainer" containerID="38f6732e14a1b4e5c8aee94df9a6cf7581b0153baee176dd320c2a411b9a96bd" Oct 30 14:38:51.926137 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:51.926109 4982 scope.go:122] "RemoveContainer" containerID="9d53a51b7e0da7bd7e9529f05caaecebad2daed53dd87b10e5b0be9178678b6a" Oct 30 14:38:51.936940 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:51.936916 4982 scope.go:122] "RemoveContainer" containerID="885d3f04c7e01e2465afbcab73aca890bdda620a71669063acd87a64fcd8693a" Oct 30 14:38:51.946360 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:51.946336 4982 scope.go:122] "RemoveContainer" containerID="51231bc9fc393839d061100ca4affb0e6f7e10185949ad657a05b441cda32c83" Oct 30 14:38:52.039911 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:52.039883 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:38:52.094211 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:52.094158 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf3a515c-pgkfd"] Oct 30 14:38:52.099317 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:52.099286 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3a515c-pgkfd" Oct 30 14:38:52.110076 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:52.110048 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7bca45f-6601-4b89-8f43-6f98b878635f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf3a515c-pgkfd\" (UID: \"e7bca45f-6601-4b89-8f43-6f98b878635f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3a515c-pgkfd" Oct 30 14:38:52.110219 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:52.110092 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7bca45f-6601-4b89-8f43-6f98b878635f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf3a515c-pgkfd\" (UID: \"e7bca45f-6601-4b89-8f43-6f98b878635f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3a515c-pgkfd" Oct 30 14:38:52.110219 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:52.110169 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m9mw\" (UniqueName: \"kubernetes.io/projected/e7bca45f-6601-4b89-8f43-6f98b878635f-kube-api-access-2m9mw\") pod \"maas-api-revoke-sub-cf3a515c-pgkfd\" (UID: \"e7bca45f-6601-4b89-8f43-6f98b878635f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3a515c-pgkfd" Oct 30 14:38:52.119796 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:52.119769 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf3a515c-pgkfd"] Oct 30 14:38:52.129698 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:52.128911 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:38:52.129698 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:52.129048 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:38:52.211233 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:52.211158 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7bca45f-6601-4b89-8f43-6f98b878635f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf3a515c-pgkfd\" (UID: \"e7bca45f-6601-4b89-8f43-6f98b878635f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3a515c-pgkfd" Oct 30 14:38:52.211233 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:52.211226 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7bca45f-6601-4b89-8f43-6f98b878635f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf3a515c-pgkfd\" (UID: \"e7bca45f-6601-4b89-8f43-6f98b878635f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3a515c-pgkfd" Oct 30 14:38:52.211470 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:52.211368 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m9mw\" (UniqueName: \"kubernetes.io/projected/e7bca45f-6601-4b89-8f43-6f98b878635f-kube-api-access-2m9mw\") pod \"maas-api-revoke-sub-cf3a515c-pgkfd\" (UID: \"e7bca45f-6601-4b89-8f43-6f98b878635f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3a515c-pgkfd" Oct 30 14:38:52.211991 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:52.211955 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7bca45f-6601-4b89-8f43-6f98b878635f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf3a515c-pgkfd\" (UID: \"e7bca45f-6601-4b89-8f43-6f98b878635f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3a515c-pgkfd" Oct 30 14:38:52.213963 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:52.213937 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7bca45f-6601-4b89-8f43-6f98b878635f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf3a515c-pgkfd\" (UID: \"e7bca45f-6601-4b89-8f43-6f98b878635f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3a515c-pgkfd" Oct 30 14:38:52.241062 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:52.241032 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m9mw\" (UniqueName: \"kubernetes.io/projected/e7bca45f-6601-4b89-8f43-6f98b878635f-kube-api-access-2m9mw\") pod \"maas-api-revoke-sub-cf3a515c-pgkfd\" (UID: \"e7bca45f-6601-4b89-8f43-6f98b878635f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3a515c-pgkfd" Oct 30 14:38:52.417024 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:52.416998 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3a515c-pgkfd" Oct 30 14:38:52.740291 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:52.740259 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf3a515c-pgkfd"] Oct 30 14:38:52.740904 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:38:52.740877 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7bca45f_6601_4b89_8f43_6f98b878635f.slice/crio-1d835efe95830b10ccd56756804d901c8d9d5a20a5b47a27a3fd6e5f74138e2e WatchSource:0}: Error finding container 1d835efe95830b10ccd56756804d901c8d9d5a20a5b47a27a3fd6e5f74138e2e: Status 404 returned error can't find the container with id 1d835efe95830b10ccd56756804d901c8d9d5a20a5b47a27a3fd6e5f74138e2e Oct 30 14:38:53.713708 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:53.713673 4982 generic.go:356] "Generic (PLEG): container finished" podID="e7bca45f-6601-4b89-8f43-6f98b878635f" containerID="694e5a20aa97cc087183432e4154630dd6de03600ff35f771b06d868e7da05b0" exitCode=0 Oct 30 14:38:53.714142 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:53.713759 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3a515c-pgkfd" event={"ID":"e7bca45f-6601-4b89-8f43-6f98b878635f","Type":"ContainerDied","Data":"694e5a20aa97cc087183432e4154630dd6de03600ff35f771b06d868e7da05b0"} Oct 30 14:38:53.714142 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:53.713815 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3a515c-pgkfd" event={"ID":"e7bca45f-6601-4b89-8f43-6f98b878635f","Type":"ContainerStarted","Data":"1d835efe95830b10ccd56756804d901c8d9d5a20a5b47a27a3fd6e5f74138e2e"} Oct 30 14:38:54.487134 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:54.487100 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:38:54.980477 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:54.980445 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3a515c-pgkfd" Oct 30 14:38:55.046770 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:55.046726 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7bca45f-6601-4b89-8f43-6f98b878635f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7bca45f-6601-4b89-8f43-6f98b878635f-maas-api-xks-ca\") pod \"e7bca45f-6601-4b89-8f43-6f98b878635f\" (UID: \"e7bca45f-6601-4b89-8f43-6f98b878635f\") " Oct 30 14:38:55.046967 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:55.046845 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7bca45f-6601-4b89-8f43-6f98b878635f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7bca45f-6601-4b89-8f43-6f98b878635f-maas-api-openshift-ca\") pod \"e7bca45f-6601-4b89-8f43-6f98b878635f\" (UID: \"e7bca45f-6601-4b89-8f43-6f98b878635f\") " Oct 30 14:38:55.046967 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:55.046910 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7bca45f-6601-4b89-8f43-6f98b878635f-kube-api-access-2m9mw\" (UniqueName: \"kubernetes.io/projected/e7bca45f-6601-4b89-8f43-6f98b878635f-kube-api-access-2m9mw\") pod \"e7bca45f-6601-4b89-8f43-6f98b878635f\" (UID: \"e7bca45f-6601-4b89-8f43-6f98b878635f\") " Oct 30 14:38:55.047606 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:55.047571 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7bca45f-6601-4b89-8f43-6f98b878635f-maas-api-openshift-ca" pod "e7bca45f-6601-4b89-8f43-6f98b878635f" (UID: "e7bca45f-6601-4b89-8f43-6f98b878635f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:38:55.049945 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:55.049911 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7bca45f-6601-4b89-8f43-6f98b878635f-kube-api-access-2m9mw" pod "e7bca45f-6601-4b89-8f43-6f98b878635f" (UID: "e7bca45f-6601-4b89-8f43-6f98b878635f"). InnerVolumeSpecName "kube-api-access-2m9mw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:38:55.055902 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:55.055877 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7bca45f-6601-4b89-8f43-6f98b878635f-maas-api-xks-ca" pod "e7bca45f-6601-4b89-8f43-6f98b878635f" (UID: "e7bca45f-6601-4b89-8f43-6f98b878635f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:38:55.148712 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:55.148601 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7bca45f-6601-4b89-8f43-6f98b878635f-maas-api-xks-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:38:55.148712 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:55.148635 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7bca45f-6601-4b89-8f43-6f98b878635f-maas-api-openshift-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:38:55.148712 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:55.148663 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2m9mw\" (UniqueName: \"kubernetes.io/projected/e7bca45f-6601-4b89-8f43-6f98b878635f-kube-api-access-2m9mw\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:38:55.723361 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:55.723321 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:38:55.732948 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:55.732917 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3a515c-pgkfd" Oct 30 14:38:55.733106 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:55.732943 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3a515c-pgkfd" event={"ID":"e7bca45f-6601-4b89-8f43-6f98b878635f","Type":"ContainerDied","Data":"1d835efe95830b10ccd56756804d901c8d9d5a20a5b47a27a3fd6e5f74138e2e"} Oct 30 14:38:55.733106 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:55.733000 4982 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d835efe95830b10ccd56756804d901c8d9d5a20a5b47a27a3fd6e5f74138e2e" Oct 30 14:38:56.557302 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:56.557265 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:38:57.325357 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:57.325324 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83df63cb-cr85q"] Oct 30 14:38:57.325913 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:57.325898 4982 cpu_manager.go:414] "containerMap: removing container" podUID="e7bca45f-6601-4b89-8f43-6f98b878635f" containerName="revoke-keys" Oct 30 14:38:57.325969 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:57.325914 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7bca45f-6601-4b89-8f43-6f98b878635f" containerName="revoke-keys" Oct 30 14:38:57.326064 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:57.326055 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7bca45f-6601-4b89-8f43-6f98b878635f" containerName="revoke-keys" Oct 30 14:38:57.329742 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:57.329725 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83df63cb-cr85q" Oct 30 14:38:57.347048 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:57.347026 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:38:57.350621 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:57.350597 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83df63cb-cr85q"] Oct 30 14:38:57.359272 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:57.359248 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:38:57.380230 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:57.380195 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e49340e-9383-4d23-a3b6-4128f67f086d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83df63cb-cr85q\" (UID: \"6e49340e-9383-4d23-a3b6-4128f67f086d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83df63cb-cr85q" Oct 30 14:38:57.380395 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:57.380315 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44zb8\" (UniqueName: \"kubernetes.io/projected/6e49340e-9383-4d23-a3b6-4128f67f086d-kube-api-access-44zb8\") pod \"maas-api-revoke-sub-83df63cb-cr85q\" (UID: \"6e49340e-9383-4d23-a3b6-4128f67f086d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83df63cb-cr85q" Oct 30 14:38:57.380470 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:57.380441 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e49340e-9383-4d23-a3b6-4128f67f086d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83df63cb-cr85q\" (UID: \"6e49340e-9383-4d23-a3b6-4128f67f086d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83df63cb-cr85q" Oct 30 14:38:57.481879 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:57.481852 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44zb8\" (UniqueName: \"kubernetes.io/projected/6e49340e-9383-4d23-a3b6-4128f67f086d-kube-api-access-44zb8\") pod \"maas-api-revoke-sub-83df63cb-cr85q\" (UID: \"6e49340e-9383-4d23-a3b6-4128f67f086d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83df63cb-cr85q" Oct 30 14:38:57.482033 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:57.481907 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e49340e-9383-4d23-a3b6-4128f67f086d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83df63cb-cr85q\" (UID: \"6e49340e-9383-4d23-a3b6-4128f67f086d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83df63cb-cr85q" Oct 30 14:38:57.482099 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:57.482084 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e49340e-9383-4d23-a3b6-4128f67f086d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83df63cb-cr85q\" (UID: \"6e49340e-9383-4d23-a3b6-4128f67f086d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83df63cb-cr85q" Oct 30 14:38:57.482763 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:57.482733 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e49340e-9383-4d23-a3b6-4128f67f086d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83df63cb-cr85q\" (UID: \"6e49340e-9383-4d23-a3b6-4128f67f086d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83df63cb-cr85q" Oct 30 14:38:57.485304 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:57.485281 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e49340e-9383-4d23-a3b6-4128f67f086d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83df63cb-cr85q\" (UID: \"6e49340e-9383-4d23-a3b6-4128f67f086d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83df63cb-cr85q" Oct 30 14:38:57.507918 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:57.507889 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44zb8\" (UniqueName: \"kubernetes.io/projected/6e49340e-9383-4d23-a3b6-4128f67f086d-kube-api-access-44zb8\") pod \"maas-api-revoke-sub-83df63cb-cr85q\" (UID: \"6e49340e-9383-4d23-a3b6-4128f67f086d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83df63cb-cr85q" Oct 30 14:38:57.642582 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:57.642513 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83df63cb-cr85q" Oct 30 14:38:57.950610 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:57.950527 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83df63cb-cr85q"] Oct 30 14:38:57.954765 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:38:57.954726 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e49340e_9383_4d23_a3b6_4128f67f086d.slice/crio-6d608ddba073b23f145803269099cf9c070b9d40de2958bcaca8e93292fc7fb4 WatchSource:0}: Error finding container 6d608ddba073b23f145803269099cf9c070b9d40de2958bcaca8e93292fc7fb4: Status 404 returned error can't find the container with id 6d608ddba073b23f145803269099cf9c070b9d40de2958bcaca8e93292fc7fb4 Oct 30 14:38:58.011733 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:38:58.011699 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:38:58.443102 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:58.443072 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:38:58.776314 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:58.776264 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83df63cb-cr85q" event={"ID":"6e49340e-9383-4d23-a3b6-4128f67f086d","Type":"ContainerStarted","Data":"762f1200fbcebeb18a5affddc0234cae8baaefedcb4b1dc5ef71bfa543b5cd25"} Oct 30 14:38:58.776752 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:58.776327 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83df63cb-cr85q" event={"ID":"6e49340e-9383-4d23-a3b6-4128f67f086d","Type":"ContainerStarted","Data":"6d608ddba073b23f145803269099cf9c070b9d40de2958bcaca8e93292fc7fb4"} Oct 30 14:38:58.823839 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:58.823790 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83df63cb-cr85q" podStartSLOduration=1.823776088 podStartE2EDuration="1.823776088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:38:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:38:58.821239783 +0000 UTC m=+3008.565373824" watchObservedRunningTime="2026-10-30 14:38:58.823776088 +0000 UTC m=+3008.567910128" Oct 30 14:38:59.785688 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:59.785627 4982 generic.go:356] "Generic (PLEG): container finished" podID="6e49340e-9383-4d23-a3b6-4128f67f086d" containerID="762f1200fbcebeb18a5affddc0234cae8baaefedcb4b1dc5ef71bfa543b5cd25" exitCode=0 Oct 30 14:38:59.786284 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:59.785710 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83df63cb-cr85q" event={"ID":"6e49340e-9383-4d23-a3b6-4128f67f086d","Type":"ContainerDied","Data":"762f1200fbcebeb18a5affddc0234cae8baaefedcb4b1dc5ef71bfa543b5cd25"} Oct 30 14:38:59.875523 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:59.875479 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-f1f206/auth-test-model-f1f206-kserve-74c48d6ddf-wx5tn"] Oct 30 14:38:59.962552 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:38:59.962519 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:39:00.407751 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:00.407703 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:39:00.573783 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:00.573724 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:39:00.978646 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:00.978621 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83df63cb-cr85q" Oct 30 14:39:01.044708 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:01.040791 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e49340e-9383-4d23-a3b6-4128f67f086d-kube-api-access-44zb8\" (UniqueName: \"kubernetes.io/projected/6e49340e-9383-4d23-a3b6-4128f67f086d-kube-api-access-44zb8\") pod \"6e49340e-9383-4d23-a3b6-4128f67f086d\" (UID: \"6e49340e-9383-4d23-a3b6-4128f67f086d\") " Oct 30 14:39:01.044708 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:01.040977 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e49340e-9383-4d23-a3b6-4128f67f086d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e49340e-9383-4d23-a3b6-4128f67f086d-maas-api-openshift-ca\") pod \"6e49340e-9383-4d23-a3b6-4128f67f086d\" (UID: \"6e49340e-9383-4d23-a3b6-4128f67f086d\") " Oct 30 14:39:01.044708 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:01.041050 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e49340e-9383-4d23-a3b6-4128f67f086d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e49340e-9383-4d23-a3b6-4128f67f086d-maas-api-xks-ca\") pod \"6e49340e-9383-4d23-a3b6-4128f67f086d\" (UID: \"6e49340e-9383-4d23-a3b6-4128f67f086d\") " Oct 30 14:39:01.057660 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:01.049757 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e49340e-9383-4d23-a3b6-4128f67f086d-maas-api-openshift-ca" pod "6e49340e-9383-4d23-a3b6-4128f67f086d" (UID: "6e49340e-9383-4d23-a3b6-4128f67f086d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:39:01.065755 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:01.064606 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e49340e-9383-4d23-a3b6-4128f67f086d-kube-api-access-44zb8" pod "6e49340e-9383-4d23-a3b6-4128f67f086d" (UID: "6e49340e-9383-4d23-a3b6-4128f67f086d"). InnerVolumeSpecName "kube-api-access-44zb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:39:01.092659 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:01.077610 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e49340e-9383-4d23-a3b6-4128f67f086d-maas-api-xks-ca" pod "6e49340e-9383-4d23-a3b6-4128f67f086d" (UID: "6e49340e-9383-4d23-a3b6-4128f67f086d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:39:01.142823 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:01.142731 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e49340e-9383-4d23-a3b6-4128f67f086d-maas-api-openshift-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:39:01.142823 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:01.142770 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e49340e-9383-4d23-a3b6-4128f67f086d-maas-api-xks-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:39:01.142823 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:01.142785 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44zb8\" (UniqueName: \"kubernetes.io/projected/6e49340e-9383-4d23-a3b6-4128f67f086d-kube-api-access-44zb8\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:39:01.382727 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:01.382689 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:39:01.604480 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:01.604440 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:39:01.802360 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:01.802273 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83df63cb-cr85q" event={"ID":"6e49340e-9383-4d23-a3b6-4128f67f086d","Type":"ContainerDied","Data":"6d608ddba073b23f145803269099cf9c070b9d40de2958bcaca8e93292fc7fb4"} Oct 30 14:39:01.802618 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:01.802309 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83df63cb-cr85q" Oct 30 14:39:01.802759 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:01.802617 4982 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d608ddba073b23f145803269099cf9c070b9d40de2958bcaca8e93292fc7fb4" Oct 30 14:39:02.493534 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:02.493499 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce8ddfab-wnzjv"] Oct 30 14:39:02.494423 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:02.494402 4982 cpu_manager.go:414] "containerMap: removing container" podUID="6e49340e-9383-4d23-a3b6-4128f67f086d" containerName="revoke-keys" Oct 30 14:39:02.494480 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:02.494427 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e49340e-9383-4d23-a3b6-4128f67f086d" containerName="revoke-keys" Oct 30 14:39:02.494599 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:02.494587 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e49340e-9383-4d23-a3b6-4128f67f086d" containerName="revoke-keys" Oct 30 14:39:02.498982 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:02.498960 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8ddfab-wnzjv" Oct 30 14:39:02.518933 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:02.518897 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce8ddfab-wnzjv"] Oct 30 14:39:02.526864 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:02.526842 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:39:02.527024 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:02.526899 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:39:02.565453 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:02.565424 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0625c6bf-319a-4be3-84fb-d9fb16127f96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce8ddfab-wnzjv\" (UID: \"0625c6bf-319a-4be3-84fb-d9fb16127f96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8ddfab-wnzjv" Oct 30 14:39:02.565674 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:02.565651 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2xpx\" (UniqueName: \"kubernetes.io/projected/0625c6bf-319a-4be3-84fb-d9fb16127f96-kube-api-access-w2xpx\") pod \"maas-api-revoke-sub-ce8ddfab-wnzjv\" (UID: \"0625c6bf-319a-4be3-84fb-d9fb16127f96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8ddfab-wnzjv" Oct 30 14:39:02.565802 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:02.565778 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0625c6bf-319a-4be3-84fb-d9fb16127f96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce8ddfab-wnzjv\" (UID: \"0625c6bf-319a-4be3-84fb-d9fb16127f96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8ddfab-wnzjv" Oct 30 14:39:02.667153 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:02.667111 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2xpx\" (UniqueName: \"kubernetes.io/projected/0625c6bf-319a-4be3-84fb-d9fb16127f96-kube-api-access-w2xpx\") pod \"maas-api-revoke-sub-ce8ddfab-wnzjv\" (UID: \"0625c6bf-319a-4be3-84fb-d9fb16127f96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8ddfab-wnzjv" Oct 30 14:39:02.667495 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:02.667464 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0625c6bf-319a-4be3-84fb-d9fb16127f96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce8ddfab-wnzjv\" (UID: \"0625c6bf-319a-4be3-84fb-d9fb16127f96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8ddfab-wnzjv" Oct 30 14:39:02.667651 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:02.667619 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0625c6bf-319a-4be3-84fb-d9fb16127f96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce8ddfab-wnzjv\" (UID: \"0625c6bf-319a-4be3-84fb-d9fb16127f96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8ddfab-wnzjv" Oct 30 14:39:02.668321 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:02.668298 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0625c6bf-319a-4be3-84fb-d9fb16127f96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce8ddfab-wnzjv\" (UID: \"0625c6bf-319a-4be3-84fb-d9fb16127f96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8ddfab-wnzjv" Oct 30 14:39:02.670167 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:02.670147 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0625c6bf-319a-4be3-84fb-d9fb16127f96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce8ddfab-wnzjv\" (UID: \"0625c6bf-319a-4be3-84fb-d9fb16127f96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8ddfab-wnzjv" Oct 30 14:39:02.697287 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:02.697254 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2xpx\" (UniqueName: \"kubernetes.io/projected/0625c6bf-319a-4be3-84fb-d9fb16127f96-kube-api-access-w2xpx\") pod \"maas-api-revoke-sub-ce8ddfab-wnzjv\" (UID: \"0625c6bf-319a-4be3-84fb-d9fb16127f96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8ddfab-wnzjv" Oct 30 14:39:02.815793 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:02.815749 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8ddfab-wnzjv" Oct 30 14:39:03.143083 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:03.143050 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce8ddfab-wnzjv"] Oct 30 14:39:03.144203 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:39:03.144175 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0625c6bf_319a_4be3_84fb_d9fb16127f96.slice/crio-9edf1d6fc744393be735c199f162cc59505a5c9b5fd97ac92519691b0eab26ac WatchSource:0}: Error finding container 9edf1d6fc744393be735c199f162cc59505a5c9b5fd97ac92519691b0eab26ac: Status 404 returned error can't find the container with id 9edf1d6fc744393be735c199f162cc59505a5c9b5fd97ac92519691b0eab26ac Oct 30 14:39:03.264891 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:03.264852 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:39:03.820828 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:03.820782 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8ddfab-wnzjv" event={"ID":"0625c6bf-319a-4be3-84fb-d9fb16127f96","Type":"ContainerStarted","Data":"6d21eb8adac7bad5d0d7babbab2c00a7366aed244a2dd3fa658c571a32768d2b"} Oct 30 14:39:03.821235 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:03.820864 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8ddfab-wnzjv" event={"ID":"0625c6bf-319a-4be3-84fb-d9fb16127f96","Type":"ContainerStarted","Data":"9edf1d6fc744393be735c199f162cc59505a5c9b5fd97ac92519691b0eab26ac"} Oct 30 14:39:03.867466 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:03.867417 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8ddfab-wnzjv" podStartSLOduration=1.867405258 podStartE2EDuration="1.867405258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:39:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:39:03.86429593 +0000 UTC m=+3013.608429970" watchObservedRunningTime="2026-10-30 14:39:03.867405258 +0000 UTC m=+3013.611539298" Oct 30 14:39:04.830464 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:04.830430 4982 generic.go:356] "Generic (PLEG): container finished" podID="0625c6bf-319a-4be3-84fb-d9fb16127f96" containerID="6d21eb8adac7bad5d0d7babbab2c00a7366aed244a2dd3fa658c571a32768d2b" exitCode=0 Oct 30 14:39:04.830902 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:04.830477 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8ddfab-wnzjv" event={"ID":"0625c6bf-319a-4be3-84fb-d9fb16127f96","Type":"ContainerDied","Data":"6d21eb8adac7bad5d0d7babbab2c00a7366aed244a2dd3fa658c571a32768d2b"} Oct 30 14:39:04.949482 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:04.949444 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:39:05.103335 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:05.103242 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:39:05.452191 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:05.452117 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0e8b9-748c768747-vlzmg"] Oct 30 14:39:05.456365 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:05.456347 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0e8b9-748c768747-vlzmg" Oct 30 14:39:05.484572 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:05.484539 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-f0e8b9-serving-cert\"" Oct 30 14:39:05.497447 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:05.497410 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0e8b9-748c768747-vlzmg"] Oct 30 14:39:05.598018 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:05.597983 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:39:05.614852 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:05.614819 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkp8s\" (UniqueName: \"kubernetes.io/projected/738cda0d-b92c-4e95-8b44-bb296e7b68dd-kube-api-access-gkp8s\") pod \"maas-api-e2e-ipp-a-f0e8b9-748c768747-vlzmg\" (UID: \"738cda0d-b92c-4e95-8b44-bb296e7b68dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0e8b9-748c768747-vlzmg" Oct 30 14:39:05.615034 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:05.614896 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/738cda0d-b92c-4e95-8b44-bb296e7b68dd-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f0e8b9-748c768747-vlzmg\" (UID: \"738cda0d-b92c-4e95-8b44-bb296e7b68dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0e8b9-748c768747-vlzmg" Oct 30 14:39:05.615034 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:05.615024 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/738cda0d-b92c-4e95-8b44-bb296e7b68dd-metrics-certs\") pod \"maas-api-e2e-ipp-a-f0e8b9-748c768747-vlzmg\" (UID: \"738cda0d-b92c-4e95-8b44-bb296e7b68dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0e8b9-748c768747-vlzmg" Oct 30 14:39:05.615137 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:05.615087 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/738cda0d-b92c-4e95-8b44-bb296e7b68dd-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f0e8b9-748c768747-vlzmg\" (UID: \"738cda0d-b92c-4e95-8b44-bb296e7b68dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0e8b9-748c768747-vlzmg" Oct 30 14:39:05.717227 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:05.717141 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/738cda0d-b92c-4e95-8b44-bb296e7b68dd-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f0e8b9-748c768747-vlzmg\" (UID: \"738cda0d-b92c-4e95-8b44-bb296e7b68dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0e8b9-748c768747-vlzmg" Oct 30 14:39:05.717368 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:05.717285 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkp8s\" (UniqueName: \"kubernetes.io/projected/738cda0d-b92c-4e95-8b44-bb296e7b68dd-kube-api-access-gkp8s\") pod \"maas-api-e2e-ipp-a-f0e8b9-748c768747-vlzmg\" (UID: \"738cda0d-b92c-4e95-8b44-bb296e7b68dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0e8b9-748c768747-vlzmg" Oct 30 14:39:05.717436 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:05.717375 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/738cda0d-b92c-4e95-8b44-bb296e7b68dd-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f0e8b9-748c768747-vlzmg\" (UID: \"738cda0d-b92c-4e95-8b44-bb296e7b68dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0e8b9-748c768747-vlzmg" Oct 30 14:39:05.717519 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:05.717499 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/738cda0d-b92c-4e95-8b44-bb296e7b68dd-metrics-certs\") pod \"maas-api-e2e-ipp-a-f0e8b9-748c768747-vlzmg\" (UID: \"738cda0d-b92c-4e95-8b44-bb296e7b68dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0e8b9-748c768747-vlzmg" Oct 30 14:39:05.718279 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:05.718243 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/738cda0d-b92c-4e95-8b44-bb296e7b68dd-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f0e8b9-748c768747-vlzmg\" (UID: \"738cda0d-b92c-4e95-8b44-bb296e7b68dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0e8b9-748c768747-vlzmg" Oct 30 14:39:05.726652 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:05.720872 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/738cda0d-b92c-4e95-8b44-bb296e7b68dd-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f0e8b9-748c768747-vlzmg\" (UID: \"738cda0d-b92c-4e95-8b44-bb296e7b68dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0e8b9-748c768747-vlzmg" Oct 30 14:39:05.726652 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:05.720919 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/738cda0d-b92c-4e95-8b44-bb296e7b68dd-metrics-certs\") pod \"maas-api-e2e-ipp-a-f0e8b9-748c768747-vlzmg\" (UID: \"738cda0d-b92c-4e95-8b44-bb296e7b68dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0e8b9-748c768747-vlzmg" Oct 30 14:39:05.743949 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:05.743922 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkp8s\" (UniqueName: \"kubernetes.io/projected/738cda0d-b92c-4e95-8b44-bb296e7b68dd-kube-api-access-gkp8s\") pod \"maas-api-e2e-ipp-a-f0e8b9-748c768747-vlzmg\" (UID: \"738cda0d-b92c-4e95-8b44-bb296e7b68dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0e8b9-748c768747-vlzmg" Oct 30 14:39:05.769737 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:05.769710 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0e8b9-748c768747-vlzmg" Oct 30 14:39:05.785280 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:05.785252 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:39:06.009946 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:06.009919 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8ddfab-wnzjv" Oct 30 14:39:06.103583 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:06.103556 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0e8b9-748c768747-vlzmg"] Oct 30 14:39:06.105531 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:39:06.105504 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod738cda0d_b92c_4e95_8b44_bb296e7b68dd.slice/crio-2b944bc3ef15e6b44b895dea611e9ac811e513fd51018777134aabe71a55249e WatchSource:0}: Error finding container 2b944bc3ef15e6b44b895dea611e9ac811e513fd51018777134aabe71a55249e: Status 404 returned error can't find the container with id 2b944bc3ef15e6b44b895dea611e9ac811e513fd51018777134aabe71a55249e Oct 30 14:39:06.122161 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:06.122138 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0625c6bf-319a-4be3-84fb-d9fb16127f96-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0625c6bf-319a-4be3-84fb-d9fb16127f96-maas-api-openshift-ca\") pod \"0625c6bf-319a-4be3-84fb-d9fb16127f96\" (UID: \"0625c6bf-319a-4be3-84fb-d9fb16127f96\") " Oct 30 14:39:06.122278 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:06.122243 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0625c6bf-319a-4be3-84fb-d9fb16127f96-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0625c6bf-319a-4be3-84fb-d9fb16127f96-maas-api-xks-ca\") pod \"0625c6bf-319a-4be3-84fb-d9fb16127f96\" (UID: \"0625c6bf-319a-4be3-84fb-d9fb16127f96\") " Oct 30 14:39:06.122480 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:06.122459 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0625c6bf-319a-4be3-84fb-d9fb16127f96-kube-api-access-w2xpx\" (UniqueName: \"kubernetes.io/projected/0625c6bf-319a-4be3-84fb-d9fb16127f96-kube-api-access-w2xpx\") pod \"0625c6bf-319a-4be3-84fb-d9fb16127f96\" (UID: \"0625c6bf-319a-4be3-84fb-d9fb16127f96\") " Oct 30 14:39:06.122572 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:06.122525 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0625c6bf-319a-4be3-84fb-d9fb16127f96-maas-api-openshift-ca" pod "0625c6bf-319a-4be3-84fb-d9fb16127f96" (UID: "0625c6bf-319a-4be3-84fb-d9fb16127f96"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:39:06.123009 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:06.122987 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0625c6bf-319a-4be3-84fb-d9fb16127f96-maas-api-openshift-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:39:06.124372 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:06.124324 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0625c6bf-319a-4be3-84fb-d9fb16127f96-maas-api-xks-ca" pod "0625c6bf-319a-4be3-84fb-d9fb16127f96" (UID: "0625c6bf-319a-4be3-84fb-d9fb16127f96"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:39:06.124892 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:06.124869 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0625c6bf-319a-4be3-84fb-d9fb16127f96-kube-api-access-w2xpx" pod "0625c6bf-319a-4be3-84fb-d9fb16127f96" (UID: "0625c6bf-319a-4be3-84fb-d9fb16127f96"). InnerVolumeSpecName "kube-api-access-w2xpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:39:06.224624 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:06.224594 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0625c6bf-319a-4be3-84fb-d9fb16127f96-maas-api-xks-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:39:06.224624 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:06.224624 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2xpx\" (UniqueName: \"kubernetes.io/projected/0625c6bf-319a-4be3-84fb-d9fb16127f96-kube-api-access-w2xpx\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:39:06.686300 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:06.686185 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:39:06.849519 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:06.849477 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-33c209-bffd664d7-kcz62"] Oct 30 14:39:06.851053 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:06.850973 4982 cpu_manager.go:414] "containerMap: removing container" podUID="0625c6bf-319a-4be3-84fb-d9fb16127f96" containerName="revoke-keys" Oct 30 14:39:06.851053 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:06.851000 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0625c6bf-319a-4be3-84fb-d9fb16127f96" containerName="revoke-keys" Oct 30 14:39:06.851254 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:06.851146 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="0625c6bf-319a-4be3-84fb-d9fb16127f96" containerName="revoke-keys" Oct 30 14:39:06.855707 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:06.855427 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c209-bffd664d7-kcz62" Oct 30 14:39:06.857271 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:06.857054 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8ddfab-wnzjv" Oct 30 14:39:06.857271 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:06.857175 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8ddfab-wnzjv" event={"ID":"0625c6bf-319a-4be3-84fb-d9fb16127f96","Type":"ContainerDied","Data":"9edf1d6fc744393be735c199f162cc59505a5c9b5fd97ac92519691b0eab26ac"} Oct 30 14:39:06.857271 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:06.857227 4982 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9edf1d6fc744393be735c199f162cc59505a5c9b5fd97ac92519691b0eab26ac" Oct 30 14:39:06.859794 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:06.859751 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0e8b9-748c768747-vlzmg" event={"ID":"738cda0d-b92c-4e95-8b44-bb296e7b68dd","Type":"ContainerStarted","Data":"9489a52983ba107be28f0dc06b54946ba7e56cbd208a478420f0a094f5d0c2d2"} Oct 30 14:39:06.859918 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:06.859804 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0e8b9-748c768747-vlzmg" event={"ID":"738cda0d-b92c-4e95-8b44-bb296e7b68dd","Type":"ContainerStarted","Data":"2b944bc3ef15e6b44b895dea611e9ac811e513fd51018777134aabe71a55249e"} Oct 30 14:39:06.860516 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:06.860496 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0e8b9-748c768747-vlzmg" Oct 30 14:39:06.861515 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:06.861493 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:39:06.873609 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:06.873586 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-33c209-bffd664d7-kcz62"] Oct 30 14:39:06.895665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:06.891867 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-33c209\"" Oct 30 14:39:06.895665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:06.891940 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-33c209-dockercfg-x5nsm\"" Oct 30 14:39:06.983714 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:06.982829 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-33c209-6f4597cc68-xwdkx"] Oct 30 14:39:07.012991 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:07.012126 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-33c209-6f4597cc68-xwdkx"] Oct 30 14:39:07.012991 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:07.012175 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0e8b9-748c768747-vlzmg" podStartSLOduration=1.625227449 podStartE2EDuration="2.012155579s" totalImagesPullingTime="386.92813ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:39:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:39:07.010063445 +0000 UTC m=+3016.754197509" watchObservedRunningTime="2026-10-30 14:39:07.012155579 +0000 UTC m=+3016.756289621" Oct 30 14:39:07.012991 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:07.012242 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-33c209-6f4597cc68-xwdkx" Oct 30 14:39:07.080777 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:07.079163 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18484bef-725d-4ac1-9b31-70195e0eeb38-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-33c209-bffd664d7-kcz62\" (UID: \"18484bef-725d-4ac1-9b31-70195e0eeb38\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c209-bffd664d7-kcz62" Oct 30 14:39:07.080777 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:07.079425 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgtt8\" (UniqueName: \"kubernetes.io/projected/18484bef-725d-4ac1-9b31-70195e0eeb38-kube-api-access-pgtt8\") pod \"payload-pre-processing-e2e-ipp-a-33c209-bffd664d7-kcz62\" (UID: \"18484bef-725d-4ac1-9b31-70195e0eeb38\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c209-bffd664d7-kcz62" Oct 30 14:39:07.181737 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:07.181675 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgtt8\" (UniqueName: \"kubernetes.io/projected/18484bef-725d-4ac1-9b31-70195e0eeb38-kube-api-access-pgtt8\") pod \"payload-pre-processing-e2e-ipp-a-33c209-bffd664d7-kcz62\" (UID: \"18484bef-725d-4ac1-9b31-70195e0eeb38\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c209-bffd664d7-kcz62" Oct 30 14:39:07.181949 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:07.181816 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz8p5\" (UniqueName: \"kubernetes.io/projected/97a44d13-3765-4192-be16-a736077a52ef-kube-api-access-kz8p5\") pod \"payload-processing-e2e-ipp-a-33c209-6f4597cc68-xwdkx\" (UID: \"97a44d13-3765-4192-be16-a736077a52ef\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-33c209-6f4597cc68-xwdkx" Oct 30 14:39:07.181949 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:07.181915 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97a44d13-3765-4192-be16-a736077a52ef-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-33c209-6f4597cc68-xwdkx\" (UID: \"97a44d13-3765-4192-be16-a736077a52ef\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-33c209-6f4597cc68-xwdkx" Oct 30 14:39:07.182091 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:07.181959 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18484bef-725d-4ac1-9b31-70195e0eeb38-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-33c209-bffd664d7-kcz62\" (UID: \"18484bef-725d-4ac1-9b31-70195e0eeb38\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c209-bffd664d7-kcz62" Oct 30 14:39:07.183122 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:07.183093 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18484bef-725d-4ac1-9b31-70195e0eeb38-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-33c209-bffd664d7-kcz62\" (UID: \"18484bef-725d-4ac1-9b31-70195e0eeb38\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c209-bffd664d7-kcz62" Oct 30 14:39:07.228566 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:07.214623 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgtt8\" (UniqueName: \"kubernetes.io/projected/18484bef-725d-4ac1-9b31-70195e0eeb38-kube-api-access-pgtt8\") pod \"payload-pre-processing-e2e-ipp-a-33c209-bffd664d7-kcz62\" (UID: \"18484bef-725d-4ac1-9b31-70195e0eeb38\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c209-bffd664d7-kcz62" Oct 30 14:39:07.283260 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:07.283165 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz8p5\" (UniqueName: \"kubernetes.io/projected/97a44d13-3765-4192-be16-a736077a52ef-kube-api-access-kz8p5\") pod \"payload-processing-e2e-ipp-a-33c209-6f4597cc68-xwdkx\" (UID: \"97a44d13-3765-4192-be16-a736077a52ef\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-33c209-6f4597cc68-xwdkx" Oct 30 14:39:07.283454 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:07.283317 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97a44d13-3765-4192-be16-a736077a52ef-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-33c209-6f4597cc68-xwdkx\" (UID: \"97a44d13-3765-4192-be16-a736077a52ef\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-33c209-6f4597cc68-xwdkx" Oct 30 14:39:07.284071 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:07.284047 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97a44d13-3765-4192-be16-a736077a52ef-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-33c209-6f4597cc68-xwdkx\" (UID: \"97a44d13-3765-4192-be16-a736077a52ef\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-33c209-6f4597cc68-xwdkx" Oct 30 14:39:07.319318 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:07.319275 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz8p5\" (UniqueName: \"kubernetes.io/projected/97a44d13-3765-4192-be16-a736077a52ef-kube-api-access-kz8p5\") pod \"payload-processing-e2e-ipp-a-33c209-6f4597cc68-xwdkx\" (UID: \"97a44d13-3765-4192-be16-a736077a52ef\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-33c209-6f4597cc68-xwdkx" Oct 30 14:39:07.397690 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:07.397621 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-33c209-6f4597cc68-xwdkx" Oct 30 14:39:07.477229 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:07.476956 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c209-bffd664d7-kcz62" Oct 30 14:39:07.720704 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:07.720670 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-33c209-6f4597cc68-xwdkx"] Oct 30 14:39:07.722426 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:39:07.722394 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97a44d13_3765_4192_be16_a736077a52ef.slice/crio-c15e8308eb2721639b795a992ead7493d0be3a7896ea9c1eff6193de1a4f535c WatchSource:0}: Error finding container c15e8308eb2721639b795a992ead7493d0be3a7896ea9c1eff6193de1a4f535c: Status 404 returned error can't find the container with id c15e8308eb2721639b795a992ead7493d0be3a7896ea9c1eff6193de1a4f535c Oct 30 14:39:07.763232 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:07.763200 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d10039ce-nkvds"] Oct 30 14:39:07.771224 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:07.770966 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d10039ce-nkvds" Oct 30 14:39:07.785441 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:07.785328 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d10039ce-nkvds"] Oct 30 14:39:07.790161 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:07.789259 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7m99\" (UniqueName: \"kubernetes.io/projected/e5ba56a4-e3c3-446b-a537-ecb24d8236d1-kube-api-access-z7m99\") pod \"maas-api-revoke-sub-d10039ce-nkvds\" (UID: \"e5ba56a4-e3c3-446b-a537-ecb24d8236d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d10039ce-nkvds" Oct 30 14:39:07.790161 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:07.789383 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5ba56a4-e3c3-446b-a537-ecb24d8236d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d10039ce-nkvds\" (UID: \"e5ba56a4-e3c3-446b-a537-ecb24d8236d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d10039ce-nkvds" Oct 30 14:39:07.790161 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:07.789415 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5ba56a4-e3c3-446b-a537-ecb24d8236d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d10039ce-nkvds\" (UID: \"e5ba56a4-e3c3-446b-a537-ecb24d8236d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d10039ce-nkvds" Oct 30 14:39:07.810974 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:07.810949 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-33c209-bffd664d7-kcz62"] Oct 30 14:39:07.818124 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:39:07.818095 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18484bef_725d_4ac1_9b31_70195e0eeb38.slice/crio-ed5d14211087a0d2aa6daf23e91980140fc4f954b9b4ba1f22599847cf40e0ba WatchSource:0}: Error finding container ed5d14211087a0d2aa6daf23e91980140fc4f954b9b4ba1f22599847cf40e0ba: Status 404 returned error can't find the container with id ed5d14211087a0d2aa6daf23e91980140fc4f954b9b4ba1f22599847cf40e0ba Oct 30 14:39:07.839383 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:07.839356 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:39:07.839568 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:07.839548 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:39:07.870023 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:07.869992 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-33c209-6f4597cc68-xwdkx" event={"ID":"97a44d13-3765-4192-be16-a736077a52ef","Type":"ContainerStarted","Data":"861c805a7801b9aecc623a7279b75a1e2535804849e88807693dd9d2129f1959"} Oct 30 14:39:07.870169 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:07.870049 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-33c209-6f4597cc68-xwdkx" event={"ID":"97a44d13-3765-4192-be16-a736077a52ef","Type":"ContainerStarted","Data":"c15e8308eb2721639b795a992ead7493d0be3a7896ea9c1eff6193de1a4f535c"} Oct 30 14:39:07.870169 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:07.870078 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-33c209-6f4597cc68-xwdkx" Oct 30 14:39:07.871286 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:07.871261 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c209-bffd664d7-kcz62" event={"ID":"18484bef-725d-4ac1-9b31-70195e0eeb38","Type":"ContainerStarted","Data":"ed5d14211087a0d2aa6daf23e91980140fc4f954b9b4ba1f22599847cf40e0ba"} Oct 30 14:39:07.890804 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:07.890780 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7m99\" (UniqueName: \"kubernetes.io/projected/e5ba56a4-e3c3-446b-a537-ecb24d8236d1-kube-api-access-z7m99\") pod \"maas-api-revoke-sub-d10039ce-nkvds\" (UID: \"e5ba56a4-e3c3-446b-a537-ecb24d8236d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d10039ce-nkvds" Oct 30 14:39:07.890994 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:07.890972 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5ba56a4-e3c3-446b-a537-ecb24d8236d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d10039ce-nkvds\" (UID: \"e5ba56a4-e3c3-446b-a537-ecb24d8236d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d10039ce-nkvds" Oct 30 14:39:07.891075 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:07.891008 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5ba56a4-e3c3-446b-a537-ecb24d8236d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d10039ce-nkvds\" (UID: \"e5ba56a4-e3c3-446b-a537-ecb24d8236d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d10039ce-nkvds" Oct 30 14:39:07.891711 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:07.891682 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5ba56a4-e3c3-446b-a537-ecb24d8236d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d10039ce-nkvds\" (UID: \"e5ba56a4-e3c3-446b-a537-ecb24d8236d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d10039ce-nkvds" Oct 30 14:39:07.893568 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:07.893546 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5ba56a4-e3c3-446b-a537-ecb24d8236d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d10039ce-nkvds\" (UID: \"e5ba56a4-e3c3-446b-a537-ecb24d8236d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d10039ce-nkvds" Oct 30 14:39:07.917392 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:07.916874 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7m99\" (UniqueName: \"kubernetes.io/projected/e5ba56a4-e3c3-446b-a537-ecb24d8236d1-kube-api-access-z7m99\") pod \"maas-api-revoke-sub-d10039ce-nkvds\" (UID: \"e5ba56a4-e3c3-446b-a537-ecb24d8236d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d10039ce-nkvds" Oct 30 14:39:07.917818 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:07.917765 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-33c209-6f4597cc68-xwdkx" podStartSLOduration=1.917749224 podStartE2EDuration="1.917749224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:39:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:39:07.915762174 +0000 UTC m=+3017.659896216" watchObservedRunningTime="2026-10-30 14:39:07.917749224 +0000 UTC m=+3017.661883265" Oct 30 14:39:08.102111 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:08.102021 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d10039ce-nkvds" Oct 30 14:39:08.144594 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:08.144559 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:39:08.413422 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:39:08.413393 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5ba56a4_e3c3_446b_a537_ecb24d8236d1.slice/crio-0bf590aaf1e056f4e24fb7e646fe6d97493ed6a649af31fc150e24beaad10868 WatchSource:0}: Error finding container 0bf590aaf1e056f4e24fb7e646fe6d97493ed6a649af31fc150e24beaad10868: Status 404 returned error can't find the container with id 0bf590aaf1e056f4e24fb7e646fe6d97493ed6a649af31fc150e24beaad10868 Oct 30 14:39:08.413829 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:08.413805 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d10039ce-nkvds"] Oct 30 14:39:08.879593 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:08.879554 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d10039ce-nkvds" event={"ID":"e5ba56a4-e3c3-446b-a537-ecb24d8236d1","Type":"ContainerStarted","Data":"5e958dd07e62c7240dd42891d79a3ec7c3f5dd019ff06c72311442a0bf9fbeb3"} Oct 30 14:39:08.879798 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:08.879621 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d10039ce-nkvds" event={"ID":"e5ba56a4-e3c3-446b-a537-ecb24d8236d1","Type":"ContainerStarted","Data":"0bf590aaf1e056f4e24fb7e646fe6d97493ed6a649af31fc150e24beaad10868"} Oct 30 14:39:08.881410 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:08.881374 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c209-bffd664d7-kcz62" event={"ID":"18484bef-725d-4ac1-9b31-70195e0eeb38","Type":"ContainerStarted","Data":"8d2395ada5069df0c0df8ced363f31b6defc6fdc1ed5c5902b1efd2b799f0ce9"} Oct 30 14:39:08.881555 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:08.881448 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c209-bffd664d7-kcz62" Oct 30 14:39:08.928590 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:08.928537 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d10039ce-nkvds" podStartSLOduration=1.928524258 podStartE2EDuration="1.928524258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:39:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:39:08.92488734 +0000 UTC m=+3018.669021381" watchObservedRunningTime="2026-10-30 14:39:08.928524258 +0000 UTC m=+3018.672658299" Oct 30 14:39:08.971709 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:08.971571 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c209-bffd664d7-kcz62" podStartSLOduration=2.971555461 podStartE2EDuration="2.971555461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:39:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:39:08.970259919 +0000 UTC m=+3018.714393958" watchObservedRunningTime="2026-10-30 14:39:08.971555461 +0000 UTC m=+3018.715689502" Oct 30 14:39:09.891627 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:09.891581 4982 generic.go:356] "Generic (PLEG): container finished" podID="e5ba56a4-e3c3-446b-a537-ecb24d8236d1" containerID="5e958dd07e62c7240dd42891d79a3ec7c3f5dd019ff06c72311442a0bf9fbeb3" exitCode=0 Oct 30 14:39:09.892076 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:09.891670 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d10039ce-nkvds" event={"ID":"e5ba56a4-e3c3-446b-a537-ecb24d8236d1","Type":"ContainerDied","Data":"5e958dd07e62c7240dd42891d79a3ec7c3f5dd019ff06c72311442a0bf9fbeb3"} Oct 30 14:39:10.359389 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:10.359347 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:39:11.072292 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:11.072268 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d10039ce-nkvds" Oct 30 14:39:11.143276 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:11.143238 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5ba56a4-e3c3-446b-a537-ecb24d8236d1-kube-api-access-z7m99\" (UniqueName: \"kubernetes.io/projected/e5ba56a4-e3c3-446b-a537-ecb24d8236d1-kube-api-access-z7m99\") pod \"e5ba56a4-e3c3-446b-a537-ecb24d8236d1\" (UID: \"e5ba56a4-e3c3-446b-a537-ecb24d8236d1\") " Oct 30 14:39:11.143443 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:11.143341 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5ba56a4-e3c3-446b-a537-ecb24d8236d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5ba56a4-e3c3-446b-a537-ecb24d8236d1-maas-api-openshift-ca\") pod \"e5ba56a4-e3c3-446b-a537-ecb24d8236d1\" (UID: \"e5ba56a4-e3c3-446b-a537-ecb24d8236d1\") " Oct 30 14:39:11.143443 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:11.143386 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5ba56a4-e3c3-446b-a537-ecb24d8236d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5ba56a4-e3c3-446b-a537-ecb24d8236d1-maas-api-xks-ca\") pod \"e5ba56a4-e3c3-446b-a537-ecb24d8236d1\" (UID: \"e5ba56a4-e3c3-446b-a537-ecb24d8236d1\") " Oct 30 14:39:11.144224 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:11.144193 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5ba56a4-e3c3-446b-a537-ecb24d8236d1-maas-api-openshift-ca" pod "e5ba56a4-e3c3-446b-a537-ecb24d8236d1" (UID: "e5ba56a4-e3c3-446b-a537-ecb24d8236d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:39:11.145969 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:11.145941 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5ba56a4-e3c3-446b-a537-ecb24d8236d1-kube-api-access-z7m99" pod "e5ba56a4-e3c3-446b-a537-ecb24d8236d1" (UID: "e5ba56a4-e3c3-446b-a537-ecb24d8236d1"). InnerVolumeSpecName "kube-api-access-z7m99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:39:11.146442 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:11.146421 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5ba56a4-e3c3-446b-a537-ecb24d8236d1-maas-api-xks-ca" pod "e5ba56a4-e3c3-446b-a537-ecb24d8236d1" (UID: "e5ba56a4-e3c3-446b-a537-ecb24d8236d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:39:11.245438 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:11.245397 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7m99\" (UniqueName: \"kubernetes.io/projected/e5ba56a4-e3c3-446b-a537-ecb24d8236d1-kube-api-access-z7m99\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:39:11.245438 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:11.245429 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5ba56a4-e3c3-446b-a537-ecb24d8236d1-maas-api-openshift-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:39:11.245438 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:11.245440 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5ba56a4-e3c3-446b-a537-ecb24d8236d1-maas-api-xks-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:39:11.909744 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:11.909712 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d10039ce-nkvds" Oct 30 14:39:11.909955 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:11.909711 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d10039ce-nkvds" event={"ID":"e5ba56a4-e3c3-446b-a537-ecb24d8236d1","Type":"ContainerDied","Data":"0bf590aaf1e056f4e24fb7e646fe6d97493ed6a649af31fc150e24beaad10868"} Oct 30 14:39:11.909955 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:11.909858 4982 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0bf590aaf1e056f4e24fb7e646fe6d97493ed6a649af31fc150e24beaad10868" Oct 30 14:39:12.289707 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:12.289614 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:39:13.422805 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:13.422773 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4bb39cb-87vp8"] Oct 30 14:39:13.423560 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:13.423536 4982 cpu_manager.go:414] "containerMap: removing container" podUID="e5ba56a4-e3c3-446b-a537-ecb24d8236d1" containerName="revoke-keys" Oct 30 14:39:13.423560 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:13.423560 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5ba56a4-e3c3-446b-a537-ecb24d8236d1" containerName="revoke-keys" Oct 30 14:39:13.423798 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:13.423783 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5ba56a4-e3c3-446b-a537-ecb24d8236d1" containerName="revoke-keys" Oct 30 14:39:13.427742 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:13.427718 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4bb39cb-87vp8" Oct 30 14:39:13.471000 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:13.470971 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4bb39cb-87vp8"] Oct 30 14:39:13.472251 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:13.472228 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b2ae3e3-d20c-49a4-a8cc-4685ba38f6cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4bb39cb-87vp8\" (UID: \"2b2ae3e3-d20c-49a4-a8cc-4685ba38f6cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4bb39cb-87vp8" Oct 30 14:39:13.472372 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:13.472284 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxkt5\" (UniqueName: \"kubernetes.io/projected/2b2ae3e3-d20c-49a4-a8cc-4685ba38f6cd-kube-api-access-bxkt5\") pod \"maas-api-revoke-sub-b4bb39cb-87vp8\" (UID: \"2b2ae3e3-d20c-49a4-a8cc-4685ba38f6cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4bb39cb-87vp8" Oct 30 14:39:13.472524 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:13.472509 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b2ae3e3-d20c-49a4-a8cc-4685ba38f6cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4bb39cb-87vp8\" (UID: \"2b2ae3e3-d20c-49a4-a8cc-4685ba38f6cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4bb39cb-87vp8" Oct 30 14:39:13.479729 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:13.479708 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:39:13.479928 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:13.479753 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:39:13.573594 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:13.573564 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b2ae3e3-d20c-49a4-a8cc-4685ba38f6cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4bb39cb-87vp8\" (UID: \"2b2ae3e3-d20c-49a4-a8cc-4685ba38f6cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4bb39cb-87vp8" Oct 30 14:39:13.573778 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:13.573608 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxkt5\" (UniqueName: \"kubernetes.io/projected/2b2ae3e3-d20c-49a4-a8cc-4685ba38f6cd-kube-api-access-bxkt5\") pod \"maas-api-revoke-sub-b4bb39cb-87vp8\" (UID: \"2b2ae3e3-d20c-49a4-a8cc-4685ba38f6cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4bb39cb-87vp8" Oct 30 14:39:13.573778 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:13.573739 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b2ae3e3-d20c-49a4-a8cc-4685ba38f6cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4bb39cb-87vp8\" (UID: \"2b2ae3e3-d20c-49a4-a8cc-4685ba38f6cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4bb39cb-87vp8" Oct 30 14:39:13.574375 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:13.574355 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b2ae3e3-d20c-49a4-a8cc-4685ba38f6cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4bb39cb-87vp8\" (UID: \"2b2ae3e3-d20c-49a4-a8cc-4685ba38f6cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4bb39cb-87vp8" Oct 30 14:39:13.576067 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:13.576040 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b2ae3e3-d20c-49a4-a8cc-4685ba38f6cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4bb39cb-87vp8\" (UID: \"2b2ae3e3-d20c-49a4-a8cc-4685ba38f6cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4bb39cb-87vp8" Oct 30 14:39:13.599611 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:13.599585 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxkt5\" (UniqueName: \"kubernetes.io/projected/2b2ae3e3-d20c-49a4-a8cc-4685ba38f6cd-kube-api-access-bxkt5\") pod \"maas-api-revoke-sub-b4bb39cb-87vp8\" (UID: \"2b2ae3e3-d20c-49a4-a8cc-4685ba38f6cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4bb39cb-87vp8" Oct 30 14:39:13.742351 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:13.742268 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4bb39cb-87vp8" Oct 30 14:39:13.887831 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:13.887801 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0e8b9-748c768747-vlzmg" Oct 30 14:39:14.058276 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:39:14.058239 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b2ae3e3_d20c_49a4_a8cc_4685ba38f6cd.slice/crio-1342e82e691eefec4ed00ceeec186a2d1b76253f0e52a8e80c8676f20403e4de WatchSource:0}: Error finding container 1342e82e691eefec4ed00ceeec186a2d1b76253f0e52a8e80c8676f20403e4de: Status 404 returned error can't find the container with id 1342e82e691eefec4ed00ceeec186a2d1b76253f0e52a8e80c8676f20403e4de Oct 30 14:39:14.059338 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:14.059307 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4bb39cb-87vp8"] Oct 30 14:39:14.349438 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:14.349406 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:39:14.899737 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:14.899674 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-f1f206/auth-test-model-f1f206-kserve-74c48d6ddf-wx5tn" podUID="8483d44c-7904-49bc-8f55-87870acc8e58" containerName="main" containerID="cri-o://8a74e3bf8d6104c6b16060e485e7738bb577b5bbbe12a7a680cc2bb2fafc5139" gracePeriod=45 Oct 30 14:39:14.936784 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:14.936751 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4bb39cb-87vp8" event={"ID":"2b2ae3e3-d20c-49a4-a8cc-4685ba38f6cd","Type":"ContainerStarted","Data":"429fa549b9c353100f0eebee444097c33fed8ae919d8e01ffa62c49e1009d113"} Oct 30 14:39:14.936921 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:14.936805 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4bb39cb-87vp8" event={"ID":"2b2ae3e3-d20c-49a4-a8cc-4685ba38f6cd","Type":"ContainerStarted","Data":"1342e82e691eefec4ed00ceeec186a2d1b76253f0e52a8e80c8676f20403e4de"} Oct 30 14:39:14.982702 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:14.982628 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4bb39cb-87vp8" podStartSLOduration=1.982610926 podStartE2EDuration="1.982610926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:39:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:39:14.980284993 +0000 UTC m=+3024.724419033" watchObservedRunningTime="2026-10-30 14:39:14.982610926 +0000 UTC m=+3024.726744970" Oct 30 14:39:15.196093 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:15.196064 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-f1f206/auth-test-model-f1f206-kserve-74c48d6ddf-wx5tn" Oct 30 14:39:15.303571 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:15.303535 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8483d44c-7904-49bc-8f55-87870acc8e58-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8483d44c-7904-49bc-8f55-87870acc8e58-tmp-dir\") pod \"8483d44c-7904-49bc-8f55-87870acc8e58\" (UID: \"8483d44c-7904-49bc-8f55-87870acc8e58\") " Oct 30 14:39:15.303785 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:15.303692 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8483d44c-7904-49bc-8f55-87870acc8e58-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8483d44c-7904-49bc-8f55-87870acc8e58-model-cache\") pod \"8483d44c-7904-49bc-8f55-87870acc8e58\" (UID: \"8483d44c-7904-49bc-8f55-87870acc8e58\") " Oct 30 14:39:15.303785 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:15.303739 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8483d44c-7904-49bc-8f55-87870acc8e58-home\" (UniqueName: \"kubernetes.io/empty-dir/8483d44c-7904-49bc-8f55-87870acc8e58-home\") pod \"8483d44c-7904-49bc-8f55-87870acc8e58\" (UID: \"8483d44c-7904-49bc-8f55-87870acc8e58\") " Oct 30 14:39:15.303920 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:15.303789 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8483d44c-7904-49bc-8f55-87870acc8e58-kube-api-access-n9t4x\" (UniqueName: \"kubernetes.io/projected/8483d44c-7904-49bc-8f55-87870acc8e58-kube-api-access-n9t4x\") pod \"8483d44c-7904-49bc-8f55-87870acc8e58\" (UID: \"8483d44c-7904-49bc-8f55-87870acc8e58\") " Oct 30 14:39:15.303920 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:15.303798 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8483d44c-7904-49bc-8f55-87870acc8e58-tmp-dir" pod "8483d44c-7904-49bc-8f55-87870acc8e58" (UID: "8483d44c-7904-49bc-8f55-87870acc8e58"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:39:15.303920 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:15.303905 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8483d44c-7904-49bc-8f55-87870acc8e58-tls-certs\" (UniqueName: \"kubernetes.io/secret/8483d44c-7904-49bc-8f55-87870acc8e58-tls-certs\") pod \"8483d44c-7904-49bc-8f55-87870acc8e58\" (UID: \"8483d44c-7904-49bc-8f55-87870acc8e58\") " Oct 30 14:39:15.304141 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:15.303940 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8483d44c-7904-49bc-8f55-87870acc8e58-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8483d44c-7904-49bc-8f55-87870acc8e58-dshm\") pod \"8483d44c-7904-49bc-8f55-87870acc8e58\" (UID: \"8483d44c-7904-49bc-8f55-87870acc8e58\") " Oct 30 14:39:15.304141 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:15.304092 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8483d44c-7904-49bc-8f55-87870acc8e58-model-cache" pod "8483d44c-7904-49bc-8f55-87870acc8e58" (UID: "8483d44c-7904-49bc-8f55-87870acc8e58"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:39:15.304306 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:15.304284 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8483d44c-7904-49bc-8f55-87870acc8e58-home" pod "8483d44c-7904-49bc-8f55-87870acc8e58" (UID: "8483d44c-7904-49bc-8f55-87870acc8e58"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:39:15.304675 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:15.304619 4982 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8483d44c-7904-49bc-8f55-87870acc8e58-tmp-dir\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:39:15.304675 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:15.304663 4982 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8483d44c-7904-49bc-8f55-87870acc8e58-model-cache\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:39:15.304675 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:15.304676 4982 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8483d44c-7904-49bc-8f55-87870acc8e58-home\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:39:15.307326 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:15.307302 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8483d44c-7904-49bc-8f55-87870acc8e58-dshm" pod "8483d44c-7904-49bc-8f55-87870acc8e58" (UID: "8483d44c-7904-49bc-8f55-87870acc8e58"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:39:15.307429 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:15.307404 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8483d44c-7904-49bc-8f55-87870acc8e58-tls-certs" pod "8483d44c-7904-49bc-8f55-87870acc8e58" (UID: "8483d44c-7904-49bc-8f55-87870acc8e58"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:39:15.307568 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:15.307518 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8483d44c-7904-49bc-8f55-87870acc8e58-kube-api-access-n9t4x" pod "8483d44c-7904-49bc-8f55-87870acc8e58" (UID: "8483d44c-7904-49bc-8f55-87870acc8e58"). InnerVolumeSpecName "kube-api-access-n9t4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:39:15.405730 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:15.405698 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9t4x\" (UniqueName: \"kubernetes.io/projected/8483d44c-7904-49bc-8f55-87870acc8e58-kube-api-access-n9t4x\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:39:15.405730 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:15.405730 4982 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8483d44c-7904-49bc-8f55-87870acc8e58-tls-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:39:15.405904 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:15.405746 4982 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8483d44c-7904-49bc-8f55-87870acc8e58-dshm\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:39:15.946227 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:15.946196 4982 generic.go:356] "Generic (PLEG): container finished" podID="8483d44c-7904-49bc-8f55-87870acc8e58" containerID="8a74e3bf8d6104c6b16060e485e7738bb577b5bbbe12a7a680cc2bb2fafc5139" exitCode=0 Oct 30 14:39:15.946724 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:15.946241 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-f1f206/auth-test-model-f1f206-kserve-74c48d6ddf-wx5tn" Oct 30 14:39:15.946724 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:15.946272 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-f1f206/auth-test-model-f1f206-kserve-74c48d6ddf-wx5tn" event={"ID":"8483d44c-7904-49bc-8f55-87870acc8e58","Type":"ContainerDied","Data":"8a74e3bf8d6104c6b16060e485e7738bb577b5bbbe12a7a680cc2bb2fafc5139"} Oct 30 14:39:15.946724 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:15.946310 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-f1f206/auth-test-model-f1f206-kserve-74c48d6ddf-wx5tn" event={"ID":"8483d44c-7904-49bc-8f55-87870acc8e58","Type":"ContainerDied","Data":"1541c740dac658f565fe209e5a1ed1b5479da3736d97ff579ad7cde93b666e6c"} Oct 30 14:39:15.946724 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:15.946327 4982 scope.go:122] "RemoveContainer" containerID="8a74e3bf8d6104c6b16060e485e7738bb577b5bbbe12a7a680cc2bb2fafc5139" Oct 30 14:39:15.948308 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:15.948287 4982 generic.go:356] "Generic (PLEG): container finished" podID="2b2ae3e3-d20c-49a4-a8cc-4685ba38f6cd" containerID="429fa549b9c353100f0eebee444097c33fed8ae919d8e01ffa62c49e1009d113" exitCode=0 Oct 30 14:39:15.948443 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:15.948368 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4bb39cb-87vp8" event={"ID":"2b2ae3e3-d20c-49a4-a8cc-4685ba38f6cd","Type":"ContainerDied","Data":"429fa549b9c353100f0eebee444097c33fed8ae919d8e01ffa62c49e1009d113"} Oct 30 14:39:15.957733 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:15.957708 4982 scope.go:122] "RemoveContainer" containerID="8a74e3bf8d6104c6b16060e485e7738bb577b5bbbe12a7a680cc2bb2fafc5139" Oct 30 14:39:15.957997 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:39:15.957979 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a74e3bf8d6104c6b16060e485e7738bb577b5bbbe12a7a680cc2bb2fafc5139\": container with ID starting with 8a74e3bf8d6104c6b16060e485e7738bb577b5bbbe12a7a680cc2bb2fafc5139 not found: ID does not exist" containerID="8a74e3bf8d6104c6b16060e485e7738bb577b5bbbe12a7a680cc2bb2fafc5139" Oct 30 14:39:15.958058 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:15.958004 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a74e3bf8d6104c6b16060e485e7738bb577b5bbbe12a7a680cc2bb2fafc5139"} err="failed to get container status \"8a74e3bf8d6104c6b16060e485e7738bb577b5bbbe12a7a680cc2bb2fafc5139\": rpc error: code = NotFound desc = could not find container \"8a74e3bf8d6104c6b16060e485e7738bb577b5bbbe12a7a680cc2bb2fafc5139\": container with ID starting with 8a74e3bf8d6104c6b16060e485e7738bb577b5bbbe12a7a680cc2bb2fafc5139 not found: ID does not exist" Oct 30 14:39:16.007100 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:16.007067 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-f1f206/auth-test-model-f1f206-kserve-74c48d6ddf-wx5tn"] Oct 30 14:39:16.020055 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:16.020020 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-f1f206/auth-test-model-f1f206-kserve-74c48d6ddf-wx5tn"] Oct 30 14:39:16.401199 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:16.401167 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8483d44c-7904-49bc-8f55-87870acc8e58" path="/var/lib/kubelet/pods/8483d44c-7904-49bc-8f55-87870acc8e58/volumes" Oct 30 14:39:16.539849 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:16.539814 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:39:17.126347 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:17.126320 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4bb39cb-87vp8" Oct 30 14:39:17.231853 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:17.231760 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b2ae3e3-d20c-49a4-a8cc-4685ba38f6cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b2ae3e3-d20c-49a4-a8cc-4685ba38f6cd-maas-api-openshift-ca\") pod \"2b2ae3e3-d20c-49a4-a8cc-4685ba38f6cd\" (UID: \"2b2ae3e3-d20c-49a4-a8cc-4685ba38f6cd\") " Oct 30 14:39:17.231853 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:17.231835 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b2ae3e3-d20c-49a4-a8cc-4685ba38f6cd-kube-api-access-bxkt5\" (UniqueName: \"kubernetes.io/projected/2b2ae3e3-d20c-49a4-a8cc-4685ba38f6cd-kube-api-access-bxkt5\") pod \"2b2ae3e3-d20c-49a4-a8cc-4685ba38f6cd\" (UID: \"2b2ae3e3-d20c-49a4-a8cc-4685ba38f6cd\") " Oct 30 14:39:17.232101 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:17.232012 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b2ae3e3-d20c-49a4-a8cc-4685ba38f6cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b2ae3e3-d20c-49a4-a8cc-4685ba38f6cd-maas-api-xks-ca\") pod \"2b2ae3e3-d20c-49a4-a8cc-4685ba38f6cd\" (UID: \"2b2ae3e3-d20c-49a4-a8cc-4685ba38f6cd\") " Oct 30 14:39:17.232202 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:17.232166 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b2ae3e3-d20c-49a4-a8cc-4685ba38f6cd-maas-api-openshift-ca" pod "2b2ae3e3-d20c-49a4-a8cc-4685ba38f6cd" (UID: "2b2ae3e3-d20c-49a4-a8cc-4685ba38f6cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:39:17.232515 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:17.232494 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b2ae3e3-d20c-49a4-a8cc-4685ba38f6cd-maas-api-openshift-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:39:17.234107 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:17.234082 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b2ae3e3-d20c-49a4-a8cc-4685ba38f6cd-maas-api-xks-ca" pod "2b2ae3e3-d20c-49a4-a8cc-4685ba38f6cd" (UID: "2b2ae3e3-d20c-49a4-a8cc-4685ba38f6cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:39:17.234437 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:17.234409 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b2ae3e3-d20c-49a4-a8cc-4685ba38f6cd-kube-api-access-bxkt5" pod "2b2ae3e3-d20c-49a4-a8cc-4685ba38f6cd" (UID: "2b2ae3e3-d20c-49a4-a8cc-4685ba38f6cd"). InnerVolumeSpecName "kube-api-access-bxkt5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:39:17.334273 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:17.334231 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxkt5\" (UniqueName: \"kubernetes.io/projected/2b2ae3e3-d20c-49a4-a8cc-4685ba38f6cd-kube-api-access-bxkt5\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:39:17.334273 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:17.334268 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b2ae3e3-d20c-49a4-a8cc-4685ba38f6cd-maas-api-xks-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:39:17.965467 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:17.965436 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4bb39cb-87vp8" Oct 30 14:39:17.965634 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:17.965465 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4bb39cb-87vp8" event={"ID":"2b2ae3e3-d20c-49a4-a8cc-4685ba38f6cd","Type":"ContainerDied","Data":"1342e82e691eefec4ed00ceeec186a2d1b76253f0e52a8e80c8676f20403e4de"} Oct 30 14:39:17.965634 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:17.965511 4982 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1342e82e691eefec4ed00ceeec186a2d1b76253f0e52a8e80c8676f20403e4de" Oct 30 14:39:18.506871 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:18.506833 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:39:18.886053 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:18.886022 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-33c209-6f4597cc68-xwdkx" Oct 30 14:39:19.895533 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:19.895502 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c209-bffd664d7-kcz62" Oct 30 14:39:20.137877 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:20.137826 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:39:20.507937 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:20.507894 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:39:22.397439 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:22.397401 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:39:23.900309 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:23.900270 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-030ea30d-cvdkq"] Oct 30 14:39:23.900885 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:23.900868 4982 cpu_manager.go:414] "containerMap: removing container" podUID="2b2ae3e3-d20c-49a4-a8cc-4685ba38f6cd" containerName="revoke-keys" Oct 30 14:39:23.900885 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:23.900884 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b2ae3e3-d20c-49a4-a8cc-4685ba38f6cd" containerName="revoke-keys" Oct 30 14:39:23.901022 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:23.900924 4982 cpu_manager.go:414] "containerMap: removing container" podUID="8483d44c-7904-49bc-8f55-87870acc8e58" containerName="main" Oct 30 14:39:23.901022 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:23.900930 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8483d44c-7904-49bc-8f55-87870acc8e58" containerName="main" Oct 30 14:39:23.901129 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:23.901026 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="8483d44c-7904-49bc-8f55-87870acc8e58" containerName="main" Oct 30 14:39:23.901129 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:23.901038 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b2ae3e3-d20c-49a4-a8cc-4685ba38f6cd" containerName="revoke-keys" Oct 30 14:39:23.904654 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:23.904624 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-030ea30d-cvdkq" Oct 30 14:39:23.925561 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:23.925531 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-030ea30d-cvdkq"] Oct 30 14:39:23.934669 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:23.934624 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:39:23.934820 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:23.934782 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:39:23.975202 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:23.975166 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkhv2\" (UniqueName: \"kubernetes.io/projected/c6ae17d8-7e51-411a-bd43-75331301f48a-kube-api-access-hkhv2\") pod \"maas-api-revoke-sub-030ea30d-cvdkq\" (UID: \"c6ae17d8-7e51-411a-bd43-75331301f48a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-030ea30d-cvdkq" Oct 30 14:39:23.975368 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:23.975232 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6ae17d8-7e51-411a-bd43-75331301f48a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-030ea30d-cvdkq\" (UID: \"c6ae17d8-7e51-411a-bd43-75331301f48a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-030ea30d-cvdkq" Oct 30 14:39:23.975411 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:23.975363 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6ae17d8-7e51-411a-bd43-75331301f48a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-030ea30d-cvdkq\" (UID: \"c6ae17d8-7e51-411a-bd43-75331301f48a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-030ea30d-cvdkq" Oct 30 14:39:24.076881 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:24.076844 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkhv2\" (UniqueName: \"kubernetes.io/projected/c6ae17d8-7e51-411a-bd43-75331301f48a-kube-api-access-hkhv2\") pod \"maas-api-revoke-sub-030ea30d-cvdkq\" (UID: \"c6ae17d8-7e51-411a-bd43-75331301f48a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-030ea30d-cvdkq" Oct 30 14:39:24.077069 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:24.076905 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6ae17d8-7e51-411a-bd43-75331301f48a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-030ea30d-cvdkq\" (UID: \"c6ae17d8-7e51-411a-bd43-75331301f48a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-030ea30d-cvdkq" Oct 30 14:39:24.077069 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:24.076974 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6ae17d8-7e51-411a-bd43-75331301f48a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-030ea30d-cvdkq\" (UID: \"c6ae17d8-7e51-411a-bd43-75331301f48a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-030ea30d-cvdkq" Oct 30 14:39:24.077688 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:24.077667 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6ae17d8-7e51-411a-bd43-75331301f48a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-030ea30d-cvdkq\" (UID: \"c6ae17d8-7e51-411a-bd43-75331301f48a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-030ea30d-cvdkq" Oct 30 14:39:24.079227 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:24.079204 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6ae17d8-7e51-411a-bd43-75331301f48a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-030ea30d-cvdkq\" (UID: \"c6ae17d8-7e51-411a-bd43-75331301f48a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-030ea30d-cvdkq" Oct 30 14:39:24.104836 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:24.104809 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkhv2\" (UniqueName: \"kubernetes.io/projected/c6ae17d8-7e51-411a-bd43-75331301f48a-kube-api-access-hkhv2\") pod \"maas-api-revoke-sub-030ea30d-cvdkq\" (UID: \"c6ae17d8-7e51-411a-bd43-75331301f48a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-030ea30d-cvdkq" Oct 30 14:39:24.219884 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:24.219805 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-030ea30d-cvdkq" Oct 30 14:39:24.452269 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:24.452237 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:39:24.534927 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:24.534897 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-030ea30d-cvdkq"] Oct 30 14:39:24.535966 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:39:24.535936 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6ae17d8_7e51_411a_bd43_75331301f48a.slice/crio-cc6278907ff35a186826b341d58054d88d687d2cebc5cb04abe38b4ac9027d05 WatchSource:0}: Error finding container cc6278907ff35a186826b341d58054d88d687d2cebc5cb04abe38b4ac9027d05: Status 404 returned error can't find the container with id cc6278907ff35a186826b341d58054d88d687d2cebc5cb04abe38b4ac9027d05 Oct 30 14:39:25.024135 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:25.024097 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-030ea30d-cvdkq" event={"ID":"c6ae17d8-7e51-411a-bd43-75331301f48a","Type":"ContainerStarted","Data":"1e387171fe5e42298d004876a1876c337581db9adbc0a959018a19c7417cea89"} Oct 30 14:39:25.024621 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:25.024160 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-030ea30d-cvdkq" event={"ID":"c6ae17d8-7e51-411a-bd43-75331301f48a","Type":"ContainerStarted","Data":"cc6278907ff35a186826b341d58054d88d687d2cebc5cb04abe38b4ac9027d05"} Oct 30 14:39:25.069751 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:25.069704 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-030ea30d-cvdkq" podStartSLOduration=2.069690072 podStartE2EDuration="2.069690072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:39:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:39:25.06812016 +0000 UTC m=+3034.812254201" watchObservedRunningTime="2026-10-30 14:39:25.069690072 +0000 UTC m=+3034.813824109" Oct 30 14:39:26.035415 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:26.035382 4982 generic.go:356] "Generic (PLEG): container finished" podID="c6ae17d8-7e51-411a-bd43-75331301f48a" containerID="1e387171fe5e42298d004876a1876c337581db9adbc0a959018a19c7417cea89" exitCode=0 Oct 30 14:39:26.035827 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:26.035446 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-030ea30d-cvdkq" event={"ID":"c6ae17d8-7e51-411a-bd43-75331301f48a","Type":"ContainerDied","Data":"1e387171fe5e42298d004876a1876c337581db9adbc0a959018a19c7417cea89"} Oct 30 14:39:26.445651 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:26.445619 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:39:27.223531 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:27.223503 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-030ea30d-cvdkq" Oct 30 14:39:27.322573 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:27.322537 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6ae17d8-7e51-411a-bd43-75331301f48a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6ae17d8-7e51-411a-bd43-75331301f48a-maas-api-openshift-ca\") pod \"c6ae17d8-7e51-411a-bd43-75331301f48a\" (UID: \"c6ae17d8-7e51-411a-bd43-75331301f48a\") " Oct 30 14:39:27.322796 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:27.322595 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6ae17d8-7e51-411a-bd43-75331301f48a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6ae17d8-7e51-411a-bd43-75331301f48a-maas-api-xks-ca\") pod \"c6ae17d8-7e51-411a-bd43-75331301f48a\" (UID: \"c6ae17d8-7e51-411a-bd43-75331301f48a\") " Oct 30 14:39:27.322796 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:27.322705 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6ae17d8-7e51-411a-bd43-75331301f48a-kube-api-access-hkhv2\" (UniqueName: \"kubernetes.io/projected/c6ae17d8-7e51-411a-bd43-75331301f48a-kube-api-access-hkhv2\") pod \"c6ae17d8-7e51-411a-bd43-75331301f48a\" (UID: \"c6ae17d8-7e51-411a-bd43-75331301f48a\") " Oct 30 14:39:27.323064 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:27.323034 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6ae17d8-7e51-411a-bd43-75331301f48a-maas-api-openshift-ca" pod "c6ae17d8-7e51-411a-bd43-75331301f48a" (UID: "c6ae17d8-7e51-411a-bd43-75331301f48a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:39:27.323266 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:27.323246 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6ae17d8-7e51-411a-bd43-75331301f48a-maas-api-openshift-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:39:27.324995 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:27.324958 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6ae17d8-7e51-411a-bd43-75331301f48a-kube-api-access-hkhv2" pod "c6ae17d8-7e51-411a-bd43-75331301f48a" (UID: "c6ae17d8-7e51-411a-bd43-75331301f48a"). InnerVolumeSpecName "kube-api-access-hkhv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:39:27.325109 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:27.325093 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6ae17d8-7e51-411a-bd43-75331301f48a-maas-api-xks-ca" pod "c6ae17d8-7e51-411a-bd43-75331301f48a" (UID: "c6ae17d8-7e51-411a-bd43-75331301f48a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:39:27.424339 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:27.424298 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6ae17d8-7e51-411a-bd43-75331301f48a-maas-api-xks-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:39:27.424339 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:27.424329 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkhv2\" (UniqueName: \"kubernetes.io/projected/c6ae17d8-7e51-411a-bd43-75331301f48a-kube-api-access-hkhv2\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:39:28.014886 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:39:28.014845 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:39:28.051074 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:28.051041 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-030ea30d-cvdkq" event={"ID":"c6ae17d8-7e51-411a-bd43-75331301f48a","Type":"ContainerDied","Data":"cc6278907ff35a186826b341d58054d88d687d2cebc5cb04abe38b4ac9027d05"} Oct 30 14:39:28.051229 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:28.051097 4982 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc6278907ff35a186826b341d58054d88d687d2cebc5cb04abe38b4ac9027d05" Oct 30 14:39:28.051229 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:28.051048 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-030ea30d-cvdkq" Oct 30 14:39:28.348404 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:28.348370 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-460b0b-5d5bdd49b-m8txx"] Oct 30 14:39:28.348990 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:28.348975 4982 cpu_manager.go:414] "containerMap: removing container" podUID="c6ae17d8-7e51-411a-bd43-75331301f48a" containerName="revoke-keys" Oct 30 14:39:28.349036 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:28.348990 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6ae17d8-7e51-411a-bd43-75331301f48a" containerName="revoke-keys" Oct 30 14:39:28.349122 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:28.349114 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6ae17d8-7e51-411a-bd43-75331301f48a" containerName="revoke-keys" Oct 30 14:39:28.352875 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:28.352856 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-460b0b-5d5bdd49b-m8txx" Oct 30 14:39:28.382870 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:28.382845 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-460b0b-serving-cert\"" Oct 30 14:39:28.388228 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:28.388199 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-460b0b-5d5bdd49b-m8txx"] Oct 30 14:39:28.436544 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:28.436514 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/311279d9-d231-4177-8401-9e9598ed9ef0-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-460b0b-5d5bdd49b-m8txx\" (UID: \"311279d9-d231-4177-8401-9e9598ed9ef0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-460b0b-5d5bdd49b-m8txx" Oct 30 14:39:28.436681 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:28.436569 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/311279d9-d231-4177-8401-9e9598ed9ef0-maas-api-tls\") pod \"maas-api-e2e-ipp-b-460b0b-5d5bdd49b-m8txx\" (UID: \"311279d9-d231-4177-8401-9e9598ed9ef0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-460b0b-5d5bdd49b-m8txx" Oct 30 14:39:28.436745 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:28.436701 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl92b\" (UniqueName: \"kubernetes.io/projected/311279d9-d231-4177-8401-9e9598ed9ef0-kube-api-access-nl92b\") pod \"maas-api-e2e-ipp-b-460b0b-5d5bdd49b-m8txx\" (UID: \"311279d9-d231-4177-8401-9e9598ed9ef0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-460b0b-5d5bdd49b-m8txx" Oct 30 14:39:28.436847 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:28.436826 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/311279d9-d231-4177-8401-9e9598ed9ef0-metrics-certs\") pod \"maas-api-e2e-ipp-b-460b0b-5d5bdd49b-m8txx\" (UID: \"311279d9-d231-4177-8401-9e9598ed9ef0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-460b0b-5d5bdd49b-m8txx" Oct 30 14:39:28.535596 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:28.535567 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:39:28.538372 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:28.538335 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl92b\" (UniqueName: \"kubernetes.io/projected/311279d9-d231-4177-8401-9e9598ed9ef0-kube-api-access-nl92b\") pod \"maas-api-e2e-ipp-b-460b0b-5d5bdd49b-m8txx\" (UID: \"311279d9-d231-4177-8401-9e9598ed9ef0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-460b0b-5d5bdd49b-m8txx" Oct 30 14:39:28.538486 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:28.538469 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/311279d9-d231-4177-8401-9e9598ed9ef0-metrics-certs\") pod \"maas-api-e2e-ipp-b-460b0b-5d5bdd49b-m8txx\" (UID: \"311279d9-d231-4177-8401-9e9598ed9ef0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-460b0b-5d5bdd49b-m8txx" Oct 30 14:39:28.538549 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:28.538534 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/311279d9-d231-4177-8401-9e9598ed9ef0-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-460b0b-5d5bdd49b-m8txx\" (UID: \"311279d9-d231-4177-8401-9e9598ed9ef0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-460b0b-5d5bdd49b-m8txx" Oct 30 14:39:28.538605 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:28.538591 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/311279d9-d231-4177-8401-9e9598ed9ef0-maas-api-tls\") pod \"maas-api-e2e-ipp-b-460b0b-5d5bdd49b-m8txx\" (UID: \"311279d9-d231-4177-8401-9e9598ed9ef0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-460b0b-5d5bdd49b-m8txx" Oct 30 14:39:28.538675 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:39:28.538659 4982 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-460b0b-serving-cert: secret "maas-api-e2e-ipp-b-460b0b-serving-cert" not found Oct 30 14:39:28.538718 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:39:28.538711 4982 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/311279d9-d231-4177-8401-9e9598ed9ef0-maas-api-tls podName:311279d9-d231-4177-8401-9e9598ed9ef0 nodeName:}" failed. No retries permitted until 2026-10-30 14:39:29.038698286 +0000 UTC m=+3038.782832329 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/311279d9-d231-4177-8401-9e9598ed9ef0-maas-api-tls") pod "maas-api-e2e-ipp-b-460b0b-5d5bdd49b-m8txx" (UID: "311279d9-d231-4177-8401-9e9598ed9ef0") : secret "maas-api-e2e-ipp-b-460b0b-serving-cert" not found Oct 30 14:39:28.539235 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:28.539211 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/311279d9-d231-4177-8401-9e9598ed9ef0-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-460b0b-5d5bdd49b-m8txx\" (UID: \"311279d9-d231-4177-8401-9e9598ed9ef0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-460b0b-5d5bdd49b-m8txx" Oct 30 14:39:28.540969 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:28.540952 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/311279d9-d231-4177-8401-9e9598ed9ef0-metrics-certs\") pod \"maas-api-e2e-ipp-b-460b0b-5d5bdd49b-m8txx\" (UID: \"311279d9-d231-4177-8401-9e9598ed9ef0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-460b0b-5d5bdd49b-m8txx" Oct 30 14:39:28.565629 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:28.565602 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl92b\" (UniqueName: \"kubernetes.io/projected/311279d9-d231-4177-8401-9e9598ed9ef0-kube-api-access-nl92b\") pod \"maas-api-e2e-ipp-b-460b0b-5d5bdd49b-m8txx\" (UID: \"311279d9-d231-4177-8401-9e9598ed9ef0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-460b0b-5d5bdd49b-m8txx" Oct 30 14:39:28.715928 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:28.715842 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:39:28.905600 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:28.905566 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:39:29.045466 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:29.045384 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/311279d9-d231-4177-8401-9e9598ed9ef0-maas-api-tls\") pod \"maas-api-e2e-ipp-b-460b0b-5d5bdd49b-m8txx\" (UID: \"311279d9-d231-4177-8401-9e9598ed9ef0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-460b0b-5d5bdd49b-m8txx" Oct 30 14:39:29.047955 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:29.047930 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/311279d9-d231-4177-8401-9e9598ed9ef0-maas-api-tls\") pod \"maas-api-e2e-ipp-b-460b0b-5d5bdd49b-m8txx\" (UID: \"311279d9-d231-4177-8401-9e9598ed9ef0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-460b0b-5d5bdd49b-m8txx" Oct 30 14:39:29.266298 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:29.266268 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-460b0b-5d5bdd49b-m8txx" Oct 30 14:39:29.594706 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:29.592221 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-460b0b-5d5bdd49b-m8txx"] Oct 30 14:39:30.068136 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:30.068103 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-460b0b-5d5bdd49b-m8txx" event={"ID":"311279d9-d231-4177-8401-9e9598ed9ef0","Type":"ContainerStarted","Data":"3720d60dfeaecd0a2c611d171a537027d76ec593dfe0fd5f74fd95e051fdb9c6"} Oct 30 14:39:30.112291 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:30.112261 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-e371c3/rate-test-model-e371c3-kserve-fb686cfb5-bn68k"] Oct 30 14:39:30.251662 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:30.251095 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:39:30.757232 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:30.757198 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:39:31.076743 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:31.076702 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-460b0b-5d5bdd49b-m8txx" event={"ID":"311279d9-d231-4177-8401-9e9598ed9ef0","Type":"ContainerStarted","Data":"eb5f36da0b2602aa195ed56201e876135b37dae4ad114efdd728c0648b186b90"} Oct 30 14:39:31.076946 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:31.076791 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-460b0b-5d5bdd49b-m8txx" Oct 30 14:39:31.142314 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:31.142265 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-460b0b-5d5bdd49b-m8txx" podStartSLOduration=2.717851323 podStartE2EDuration="3.142250742s" totalImagesPullingTime="424.399419ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:39:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:39:31.139924752 +0000 UTC m=+3040.884058793" watchObservedRunningTime="2026-10-30 14:39:31.142250742 +0000 UTC m=+3040.886384783" Oct 30 14:39:32.764258 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:32.764218 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:39:34.833835 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:34.833799 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:39:36.725560 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:36.725524 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:39:37.089713 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:37.089635 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-460b0b-5d5bdd49b-m8txx" Oct 30 14:39:38.504904 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:38.504865 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:39:39.949372 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:39.949332 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:39:40.912080 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:40.912036 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:39:42.038010 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:42.037967 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb035005-wvc7d"] Oct 30 14:39:42.050978 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:42.050949 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb035005-wvc7d"] Oct 30 14:39:42.400215 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:42.400179 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f3b8815-8ec6-486f-997f-a0bd815c0817" path="/var/lib/kubelet/pods/0f3b8815-8ec6-486f-997f-a0bd815c0817/volumes" Oct 30 14:39:42.693148 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:42.693071 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:39:43.030209 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:43.030123 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-a59ecb/rate-test-model-a59ecb-kserve-8678fc4b5c-pr4bs"] Oct 30 14:39:43.072151 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:43.072118 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:39:43.670242 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:43.670210 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:39:44.260115 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:44.260075 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:39:44.434511 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:44.434432 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:39:44.747443 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:44.747395 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:39:45.140774 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:45.140726 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-e371c3/rate-test-model-e371c3-kserve-fb686cfb5-bn68k" podUID="ec8a22d7-6673-4ebe-aada-d44647d2f005" containerName="main" containerID="cri-o://eaf50028ca79ee5852b13645f6c0707704c0d784f6662a222c567b2987d12c22" gracePeriod=45 Oct 30 14:39:45.468156 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:45.468121 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-e371c3/rate-test-model-e371c3-kserve-fb686cfb5-bn68k" Oct 30 14:39:45.574816 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:45.574778 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec8a22d7-6673-4ebe-aada-d44647d2f005-home\" (UniqueName: \"kubernetes.io/empty-dir/ec8a22d7-6673-4ebe-aada-d44647d2f005-home\") pod \"ec8a22d7-6673-4ebe-aada-d44647d2f005\" (UID: \"ec8a22d7-6673-4ebe-aada-d44647d2f005\") " Oct 30 14:39:45.574959 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:45.574826 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec8a22d7-6673-4ebe-aada-d44647d2f005-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec8a22d7-6673-4ebe-aada-d44647d2f005-dshm\") pod \"ec8a22d7-6673-4ebe-aada-d44647d2f005\" (UID: \"ec8a22d7-6673-4ebe-aada-d44647d2f005\") " Oct 30 14:39:45.574959 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:45.574869 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec8a22d7-6673-4ebe-aada-d44647d2f005-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec8a22d7-6673-4ebe-aada-d44647d2f005-tmp-dir\") pod \"ec8a22d7-6673-4ebe-aada-d44647d2f005\" (UID: \"ec8a22d7-6673-4ebe-aada-d44647d2f005\") " Oct 30 14:39:45.575080 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:45.575009 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec8a22d7-6673-4ebe-aada-d44647d2f005-kube-api-access-hfq8h\" (UniqueName: \"kubernetes.io/projected/ec8a22d7-6673-4ebe-aada-d44647d2f005-kube-api-access-hfq8h\") pod \"ec8a22d7-6673-4ebe-aada-d44647d2f005\" (UID: \"ec8a22d7-6673-4ebe-aada-d44647d2f005\") " Oct 30 14:39:45.575080 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:45.575046 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec8a22d7-6673-4ebe-aada-d44647d2f005-home" pod "ec8a22d7-6673-4ebe-aada-d44647d2f005" (UID: "ec8a22d7-6673-4ebe-aada-d44647d2f005"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:39:45.575205 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:45.575168 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec8a22d7-6673-4ebe-aada-d44647d2f005-tmp-dir" pod "ec8a22d7-6673-4ebe-aada-d44647d2f005" (UID: "ec8a22d7-6673-4ebe-aada-d44647d2f005"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:39:45.575205 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:45.575189 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec8a22d7-6673-4ebe-aada-d44647d2f005-tls-certs\" (UniqueName: \"kubernetes.io/secret/ec8a22d7-6673-4ebe-aada-d44647d2f005-tls-certs\") pod \"ec8a22d7-6673-4ebe-aada-d44647d2f005\" (UID: \"ec8a22d7-6673-4ebe-aada-d44647d2f005\") " Oct 30 14:39:45.575369 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:45.575349 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec8a22d7-6673-4ebe-aada-d44647d2f005-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec8a22d7-6673-4ebe-aada-d44647d2f005-model-cache\") pod \"ec8a22d7-6673-4ebe-aada-d44647d2f005\" (UID: \"ec8a22d7-6673-4ebe-aada-d44647d2f005\") " Oct 30 14:39:45.575775 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:45.575745 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec8a22d7-6673-4ebe-aada-d44647d2f005-model-cache" pod "ec8a22d7-6673-4ebe-aada-d44647d2f005" (UID: "ec8a22d7-6673-4ebe-aada-d44647d2f005"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:39:45.576052 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:45.576024 4982 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec8a22d7-6673-4ebe-aada-d44647d2f005-home\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:39:45.576173 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:45.576055 4982 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec8a22d7-6673-4ebe-aada-d44647d2f005-tmp-dir\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:39:45.576173 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:45.576072 4982 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec8a22d7-6673-4ebe-aada-d44647d2f005-model-cache\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:39:45.577302 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:45.577274 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec8a22d7-6673-4ebe-aada-d44647d2f005-dshm" pod "ec8a22d7-6673-4ebe-aada-d44647d2f005" (UID: "ec8a22d7-6673-4ebe-aada-d44647d2f005"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:39:45.578209 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:45.578184 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec8a22d7-6673-4ebe-aada-d44647d2f005-tls-certs" pod "ec8a22d7-6673-4ebe-aada-d44647d2f005" (UID: "ec8a22d7-6673-4ebe-aada-d44647d2f005"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:39:45.578209 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:45.578200 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec8a22d7-6673-4ebe-aada-d44647d2f005-kube-api-access-hfq8h" pod "ec8a22d7-6673-4ebe-aada-d44647d2f005" (UID: "ec8a22d7-6673-4ebe-aada-d44647d2f005"). InnerVolumeSpecName "kube-api-access-hfq8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:39:45.677039 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:45.677003 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfq8h\" (UniqueName: \"kubernetes.io/projected/ec8a22d7-6673-4ebe-aada-d44647d2f005-kube-api-access-hfq8h\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:39:45.677039 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:45.677035 4982 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec8a22d7-6673-4ebe-aada-d44647d2f005-tls-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:39:45.677039 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:45.677046 4982 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec8a22d7-6673-4ebe-aada-d44647d2f005-dshm\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:39:46.207928 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:46.207892 4982 generic.go:356] "Generic (PLEG): container finished" podID="ec8a22d7-6673-4ebe-aada-d44647d2f005" containerID="eaf50028ca79ee5852b13645f6c0707704c0d784f6662a222c567b2987d12c22" exitCode=0 Oct 30 14:39:46.208130 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:46.207948 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-e371c3/rate-test-model-e371c3-kserve-fb686cfb5-bn68k" Oct 30 14:39:46.208130 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:46.207974 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-e371c3/rate-test-model-e371c3-kserve-fb686cfb5-bn68k" event={"ID":"ec8a22d7-6673-4ebe-aada-d44647d2f005","Type":"ContainerDied","Data":"eaf50028ca79ee5852b13645f6c0707704c0d784f6662a222c567b2987d12c22"} Oct 30 14:39:46.208130 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:46.208030 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-e371c3/rate-test-model-e371c3-kserve-fb686cfb5-bn68k" event={"ID":"ec8a22d7-6673-4ebe-aada-d44647d2f005","Type":"ContainerDied","Data":"5857eb274be8566f8d3355ab7e9e0cd9ae964819c5f1a71cd8cd8a0cb3ec4017"} Oct 30 14:39:46.208130 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:46.208057 4982 scope.go:122] "RemoveContainer" containerID="eaf50028ca79ee5852b13645f6c0707704c0d784f6662a222c567b2987d12c22" Oct 30 14:39:46.222206 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:46.222177 4982 scope.go:122] "RemoveContainer" containerID="eaf50028ca79ee5852b13645f6c0707704c0d784f6662a222c567b2987d12c22" Oct 30 14:39:46.222513 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:39:46.222483 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaf50028ca79ee5852b13645f6c0707704c0d784f6662a222c567b2987d12c22\": container with ID starting with eaf50028ca79ee5852b13645f6c0707704c0d784f6662a222c567b2987d12c22 not found: ID does not exist" containerID="eaf50028ca79ee5852b13645f6c0707704c0d784f6662a222c567b2987d12c22" Oct 30 14:39:46.222618 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:46.222522 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaf50028ca79ee5852b13645f6c0707704c0d784f6662a222c567b2987d12c22"} err="failed to get container status \"eaf50028ca79ee5852b13645f6c0707704c0d784f6662a222c567b2987d12c22\": rpc error: code = NotFound desc = could not find container \"eaf50028ca79ee5852b13645f6c0707704c0d784f6662a222c567b2987d12c22\": container with ID starting with eaf50028ca79ee5852b13645f6c0707704c0d784f6662a222c567b2987d12c22 not found: ID does not exist" Oct 30 14:39:46.272780 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:46.272740 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-e371c3/rate-test-model-e371c3-kserve-fb686cfb5-bn68k"] Oct 30 14:39:46.284832 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:46.284793 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-e371c3/rate-test-model-e371c3-kserve-fb686cfb5-bn68k"] Oct 30 14:39:46.402580 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:46.402546 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec8a22d7-6673-4ebe-aada-d44647d2f005" path="/var/lib/kubelet/pods/ec8a22d7-6673-4ebe-aada-d44647d2f005/volumes" Oct 30 14:39:46.517184 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:46.517112 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:39:46.749790 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:46.749753 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:39:48.682861 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:48.682825 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:39:50.651292 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:50.651224 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:39:52.001893 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:52.001855 4982 scope.go:122] "RemoveContainer" containerID="7c8cdaa77d121b2b75e2b8ab3ae5d2c77b28e0253583ba6cf28c9d36915cdcb3" Oct 30 14:39:52.014225 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:52.014201 4982 scope.go:122] "RemoveContainer" containerID="ec5aaec370b4d4fc4276ba6c9306319b043c9d9eb1808fcc812f9c46548b5ea3" Oct 30 14:39:52.026200 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:52.026174 4982 scope.go:122] "RemoveContainer" containerID="67580ea999e193191948f43619c552c074b802a27b9a3003caf6e39389186969" Oct 30 14:39:52.039431 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:52.039406 4982 scope.go:122] "RemoveContainer" containerID="9848c739c54ccfeadcb7297f715799b332288bbb605ca1a9c06d9fd719956df8" Oct 30 14:39:52.042455 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:52.042428 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d8e34e9-5r5zs"] Oct 30 14:39:52.051212 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:52.051186 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d8e34e9-5r5zs"] Oct 30 14:39:52.401632 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:52.401596 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01d8ac6a-ba15-48c1-b33a-7b18910666e7" path="/var/lib/kubelet/pods/01d8ac6a-ba15-48c1-b33a-7b18910666e7/volumes" Oct 30 14:39:52.740958 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:52.740877 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:39:55.234040 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:55.234001 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:39:56.888105 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:56.888067 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-33c209-56ff9dc7df-lr8ft"] Oct 30 14:39:56.889048 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:56.889025 4982 cpu_manager.go:414] "containerMap: removing container" podUID="ec8a22d7-6673-4ebe-aada-d44647d2f005" containerName="main" Oct 30 14:39:56.889120 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:56.889050 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec8a22d7-6673-4ebe-aada-d44647d2f005" containerName="main" Oct 30 14:39:56.889224 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:56.889212 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec8a22d7-6673-4ebe-aada-d44647d2f005" containerName="main" Oct 30 14:39:56.893666 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:56.893632 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33c209-56ff9dc7df-lr8ft" Oct 30 14:39:56.916946 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:56.916897 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-33c209-56ff9dc7df-lr8ft"] Oct 30 14:39:56.922740 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:56.922692 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-33c209-serving-cert\"" Oct 30 14:39:57.014154 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:57.014113 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:39:57.017609 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:57.017577 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6c9e188-d466-4ad8-b72c-f89717ff9f47-maas-api-tls\") pod \"maas-api-e2e-ipp-a-33c209-56ff9dc7df-lr8ft\" (UID: \"e6c9e188-d466-4ad8-b72c-f89717ff9f47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33c209-56ff9dc7df-lr8ft" Oct 30 14:39:57.017753 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:57.017733 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klp2j\" (UniqueName: \"kubernetes.io/projected/e6c9e188-d466-4ad8-b72c-f89717ff9f47-kube-api-access-klp2j\") pod \"maas-api-e2e-ipp-a-33c209-56ff9dc7df-lr8ft\" (UID: \"e6c9e188-d466-4ad8-b72c-f89717ff9f47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33c209-56ff9dc7df-lr8ft" Oct 30 14:39:57.017801 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:57.017788 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6c9e188-d466-4ad8-b72c-f89717ff9f47-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-33c209-56ff9dc7df-lr8ft\" (UID: \"e6c9e188-d466-4ad8-b72c-f89717ff9f47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33c209-56ff9dc7df-lr8ft" Oct 30 14:39:57.017872 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:57.017858 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6c9e188-d466-4ad8-b72c-f89717ff9f47-metrics-certs\") pod \"maas-api-e2e-ipp-a-33c209-56ff9dc7df-lr8ft\" (UID: \"e6c9e188-d466-4ad8-b72c-f89717ff9f47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33c209-56ff9dc7df-lr8ft" Oct 30 14:39:57.119353 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:57.119314 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6c9e188-d466-4ad8-b72c-f89717ff9f47-maas-api-tls\") pod \"maas-api-e2e-ipp-a-33c209-56ff9dc7df-lr8ft\" (UID: \"e6c9e188-d466-4ad8-b72c-f89717ff9f47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33c209-56ff9dc7df-lr8ft" Oct 30 14:39:57.119607 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:57.119425 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klp2j\" (UniqueName: \"kubernetes.io/projected/e6c9e188-d466-4ad8-b72c-f89717ff9f47-kube-api-access-klp2j\") pod \"maas-api-e2e-ipp-a-33c209-56ff9dc7df-lr8ft\" (UID: \"e6c9e188-d466-4ad8-b72c-f89717ff9f47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33c209-56ff9dc7df-lr8ft" Oct 30 14:39:57.119607 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:57.119456 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6c9e188-d466-4ad8-b72c-f89717ff9f47-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-33c209-56ff9dc7df-lr8ft\" (UID: \"e6c9e188-d466-4ad8-b72c-f89717ff9f47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33c209-56ff9dc7df-lr8ft" Oct 30 14:39:57.119607 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:57.119530 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6c9e188-d466-4ad8-b72c-f89717ff9f47-metrics-certs\") pod \"maas-api-e2e-ipp-a-33c209-56ff9dc7df-lr8ft\" (UID: \"e6c9e188-d466-4ad8-b72c-f89717ff9f47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33c209-56ff9dc7df-lr8ft" Oct 30 14:39:57.120326 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:57.120300 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6c9e188-d466-4ad8-b72c-f89717ff9f47-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-33c209-56ff9dc7df-lr8ft\" (UID: \"e6c9e188-d466-4ad8-b72c-f89717ff9f47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33c209-56ff9dc7df-lr8ft" Oct 30 14:39:57.122131 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:57.122097 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6c9e188-d466-4ad8-b72c-f89717ff9f47-maas-api-tls\") pod \"maas-api-e2e-ipp-a-33c209-56ff9dc7df-lr8ft\" (UID: \"e6c9e188-d466-4ad8-b72c-f89717ff9f47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33c209-56ff9dc7df-lr8ft" Oct 30 14:39:57.122445 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:57.122418 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6c9e188-d466-4ad8-b72c-f89717ff9f47-metrics-certs\") pod \"maas-api-e2e-ipp-a-33c209-56ff9dc7df-lr8ft\" (UID: \"e6c9e188-d466-4ad8-b72c-f89717ff9f47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33c209-56ff9dc7df-lr8ft" Oct 30 14:39:57.160974 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:57.160918 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klp2j\" (UniqueName: \"kubernetes.io/projected/e6c9e188-d466-4ad8-b72c-f89717ff9f47-kube-api-access-klp2j\") pod \"maas-api-e2e-ipp-a-33c209-56ff9dc7df-lr8ft\" (UID: \"e6c9e188-d466-4ad8-b72c-f89717ff9f47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33c209-56ff9dc7df-lr8ft" Oct 30 14:39:57.199822 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:57.199797 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:39:57.207923 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:57.207891 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33c209-56ff9dc7df-lr8ft" Oct 30 14:39:57.341461 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:57.341417 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:39:57.523787 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:57.523762 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-33c209-56ff9dc7df-lr8ft"] Oct 30 14:39:57.525896 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:39:57.525872 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6c9e188_d466_4ad8_b72c_f89717ff9f47.slice/crio-011f26e0fd514ef4f92bc8beaf4569075b562ada99227f1c203b65419b3cffa9 WatchSource:0}: Error finding container 011f26e0fd514ef4f92bc8beaf4569075b562ada99227f1c203b65419b3cffa9: Status 404 returned error can't find the container with id 011f26e0fd514ef4f92bc8beaf4569075b562ada99227f1c203b65419b3cffa9 Oct 30 14:39:57.911705 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:57.911674 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:39:58.056744 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:58.055960 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-a59ecb/rate-test-model-a59ecb-kserve-8678fc4b5c-pr4bs" podUID="8cb47566-b4c9-4384-a2e8-f3428ad8b9a4" containerName="main" containerID="cri-o://d8ed49bd890a704717ac3adead502823c735eff30b6fbb2f47b0612c15c091d0" gracePeriod=45 Oct 30 14:39:58.060908 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:39:58.060872 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:39:58.323017 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:58.321751 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33c209-56ff9dc7df-lr8ft" event={"ID":"e6c9e188-d466-4ad8-b72c-f89717ff9f47","Type":"ContainerStarted","Data":"952d091925135db74f36260a933e533634d7613a5d59e50705d5dae69c9330c9"} Oct 30 14:39:58.323017 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:58.321814 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33c209-56ff9dc7df-lr8ft" event={"ID":"e6c9e188-d466-4ad8-b72c-f89717ff9f47","Type":"ContainerStarted","Data":"011f26e0fd514ef4f92bc8beaf4569075b562ada99227f1c203b65419b3cffa9"} Oct 30 14:39:58.323292 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:58.323110 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33c209-56ff9dc7df-lr8ft" Oct 30 14:39:58.330658 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:58.325900 4982 generic.go:356] "Generic (PLEG): container finished" podID="8cb47566-b4c9-4384-a2e8-f3428ad8b9a4" containerID="d8ed49bd890a704717ac3adead502823c735eff30b6fbb2f47b0612c15c091d0" exitCode=0 Oct 30 14:39:58.330658 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:58.325957 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-a59ecb/rate-test-model-a59ecb-kserve-8678fc4b5c-pr4bs" event={"ID":"8cb47566-b4c9-4384-a2e8-f3428ad8b9a4","Type":"ContainerDied","Data":"d8ed49bd890a704717ac3adead502823c735eff30b6fbb2f47b0612c15c091d0"} Oct 30 14:39:58.403628 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:58.403508 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33c209-56ff9dc7df-lr8ft" podStartSLOduration=2.071988569 podStartE2EDuration="2.403488785s" totalImagesPullingTime="331.500216ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:39:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:39:58.373025797 +0000 UTC m=+3068.117159856" watchObservedRunningTime="2026-10-30 14:39:58.403488785 +0000 UTC m=+3068.147622826" Oct 30 14:39:58.429833 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:58.429416 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-a59ecb/rate-test-model-a59ecb-kserve-8678fc4b5c-pr4bs" Oct 30 14:39:58.543391 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:58.543344 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cb47566-b4c9-4384-a2e8-f3428ad8b9a4-kube-api-access-z28vd\" (UniqueName: \"kubernetes.io/projected/8cb47566-b4c9-4384-a2e8-f3428ad8b9a4-kube-api-access-z28vd\") pod \"8cb47566-b4c9-4384-a2e8-f3428ad8b9a4\" (UID: \"8cb47566-b4c9-4384-a2e8-f3428ad8b9a4\") " Oct 30 14:39:58.547259 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:58.547214 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cb47566-b4c9-4384-a2e8-f3428ad8b9a4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cb47566-b4c9-4384-a2e8-f3428ad8b9a4-tmp-dir\") pod \"8cb47566-b4c9-4384-a2e8-f3428ad8b9a4\" (UID: \"8cb47566-b4c9-4384-a2e8-f3428ad8b9a4\") " Oct 30 14:39:58.547597 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:58.547544 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cb47566-b4c9-4384-a2e8-f3428ad8b9a4-tls-certs\" (UniqueName: \"kubernetes.io/secret/8cb47566-b4c9-4384-a2e8-f3428ad8b9a4-tls-certs\") pod \"8cb47566-b4c9-4384-a2e8-f3428ad8b9a4\" (UID: \"8cb47566-b4c9-4384-a2e8-f3428ad8b9a4\") " Oct 30 14:39:58.547796 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:58.547780 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cb47566-b4c9-4384-a2e8-f3428ad8b9a4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cb47566-b4c9-4384-a2e8-f3428ad8b9a4-dshm\") pod \"8cb47566-b4c9-4384-a2e8-f3428ad8b9a4\" (UID: \"8cb47566-b4c9-4384-a2e8-f3428ad8b9a4\") " Oct 30 14:39:58.548085 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:58.548070 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cb47566-b4c9-4384-a2e8-f3428ad8b9a4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cb47566-b4c9-4384-a2e8-f3428ad8b9a4-model-cache\") pod \"8cb47566-b4c9-4384-a2e8-f3428ad8b9a4\" (UID: \"8cb47566-b4c9-4384-a2e8-f3428ad8b9a4\") " Oct 30 14:39:58.548348 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:58.548322 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cb47566-b4c9-4384-a2e8-f3428ad8b9a4-home\" (UniqueName: \"kubernetes.io/empty-dir/8cb47566-b4c9-4384-a2e8-f3428ad8b9a4-home\") pod \"8cb47566-b4c9-4384-a2e8-f3428ad8b9a4\" (UID: \"8cb47566-b4c9-4384-a2e8-f3428ad8b9a4\") " Oct 30 14:39:58.551195 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:58.547606 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cb47566-b4c9-4384-a2e8-f3428ad8b9a4-tmp-dir" pod "8cb47566-b4c9-4384-a2e8-f3428ad8b9a4" (UID: "8cb47566-b4c9-4384-a2e8-f3428ad8b9a4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:39:58.551573 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:58.551553 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cb47566-b4c9-4384-a2e8-f3428ad8b9a4-model-cache" pod "8cb47566-b4c9-4384-a2e8-f3428ad8b9a4" (UID: "8cb47566-b4c9-4384-a2e8-f3428ad8b9a4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:39:58.551709 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:58.548138 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:39:58.553294 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:58.552736 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cb47566-b4c9-4384-a2e8-f3428ad8b9a4-home" pod "8cb47566-b4c9-4384-a2e8-f3428ad8b9a4" (UID: "8cb47566-b4c9-4384-a2e8-f3428ad8b9a4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:39:58.556472 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:58.556446 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cb47566-b4c9-4384-a2e8-f3428ad8b9a4-kube-api-access-z28vd" pod "8cb47566-b4c9-4384-a2e8-f3428ad8b9a4" (UID: "8cb47566-b4c9-4384-a2e8-f3428ad8b9a4"). InnerVolumeSpecName "kube-api-access-z28vd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:39:58.557006 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:58.556988 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cb47566-b4c9-4384-a2e8-f3428ad8b9a4-dshm" pod "8cb47566-b4c9-4384-a2e8-f3428ad8b9a4" (UID: "8cb47566-b4c9-4384-a2e8-f3428ad8b9a4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:39:58.557336 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:58.557317 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cb47566-b4c9-4384-a2e8-f3428ad8b9a4-tls-certs" pod "8cb47566-b4c9-4384-a2e8-f3428ad8b9a4" (UID: "8cb47566-b4c9-4384-a2e8-f3428ad8b9a4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:39:58.650342 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:58.650316 4982 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cb47566-b4c9-4384-a2e8-f3428ad8b9a4-tls-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:39:58.650342 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:58.650342 4982 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cb47566-b4c9-4384-a2e8-f3428ad8b9a4-dshm\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:39:58.650507 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:58.650351 4982 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cb47566-b4c9-4384-a2e8-f3428ad8b9a4-model-cache\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:39:58.650507 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:58.650360 4982 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cb47566-b4c9-4384-a2e8-f3428ad8b9a4-home\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:39:58.650507 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:58.650369 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z28vd\" (UniqueName: \"kubernetes.io/projected/8cb47566-b4c9-4384-a2e8-f3428ad8b9a4-kube-api-access-z28vd\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:39:58.650507 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:58.650377 4982 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cb47566-b4c9-4384-a2e8-f3428ad8b9a4-tmp-dir\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:39:58.920289 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:58.920244 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5"] Oct 30 14:39:58.921821 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:58.921791 4982 cpu_manager.go:414] "containerMap: removing container" podUID="8cb47566-b4c9-4384-a2e8-f3428ad8b9a4" containerName="main" Oct 30 14:39:58.921821 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:58.921821 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cb47566-b4c9-4384-a2e8-f3428ad8b9a4" containerName="main" Oct 30 14:39:58.922060 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:58.922044 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cb47566-b4c9-4384-a2e8-f3428ad8b9a4" containerName="main" Oct 30 14:39:58.933947 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:58.933922 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5" Oct 30 14:39:58.953902 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:58.953876 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5"] Oct 30 14:39:58.962624 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:58.962597 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-a1854b-openshift-default-dockercfg-8wvxp\"" Oct 30 14:39:58.996563 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:58.996530 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:39:59.057430 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:59.057390 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cbc37dd-65f1-40f7-8b5a-0ab581c99f86-workload-socket\") pod \"e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5\" (UID: \"7cbc37dd-65f1-40f7-8b5a-0ab581c99f86\") " pod="openshift-ingress/e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5" Oct 30 14:39:59.062667 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:59.057813 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cbc37dd-65f1-40f7-8b5a-0ab581c99f86-istio-envoy\") pod \"e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5\" (UID: \"7cbc37dd-65f1-40f7-8b5a-0ab581c99f86\") " pod="openshift-ingress/e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5" Oct 30 14:39:59.062920 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:59.062893 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgb42\" (UniqueName: \"kubernetes.io/projected/7cbc37dd-65f1-40f7-8b5a-0ab581c99f86-kube-api-access-cgb42\") pod \"e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5\" (UID: \"7cbc37dd-65f1-40f7-8b5a-0ab581c99f86\") " pod="openshift-ingress/e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5" Oct 30 14:39:59.071878 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:59.063083 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cbc37dd-65f1-40f7-8b5a-0ab581c99f86-istio-token\") pod \"e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5\" (UID: \"7cbc37dd-65f1-40f7-8b5a-0ab581c99f86\") " pod="openshift-ingress/e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5" Oct 30 14:39:59.074711 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:59.074628 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cbc37dd-65f1-40f7-8b5a-0ab581c99f86-istio-data\") pod \"e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5\" (UID: \"7cbc37dd-65f1-40f7-8b5a-0ab581c99f86\") " pod="openshift-ingress/e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5" Oct 30 14:39:59.110455 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:59.110420 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cbc37dd-65f1-40f7-8b5a-0ab581c99f86-credential-socket\") pod \"e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5\" (UID: \"7cbc37dd-65f1-40f7-8b5a-0ab581c99f86\") " pod="openshift-ingress/e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5" Oct 30 14:39:59.110789 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:59.110767 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cbc37dd-65f1-40f7-8b5a-0ab581c99f86-istiod-ca-cert\") pod \"e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5\" (UID: \"7cbc37dd-65f1-40f7-8b5a-0ab581c99f86\") " pod="openshift-ingress/e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5" Oct 30 14:39:59.111043 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:59.111014 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cbc37dd-65f1-40f7-8b5a-0ab581c99f86-workload-certs\") pod \"e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5\" (UID: \"7cbc37dd-65f1-40f7-8b5a-0ab581c99f86\") " pod="openshift-ingress/e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5" Oct 30 14:39:59.111343 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:59.111326 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cbc37dd-65f1-40f7-8b5a-0ab581c99f86-istio-podinfo\") pod \"e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5\" (UID: \"7cbc37dd-65f1-40f7-8b5a-0ab581c99f86\") " pod="openshift-ingress/e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5" Oct 30 14:39:59.212624 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:59.212418 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cbc37dd-65f1-40f7-8b5a-0ab581c99f86-credential-socket\") pod \"e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5\" (UID: \"7cbc37dd-65f1-40f7-8b5a-0ab581c99f86\") " pod="openshift-ingress/e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5" Oct 30 14:39:59.212624 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:59.212478 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cbc37dd-65f1-40f7-8b5a-0ab581c99f86-istiod-ca-cert\") pod \"e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5\" (UID: \"7cbc37dd-65f1-40f7-8b5a-0ab581c99f86\") " pod="openshift-ingress/e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5" Oct 30 14:39:59.212624 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:59.212546 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cbc37dd-65f1-40f7-8b5a-0ab581c99f86-workload-certs\") pod \"e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5\" (UID: \"7cbc37dd-65f1-40f7-8b5a-0ab581c99f86\") " pod="openshift-ingress/e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5" Oct 30 14:39:59.212962 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:59.212684 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cbc37dd-65f1-40f7-8b5a-0ab581c99f86-istio-podinfo\") pod \"e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5\" (UID: \"7cbc37dd-65f1-40f7-8b5a-0ab581c99f86\") " pod="openshift-ingress/e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5" Oct 30 14:39:59.212962 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:59.212811 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cbc37dd-65f1-40f7-8b5a-0ab581c99f86-workload-socket\") pod \"e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5\" (UID: \"7cbc37dd-65f1-40f7-8b5a-0ab581c99f86\") " pod="openshift-ingress/e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5" Oct 30 14:39:59.212962 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:59.212858 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cbc37dd-65f1-40f7-8b5a-0ab581c99f86-istio-envoy\") pod \"e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5\" (UID: \"7cbc37dd-65f1-40f7-8b5a-0ab581c99f86\") " pod="openshift-ingress/e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5" Oct 30 14:39:59.212962 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:59.212880 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cbc37dd-65f1-40f7-8b5a-0ab581c99f86-credential-socket\") pod \"e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5\" (UID: \"7cbc37dd-65f1-40f7-8b5a-0ab581c99f86\") " pod="openshift-ingress/e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5" Oct 30 14:39:59.212962 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:59.212893 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgb42\" (UniqueName: \"kubernetes.io/projected/7cbc37dd-65f1-40f7-8b5a-0ab581c99f86-kube-api-access-cgb42\") pod \"e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5\" (UID: \"7cbc37dd-65f1-40f7-8b5a-0ab581c99f86\") " pod="openshift-ingress/e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5" Oct 30 14:39:59.212962 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:59.212947 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cbc37dd-65f1-40f7-8b5a-0ab581c99f86-istio-token\") pod \"e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5\" (UID: \"7cbc37dd-65f1-40f7-8b5a-0ab581c99f86\") " pod="openshift-ingress/e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5" Oct 30 14:39:59.213266 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:59.212991 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cbc37dd-65f1-40f7-8b5a-0ab581c99f86-istio-data\") pod \"e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5\" (UID: \"7cbc37dd-65f1-40f7-8b5a-0ab581c99f86\") " pod="openshift-ingress/e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5" Oct 30 14:39:59.213386 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:59.213361 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cbc37dd-65f1-40f7-8b5a-0ab581c99f86-istio-data\") pod \"e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5\" (UID: \"7cbc37dd-65f1-40f7-8b5a-0ab581c99f86\") " pod="openshift-ingress/e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5" Oct 30 14:39:59.214339 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:59.213876 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cbc37dd-65f1-40f7-8b5a-0ab581c99f86-istiod-ca-cert\") pod \"e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5\" (UID: \"7cbc37dd-65f1-40f7-8b5a-0ab581c99f86\") " pod="openshift-ingress/e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5" Oct 30 14:39:59.214473 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:59.214174 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cbc37dd-65f1-40f7-8b5a-0ab581c99f86-workload-certs\") pod \"e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5\" (UID: \"7cbc37dd-65f1-40f7-8b5a-0ab581c99f86\") " pod="openshift-ingress/e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5" Oct 30 14:39:59.214556 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:59.214291 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cbc37dd-65f1-40f7-8b5a-0ab581c99f86-workload-socket\") pod \"e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5\" (UID: \"7cbc37dd-65f1-40f7-8b5a-0ab581c99f86\") " pod="openshift-ingress/e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5" Oct 30 14:39:59.216305 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:59.216279 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cbc37dd-65f1-40f7-8b5a-0ab581c99f86-istio-envoy\") pod \"e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5\" (UID: \"7cbc37dd-65f1-40f7-8b5a-0ab581c99f86\") " pod="openshift-ingress/e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5" Oct 30 14:39:59.217209 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:59.217183 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cbc37dd-65f1-40f7-8b5a-0ab581c99f86-istio-podinfo\") pod \"e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5\" (UID: \"7cbc37dd-65f1-40f7-8b5a-0ab581c99f86\") " pod="openshift-ingress/e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5" Oct 30 14:39:59.226175 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:59.226150 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:39:59.239523 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:59.239493 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cbc37dd-65f1-40f7-8b5a-0ab581c99f86-istio-token\") pod \"e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5\" (UID: \"7cbc37dd-65f1-40f7-8b5a-0ab581c99f86\") " pod="openshift-ingress/e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5" Oct 30 14:39:59.239821 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:59.239804 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgb42\" (UniqueName: \"kubernetes.io/projected/7cbc37dd-65f1-40f7-8b5a-0ab581c99f86-kube-api-access-cgb42\") pod \"e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5\" (UID: \"7cbc37dd-65f1-40f7-8b5a-0ab581c99f86\") " pod="openshift-ingress/e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5" Oct 30 14:39:59.250201 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:59.250180 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5" Oct 30 14:39:59.265896 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:59.265846 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-a59ecb/rate-test-model-a59ecb-kserve-8678fc4b5c-pr4bs" podUID="8cb47566-b4c9-4384-a2e8-f3428ad8b9a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": context deadline exceeded" Oct 30 14:39:59.342117 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:59.341921 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-a59ecb/rate-test-model-a59ecb-kserve-8678fc4b5c-pr4bs" event={"ID":"8cb47566-b4c9-4384-a2e8-f3428ad8b9a4","Type":"ContainerDied","Data":"2eb82e7e0b1f94b3f232ff6f038abda059f0f6eadc92a9e19694f222487479e4"} Oct 30 14:39:59.342117 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:59.341986 4982 scope.go:122] "RemoveContainer" containerID="d8ed49bd890a704717ac3adead502823c735eff30b6fbb2f47b0612c15c091d0" Oct 30 14:39:59.342117 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:59.341932 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-a59ecb/rate-test-model-a59ecb-kserve-8678fc4b5c-pr4bs" Oct 30 14:39:59.406990 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:59.406923 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-a59ecb/rate-test-model-a59ecb-kserve-8678fc4b5c-pr4bs"] Oct 30 14:39:59.421549 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:59.420161 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-a59ecb/rate-test-model-a59ecb-kserve-8678fc4b5c-pr4bs"] Oct 30 14:39:59.513188 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:59.512923 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:39:59.607618 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:59.607589 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5"] Oct 30 14:39:59.608953 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:39:59.608926 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cbc37dd_65f1_40f7_8b5a_0ab581c99f86.slice/crio-4a21489fa65866af64c4afd46abd7699d4f221e0c27dd04931c47039bc5e581c WatchSource:0}: Error finding container 4a21489fa65866af64c4afd46abd7699d4f221e0c27dd04931c47039bc5e581c: Status 404 returned error can't find the container with id 4a21489fa65866af64c4afd46abd7699d4f221e0c27dd04931c47039bc5e581c Oct 30 14:39:59.612073 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:59.612037 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:39:59.612168 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:59.612108 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:39:59.612168 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:39:59.612144 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:40:00.348491 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:00.348444 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5" event={"ID":"7cbc37dd-65f1-40f7-8b5a-0ab581c99f86","Type":"ContainerStarted","Data":"42d6a9828625667b4d322e348b24ce5bae9dd7647f0ad389a0c84dcc9275d6ff"} Oct 30 14:40:00.348980 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:00.348557 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5" event={"ID":"7cbc37dd-65f1-40f7-8b5a-0ab581c99f86","Type":"ContainerStarted","Data":"4a21489fa65866af64c4afd46abd7699d4f221e0c27dd04931c47039bc5e581c"} Oct 30 14:40:00.404359 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:00.404226 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5" podStartSLOduration=2.404208144 podStartE2EDuration="2.404208144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:39:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:40:00.399854957 +0000 UTC m=+3070.143989001" watchObservedRunningTime="2026-10-30 14:40:00.404208144 +0000 UTC m=+3070.148342187" Oct 30 14:40:00.408260 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:00.408229 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cb47566-b4c9-4384-a2e8-f3428ad8b9a4" path="/var/lib/kubelet/pods/8cb47566-b4c9-4384-a2e8-f3428ad8b9a4/volumes" Oct 30 14:40:01.074001 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:01.073954 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:40:01.251442 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:01.251410 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5" Oct 30 14:40:01.257906 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:01.257878 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5" Oct 30 14:40:01.356296 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:01.356218 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5" Oct 30 14:40:01.357209 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:01.357190 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5" Oct 30 14:40:02.475769 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:02.475739 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:40:02.908404 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:02.908366 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:40:03.114815 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:03.114777 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:40:03.307412 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:03.307375 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:40:04.602190 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:04.602156 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:40:04.673352 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:04.673284 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a1854b-6bd77d5846-984lk"] Oct 30 14:40:04.678882 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:04.678862 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a1854b-6bd77d5846-984lk" Oct 30 14:40:04.701528 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:04.701502 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a1854b-6bd77d5846-984lk"] Oct 30 14:40:04.712038 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:04.709738 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-a1854b\"" Oct 30 14:40:04.712038 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:04.709836 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-a1854b-dockercfg-dt6nt\"" Oct 30 14:40:04.763056 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:04.763029 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:40:04.793079 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:04.793046 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gv4t\" (UniqueName: \"kubernetes.io/projected/3d7690af-86af-4404-a598-852078e649fa-kube-api-access-4gv4t\") pod \"payload-processing-e2e-ipp-a-a1854b-6bd77d5846-984lk\" (UID: \"3d7690af-86af-4404-a598-852078e649fa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a1854b-6bd77d5846-984lk" Oct 30 14:40:04.793267 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:04.793140 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d7690af-86af-4404-a598-852078e649fa-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a1854b-6bd77d5846-984lk\" (UID: \"3d7690af-86af-4404-a598-852078e649fa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a1854b-6bd77d5846-984lk" Oct 30 14:40:04.903663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:04.894560 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gv4t\" (UniqueName: \"kubernetes.io/projected/3d7690af-86af-4404-a598-852078e649fa-kube-api-access-4gv4t\") pod \"payload-processing-e2e-ipp-a-a1854b-6bd77d5846-984lk\" (UID: \"3d7690af-86af-4404-a598-852078e649fa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a1854b-6bd77d5846-984lk" Oct 30 14:40:04.903663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:04.894659 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d7690af-86af-4404-a598-852078e649fa-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a1854b-6bd77d5846-984lk\" (UID: \"3d7690af-86af-4404-a598-852078e649fa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a1854b-6bd77d5846-984lk" Oct 30 14:40:04.903663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:04.895803 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d7690af-86af-4404-a598-852078e649fa-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a1854b-6bd77d5846-984lk\" (UID: \"3d7690af-86af-4404-a598-852078e649fa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a1854b-6bd77d5846-984lk" Oct 30 14:40:04.928707 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:04.927796 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gv4t\" (UniqueName: \"kubernetes.io/projected/3d7690af-86af-4404-a598-852078e649fa-kube-api-access-4gv4t\") pod \"payload-processing-e2e-ipp-a-a1854b-6bd77d5846-984lk\" (UID: \"3d7690af-86af-4404-a598-852078e649fa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a1854b-6bd77d5846-984lk" Oct 30 14:40:04.995374 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:04.995334 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a1854b-6bd77d5846-984lk" Oct 30 14:40:05.239978 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:05.239888 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:40:05.328392 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:40:05.328355 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d7690af_86af_4404_a598_852078e649fa.slice/crio-d9090736e3bafc8f783548acc3bb3aacfa001cd8b1913945fc4bc36846a63c35 WatchSource:0}: Error finding container d9090736e3bafc8f783548acc3bb3aacfa001cd8b1913945fc4bc36846a63c35: Status 404 returned error can't find the container with id d9090736e3bafc8f783548acc3bb3aacfa001cd8b1913945fc4bc36846a63c35 Oct 30 14:40:05.328567 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:05.328413 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a1854b-6bd77d5846-984lk"] Oct 30 14:40:05.363799 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:05.363774 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33c209-56ff9dc7df-lr8ft" Oct 30 14:40:05.389563 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:05.389534 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a1854b-6bd77d5846-984lk" event={"ID":"3d7690af-86af-4404-a598-852078e649fa","Type":"ContainerStarted","Data":"d9090736e3bafc8f783548acc3bb3aacfa001cd8b1913945fc4bc36846a63c35"} Oct 30 14:40:05.390750 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:05.390726 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:40:05.567147 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:05.567110 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:40:05.725759 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:05.721486 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:40:06.401702 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:06.401665 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a1854b-6bd77d5846-984lk" event={"ID":"3d7690af-86af-4404-a598-852078e649fa","Type":"ContainerStarted","Data":"e209456e2bdca4070b2f92abb353d330e2f4930962be0dcafb028f0e46f4f802"} Oct 30 14:40:06.401862 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:06.401732 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a1854b-6bd77d5846-984lk" Oct 30 14:40:06.452462 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:06.452405 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-a1854b-6bd77d5846-984lk" podStartSLOduration=2.452387295 podStartE2EDuration="2.452387295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:40:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:40:06.451687057 +0000 UTC m=+3076.195821093" watchObservedRunningTime="2026-10-30 14:40:06.452387295 +0000 UTC m=+3076.196521342" Oct 30 14:40:07.843658 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:07.839036 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:40:07.993878 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:07.993820 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:40:08.146187 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:08.146079 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:40:09.762946 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:09.762912 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:40:09.855956 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:09.855919 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6176c3-749f86c99b-wfgdk"] Oct 30 14:40:09.860609 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:09.860589 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6176c3-749f86c99b-wfgdk" Oct 30 14:40:09.891408 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:09.889910 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-6176c3\"" Oct 30 14:40:09.912663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:09.912217 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6176c3-749f86c99b-wfgdk"] Oct 30 14:40:09.951584 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:09.951542 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:40:10.029721 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:10.019299 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad63625b-56af-46f1-be5c-f65fb08d97af-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6176c3-749f86c99b-wfgdk\" (UID: \"ad63625b-56af-46f1-be5c-f65fb08d97af\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6176c3-749f86c99b-wfgdk" Oct 30 14:40:10.030238 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:10.030202 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp45d\" (UniqueName: \"kubernetes.io/projected/ad63625b-56af-46f1-be5c-f65fb08d97af-kube-api-access-gp45d\") pod \"payload-processing-e2e-ipp-a-6176c3-749f86c99b-wfgdk\" (UID: \"ad63625b-56af-46f1-be5c-f65fb08d97af\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6176c3-749f86c99b-wfgdk" Oct 30 14:40:10.134664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:10.134559 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp45d\" (UniqueName: \"kubernetes.io/projected/ad63625b-56af-46f1-be5c-f65fb08d97af-kube-api-access-gp45d\") pod \"payload-processing-e2e-ipp-a-6176c3-749f86c99b-wfgdk\" (UID: \"ad63625b-56af-46f1-be5c-f65fb08d97af\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6176c3-749f86c99b-wfgdk" Oct 30 14:40:10.135112 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:10.135082 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad63625b-56af-46f1-be5c-f65fb08d97af-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6176c3-749f86c99b-wfgdk\" (UID: \"ad63625b-56af-46f1-be5c-f65fb08d97af\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6176c3-749f86c99b-wfgdk" Oct 30 14:40:10.136291 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:10.136267 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad63625b-56af-46f1-be5c-f65fb08d97af-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6176c3-749f86c99b-wfgdk\" (UID: \"ad63625b-56af-46f1-be5c-f65fb08d97af\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6176c3-749f86c99b-wfgdk" Oct 30 14:40:10.192663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:10.187759 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp45d\" (UniqueName: \"kubernetes.io/projected/ad63625b-56af-46f1-be5c-f65fb08d97af-kube-api-access-gp45d\") pod \"payload-processing-e2e-ipp-a-6176c3-749f86c99b-wfgdk\" (UID: \"ad63625b-56af-46f1-be5c-f65fb08d97af\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6176c3-749f86c99b-wfgdk" Oct 30 14:40:10.256374 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:10.256312 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:40:10.475822 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:10.475785 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6176c3-749f86c99b-wfgdk" Oct 30 14:40:10.831325 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:10.831131 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6176c3-749f86c99b-wfgdk"] Oct 30 14:40:10.833192 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:40:10.833157 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad63625b_56af_46f1_be5c_f65fb08d97af.slice/crio-cef274e931ee9a50160048c631610837ee3ebf324005c20a47545d0f9dd2ae52 WatchSource:0}: Error finding container cef274e931ee9a50160048c631610837ee3ebf324005c20a47545d0f9dd2ae52: Status 404 returned error can't find the container with id cef274e931ee9a50160048c631610837ee3ebf324005c20a47545d0f9dd2ae52 Oct 30 14:40:11.447779 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:11.447741 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6176c3-749f86c99b-wfgdk" event={"ID":"ad63625b-56af-46f1-be5c-f65fb08d97af","Type":"ContainerStarted","Data":"bc0cc2e0d9f6062385c53865e955fb7f3212f8c84280b80846b520b3e5f2d742"} Oct 30 14:40:11.447967 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:11.447794 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6176c3-749f86c99b-wfgdk" event={"ID":"ad63625b-56af-46f1-be5c-f65fb08d97af","Type":"ContainerStarted","Data":"cef274e931ee9a50160048c631610837ee3ebf324005c20a47545d0f9dd2ae52"} Oct 30 14:40:11.447967 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:11.447825 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-6176c3-749f86c99b-wfgdk" Oct 30 14:40:12.324591 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:12.322857 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:40:14.448368 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:14.448323 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:40:16.288484 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:16.288445 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:40:17.416699 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:17.416632 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a1854b-6bd77d5846-984lk" Oct 30 14:40:17.464972 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:17.464920 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-6176c3-749f86c99b-wfgdk" podStartSLOduration=8.464905636 podStartE2EDuration="8.464905636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:40:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:40:11.510937235 +0000 UTC m=+3081.255071275" watchObservedRunningTime="2026-10-30 14:40:17.464905636 +0000 UTC m=+3087.209039695" Oct 30 14:40:18.029369 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:18.029333 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:40:18.308910 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:18.308876 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:40:18.524365 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:18.524316 4982 generic.go:356] "Generic (PLEG): container finished" podID="463ccf6c-67b5-45b6-970c-ee89fc36a12b" containerID="a99ebb3920b7a686f4de743c079efb06dc7326eb8abf9ca22b6c033fae601b79" exitCode=0 Oct 30 14:40:18.524910 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:18.524400 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js" event={"ID":"463ccf6c-67b5-45b6-970c-ee89fc36a12b","Type":"ContainerDied","Data":"a99ebb3920b7a686f4de743c079efb06dc7326eb8abf9ca22b6c033fae601b79"} Oct 30 14:40:18.524910 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:18.524900 4982 scope.go:122] "RemoveContainer" containerID="a99ebb3920b7a686f4de743c079efb06dc7326eb8abf9ca22b6c033fae601b79" Oct 30 14:40:18.526767 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:18.526735 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:40:18.526909 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:18.526820 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:40:18.526909 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:18.526871 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:40:19.534269 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:19.534236 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js" event={"ID":"463ccf6c-67b5-45b6-970c-ee89fc36a12b","Type":"ContainerStarted","Data":"a86dba336c731ca5f3563df4f5bfa9420a6257540891a3daff4c54c4d1553a4e"} Oct 30 14:40:20.017878 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:20.017842 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js" Oct 30 14:40:20.019046 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:20.019016 4982 patch_prober.go:29] interesting pod/e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 30 14:40:20.019202 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:20.019088 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js" podUID="463ccf6c-67b5-45b6-970c-ee89fc36a12b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 30 14:40:20.215432 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:20.215065 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:40:21.017846 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:21.017802 4982 patch_prober.go:29] interesting pod/e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 30 14:40:21.018358 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:21.017870 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js" podUID="463ccf6c-67b5-45b6-970c-ee89fc36a12b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 30 14:40:21.301713 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:21.301463 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz"] Oct 30 14:40:21.321313 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:21.320589 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz" Oct 30 14:40:21.321313 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:21.321235 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz"] Oct 30 14:40:21.340226 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:21.340206 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-75443e-openshift-default-dockercfg-r7cs9\"" Oct 30 14:40:21.443743 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:21.443703 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b30286e-1e19-4330-94e0-15decca18f76-istio-token\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz\" (UID: \"1b30286e-1e19-4330-94e0-15decca18f76\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz" Oct 30 14:40:21.443909 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:21.443778 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b30286e-1e19-4330-94e0-15decca18f76-istiod-ca-cert\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz\" (UID: \"1b30286e-1e19-4330-94e0-15decca18f76\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz" Oct 30 14:40:21.443909 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:21.443883 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b30286e-1e19-4330-94e0-15decca18f76-credential-socket\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz\" (UID: \"1b30286e-1e19-4330-94e0-15decca18f76\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz" Oct 30 14:40:21.444010 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:21.443992 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b30286e-1e19-4330-94e0-15decca18f76-istio-podinfo\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz\" (UID: \"1b30286e-1e19-4330-94e0-15decca18f76\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz" Oct 30 14:40:21.444126 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:21.444101 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b30286e-1e19-4330-94e0-15decca18f76-istio-data\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz\" (UID: \"1b30286e-1e19-4330-94e0-15decca18f76\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz" Oct 30 14:40:21.444215 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:21.444200 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b30286e-1e19-4330-94e0-15decca18f76-workload-certs\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz\" (UID: \"1b30286e-1e19-4330-94e0-15decca18f76\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz" Oct 30 14:40:21.444272 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:21.444231 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b30286e-1e19-4330-94e0-15decca18f76-istio-envoy\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz\" (UID: \"1b30286e-1e19-4330-94e0-15decca18f76\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz" Oct 30 14:40:21.444376 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:21.444354 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zkhj\" (UniqueName: \"kubernetes.io/projected/1b30286e-1e19-4330-94e0-15decca18f76-kube-api-access-8zkhj\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz\" (UID: \"1b30286e-1e19-4330-94e0-15decca18f76\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz" Oct 30 14:40:21.444446 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:21.444396 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b30286e-1e19-4330-94e0-15decca18f76-workload-socket\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz\" (UID: \"1b30286e-1e19-4330-94e0-15decca18f76\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz" Oct 30 14:40:21.530474 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:21.530445 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js" Oct 30 14:40:21.545297 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:21.545267 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b30286e-1e19-4330-94e0-15decca18f76-istio-data\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz\" (UID: \"1b30286e-1e19-4330-94e0-15decca18f76\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz" Oct 30 14:40:21.545465 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:21.545358 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b30286e-1e19-4330-94e0-15decca18f76-workload-certs\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz\" (UID: \"1b30286e-1e19-4330-94e0-15decca18f76\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz" Oct 30 14:40:21.545465 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:21.545383 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b30286e-1e19-4330-94e0-15decca18f76-istio-envoy\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz\" (UID: \"1b30286e-1e19-4330-94e0-15decca18f76\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz" Oct 30 14:40:21.545465 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:21.545455 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zkhj\" (UniqueName: \"kubernetes.io/projected/1b30286e-1e19-4330-94e0-15decca18f76-kube-api-access-8zkhj\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz\" (UID: \"1b30286e-1e19-4330-94e0-15decca18f76\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz" Oct 30 14:40:21.545679 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:21.545487 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b30286e-1e19-4330-94e0-15decca18f76-workload-socket\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz\" (UID: \"1b30286e-1e19-4330-94e0-15decca18f76\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz" Oct 30 14:40:21.545679 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:21.545573 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b30286e-1e19-4330-94e0-15decca18f76-istio-token\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz\" (UID: \"1b30286e-1e19-4330-94e0-15decca18f76\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz" Oct 30 14:40:21.545679 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:21.545663 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b30286e-1e19-4330-94e0-15decca18f76-istiod-ca-cert\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz\" (UID: \"1b30286e-1e19-4330-94e0-15decca18f76\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz" Oct 30 14:40:21.545845 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:21.545706 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b30286e-1e19-4330-94e0-15decca18f76-credential-socket\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz\" (UID: \"1b30286e-1e19-4330-94e0-15decca18f76\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz" Oct 30 14:40:21.545845 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:21.545758 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b30286e-1e19-4330-94e0-15decca18f76-istio-data\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz\" (UID: \"1b30286e-1e19-4330-94e0-15decca18f76\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz" Oct 30 14:40:21.545845 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:21.545779 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b30286e-1e19-4330-94e0-15decca18f76-istio-podinfo\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz\" (UID: \"1b30286e-1e19-4330-94e0-15decca18f76\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz" Oct 30 14:40:21.545845 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:21.545792 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b30286e-1e19-4330-94e0-15decca18f76-workload-certs\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz\" (UID: \"1b30286e-1e19-4330-94e0-15decca18f76\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz" Oct 30 14:40:21.546098 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:21.546079 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b30286e-1e19-4330-94e0-15decca18f76-workload-socket\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz\" (UID: \"1b30286e-1e19-4330-94e0-15decca18f76\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz" Oct 30 14:40:21.546162 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:21.546127 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b30286e-1e19-4330-94e0-15decca18f76-credential-socket\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz\" (UID: \"1b30286e-1e19-4330-94e0-15decca18f76\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz" Oct 30 14:40:21.546559 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:21.546530 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b30286e-1e19-4330-94e0-15decca18f76-istiod-ca-cert\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz\" (UID: \"1b30286e-1e19-4330-94e0-15decca18f76\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz" Oct 30 14:40:21.549084 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:21.549060 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b30286e-1e19-4330-94e0-15decca18f76-istio-envoy\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz\" (UID: \"1b30286e-1e19-4330-94e0-15decca18f76\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz" Oct 30 14:40:21.549356 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:21.549336 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b30286e-1e19-4330-94e0-15decca18f76-istio-podinfo\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz\" (UID: \"1b30286e-1e19-4330-94e0-15decca18f76\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz" Oct 30 14:40:21.574220 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:21.574189 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b30286e-1e19-4330-94e0-15decca18f76-istio-token\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz\" (UID: \"1b30286e-1e19-4330-94e0-15decca18f76\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz" Oct 30 14:40:21.574317 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:21.574273 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zkhj\" (UniqueName: \"kubernetes.io/projected/1b30286e-1e19-4330-94e0-15decca18f76-kube-api-access-8zkhj\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz\" (UID: \"1b30286e-1e19-4330-94e0-15decca18f76\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz" Oct 30 14:40:21.638398 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:21.638368 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz" Oct 30 14:40:22.020669 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:22.018114 4982 patch_prober.go:29] interesting pod/e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 30 14:40:22.020669 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:22.018181 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js" podUID="463ccf6c-67b5-45b6-970c-ee89fc36a12b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 30 14:40:22.020669 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:22.018715 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz"] Oct 30 14:40:22.021984 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:22.021952 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:40:22.022103 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:22.022024 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:40:22.022103 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:22.022057 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:40:22.117007 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:22.113509 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:40:22.457767 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:22.457740 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-6176c3-749f86c99b-wfgdk" Oct 30 14:40:22.563681 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:22.561762 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz" event={"ID":"1b30286e-1e19-4330-94e0-15decca18f76","Type":"ContainerStarted","Data":"b5838525fa8be1cbed04620f11312410d6454138abb1249cdb958417ecb25d78"} Oct 30 14:40:22.563681 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:22.561829 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz" event={"ID":"1b30286e-1e19-4330-94e0-15decca18f76","Type":"ContainerStarted","Data":"b0a994dcb95c6e597300597adce9d37864dec4a52ff481c9985107bbaa08589c"} Oct 30 14:40:22.639728 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:22.638980 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz" Oct 30 14:40:23.018659 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:23.017677 4982 patch_prober.go:29] interesting pod/e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 30 14:40:23.018659 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:23.017766 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js" podUID="463ccf6c-67b5-45b6-970c-ee89fc36a12b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 30 14:40:23.647664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:23.640513 4982 patch_prober.go:29] interesting pod/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Oct 30 14:40:23.647664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:23.640582 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz" podUID="1b30286e-1e19-4330-94e0-15decca18f76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Oct 30 14:40:24.019716 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:24.018020 4982 patch_prober.go:29] interesting pod/e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 30 14:40:24.019716 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:24.018084 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js" podUID="463ccf6c-67b5-45b6-970c-ee89fc36a12b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 30 14:40:24.093665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:24.089006 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:40:24.639594 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:24.639552 4982 patch_prober.go:29] interesting pod/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Oct 30 14:40:24.639780 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:24.639630 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz" podUID="1b30286e-1e19-4330-94e0-15decca18f76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Oct 30 14:40:25.209369 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:25.206548 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js" Oct 30 14:40:25.211074 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:25.211049 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js" Oct 30 14:40:25.275523 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:25.275461 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz" podStartSLOduration=4.275442809 podStartE2EDuration="4.275442809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:40:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:40:22.63431056 +0000 UTC m=+3092.378444601" watchObservedRunningTime="2026-10-30 14:40:25.275442809 +0000 UTC m=+3095.019576856" Oct 30 14:40:25.639114 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:25.639072 4982 patch_prober.go:29] interesting pod/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Oct 30 14:40:25.639284 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:25.639131 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz" podUID="1b30286e-1e19-4330-94e0-15decca18f76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Oct 30 14:40:25.968743 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:25.968655 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:40:26.120452 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:26.120417 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:40:26.267203 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:26.267097 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:40:26.589063 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:26.589027 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:40:26.642062 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:26.642016 4982 patch_prober.go:29] interesting pod/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Oct 30 14:40:26.642248 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:26.642098 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz" podUID="1b30286e-1e19-4330-94e0-15decca18f76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Oct 30 14:40:26.847436 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:26.847343 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:40:27.638785 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:27.638754 4982 patch_prober.go:29] interesting pod/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Oct 30 14:40:27.639197 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:27.638814 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz" podUID="1b30286e-1e19-4330-94e0-15decca18f76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Oct 30 14:40:28.032874 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:40:28.032792 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:40:28.638747 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:28.638712 4982 patch_prober.go:29] interesting pod/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Oct 30 14:40:28.638931 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:28.638774 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz" podUID="1b30286e-1e19-4330-94e0-15decca18f76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Oct 30 14:40:28.988073 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:28.987989 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:40:29.644912 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:29.644879 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz" Oct 30 14:40:29.645398 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:29.645225 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz" Oct 30 14:40:29.645947 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:29.645924 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz" Oct 30 14:40:31.146033 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:31.145999 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:40:33.161463 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:33.161387 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:40:35.314372 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:35.314323 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:40:35.329421 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:35.329389 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-75443e/rate-test-model-75443e-kserve-6b64d597fc-74z2f"] Oct 30 14:40:35.562228 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:35.562176 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:40:37.487186 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:37.487149 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:40:39.922198 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:39.922159 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:40:41.864618 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:41.864585 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:40:43.820258 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:43.820221 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:40:45.745341 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:45.745293 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:40:46.375677 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:46.375535 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-0efe43/rate-test-model-0efe43-kserve-6f8954844f-d5kvb"] Oct 30 14:40:46.462283 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:46.462250 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:40:46.694593 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:46.694504 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:40:47.089246 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:47.089215 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:40:47.606258 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:47.606222 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:40:47.863965 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:47.863881 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:40:49.657434 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:49.657397 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2"] Oct 30 14:40:49.663200 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:49.663182 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2" Oct 30 14:40:49.684288 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:49.684266 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-c7c37b-openshift-default-dockercfg-xbblf\"" Oct 30 14:40:49.701414 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:49.701386 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2"] Oct 30 14:40:49.832429 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:49.832392 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c75528-f183-4379-841e-e413ac8d9c35-workload-socket\") pod \"e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2\" (UID: \"f8c75528-f183-4379-841e-e413ac8d9c35\") " pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2" Oct 30 14:40:49.832604 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:49.832440 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8c75528-f183-4379-841e-e413ac8d9c35-istio-podinfo\") pod \"e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2\" (UID: \"f8c75528-f183-4379-841e-e413ac8d9c35\") " pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2" Oct 30 14:40:49.832665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:49.832594 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8c75528-f183-4379-841e-e413ac8d9c35-istio-data\") pod \"e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2\" (UID: \"f8c75528-f183-4379-841e-e413ac8d9c35\") " pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2" Oct 30 14:40:49.832701 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:49.832671 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8c75528-f183-4379-841e-e413ac8d9c35-istio-envoy\") pod \"e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2\" (UID: \"f8c75528-f183-4379-841e-e413ac8d9c35\") " pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2" Oct 30 14:40:49.832797 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:49.832775 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8c75528-f183-4379-841e-e413ac8d9c35-workload-certs\") pod \"e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2\" (UID: \"f8c75528-f183-4379-841e-e413ac8d9c35\") " pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2" Oct 30 14:40:49.832935 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:49.832921 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8c75528-f183-4379-841e-e413ac8d9c35-istiod-ca-cert\") pod \"e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2\" (UID: \"f8c75528-f183-4379-841e-e413ac8d9c35\") " pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2" Oct 30 14:40:49.832988 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:49.832972 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg7px\" (UniqueName: \"kubernetes.io/projected/f8c75528-f183-4379-841e-e413ac8d9c35-kube-api-access-jg7px\") pod \"e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2\" (UID: \"f8c75528-f183-4379-841e-e413ac8d9c35\") " pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2" Oct 30 14:40:49.833024 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:49.833017 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c75528-f183-4379-841e-e413ac8d9c35-credential-socket\") pod \"e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2\" (UID: \"f8c75528-f183-4379-841e-e413ac8d9c35\") " pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2" Oct 30 14:40:49.833056 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:49.833041 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8c75528-f183-4379-841e-e413ac8d9c35-istio-token\") pod \"e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2\" (UID: \"f8c75528-f183-4379-841e-e413ac8d9c35\") " pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2" Oct 30 14:40:49.934909 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:49.934835 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8c75528-f183-4379-841e-e413ac8d9c35-workload-certs\") pod \"e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2\" (UID: \"f8c75528-f183-4379-841e-e413ac8d9c35\") " pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2" Oct 30 14:40:49.935062 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:49.934926 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8c75528-f183-4379-841e-e413ac8d9c35-istiod-ca-cert\") pod \"e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2\" (UID: \"f8c75528-f183-4379-841e-e413ac8d9c35\") " pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2" Oct 30 14:40:49.935062 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:49.934970 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg7px\" (UniqueName: \"kubernetes.io/projected/f8c75528-f183-4379-841e-e413ac8d9c35-kube-api-access-jg7px\") pod \"e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2\" (UID: \"f8c75528-f183-4379-841e-e413ac8d9c35\") " pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2" Oct 30 14:40:49.935062 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:49.934997 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c75528-f183-4379-841e-e413ac8d9c35-credential-socket\") pod \"e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2\" (UID: \"f8c75528-f183-4379-841e-e413ac8d9c35\") " pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2" Oct 30 14:40:49.935062 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:49.935029 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8c75528-f183-4379-841e-e413ac8d9c35-istio-token\") pod \"e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2\" (UID: \"f8c75528-f183-4379-841e-e413ac8d9c35\") " pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2" Oct 30 14:40:49.935274 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:49.935108 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c75528-f183-4379-841e-e413ac8d9c35-workload-socket\") pod \"e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2\" (UID: \"f8c75528-f183-4379-841e-e413ac8d9c35\") " pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2" Oct 30 14:40:49.935274 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:49.935155 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8c75528-f183-4379-841e-e413ac8d9c35-istio-podinfo\") pod \"e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2\" (UID: \"f8c75528-f183-4379-841e-e413ac8d9c35\") " pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2" Oct 30 14:40:49.935274 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:49.935227 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8c75528-f183-4379-841e-e413ac8d9c35-istio-data\") pod \"e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2\" (UID: \"f8c75528-f183-4379-841e-e413ac8d9c35\") " pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2" Oct 30 14:40:49.935274 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:49.935252 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8c75528-f183-4379-841e-e413ac8d9c35-istio-envoy\") pod \"e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2\" (UID: \"f8c75528-f183-4379-841e-e413ac8d9c35\") " pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2" Oct 30 14:40:49.935542 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:49.935312 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c75528-f183-4379-841e-e413ac8d9c35-credential-socket\") pod \"e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2\" (UID: \"f8c75528-f183-4379-841e-e413ac8d9c35\") " pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2" Oct 30 14:40:49.935542 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:49.935252 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8c75528-f183-4379-841e-e413ac8d9c35-workload-certs\") pod \"e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2\" (UID: \"f8c75528-f183-4379-841e-e413ac8d9c35\") " pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2" Oct 30 14:40:49.935542 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:49.935484 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c75528-f183-4379-841e-e413ac8d9c35-workload-socket\") pod \"e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2\" (UID: \"f8c75528-f183-4379-841e-e413ac8d9c35\") " pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2" Oct 30 14:40:49.935542 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:49.935521 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8c75528-f183-4379-841e-e413ac8d9c35-istio-data\") pod \"e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2\" (UID: \"f8c75528-f183-4379-841e-e413ac8d9c35\") " pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2" Oct 30 14:40:49.935784 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:49.935719 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8c75528-f183-4379-841e-e413ac8d9c35-istiod-ca-cert\") pod \"e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2\" (UID: \"f8c75528-f183-4379-841e-e413ac8d9c35\") " pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2" Oct 30 14:40:49.937478 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:49.937456 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8c75528-f183-4379-841e-e413ac8d9c35-istio-envoy\") pod \"e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2\" (UID: \"f8c75528-f183-4379-841e-e413ac8d9c35\") " pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2" Oct 30 14:40:49.937776 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:49.937755 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8c75528-f183-4379-841e-e413ac8d9c35-istio-podinfo\") pod \"e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2\" (UID: \"f8c75528-f183-4379-841e-e413ac8d9c35\") " pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2" Oct 30 14:40:49.961091 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:49.961063 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg7px\" (UniqueName: \"kubernetes.io/projected/f8c75528-f183-4379-841e-e413ac8d9c35-kube-api-access-jg7px\") pod \"e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2\" (UID: \"f8c75528-f183-4379-841e-e413ac8d9c35\") " pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2" Oct 30 14:40:49.961163 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:49.961085 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8c75528-f183-4379-841e-e413ac8d9c35-istio-token\") pod \"e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2\" (UID: \"f8c75528-f183-4379-841e-e413ac8d9c35\") " pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2" Oct 30 14:40:49.978048 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:49.978023 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2" Oct 30 14:40:50.321206 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:40:50.321165 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8c75528_f183_4379_841e_e413ac8d9c35.slice/crio-1184e2c852511f0baa2dc2ae1095a5e9da783022012781039ecddbaa99d7b9a9 WatchSource:0}: Error finding container 1184e2c852511f0baa2dc2ae1095a5e9da783022012781039ecddbaa99d7b9a9: Status 404 returned error can't find the container with id 1184e2c852511f0baa2dc2ae1095a5e9da783022012781039ecddbaa99d7b9a9 Oct 30 14:40:50.322857 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:50.322827 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2"] Oct 30 14:40:50.324997 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:50.324961 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:40:50.325084 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:50.325044 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:40:50.325125 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:50.325087 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:40:50.358603 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:50.358575 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w6-75443e/rate-test-model-75443e-kserve-6b64d597fc-74z2f" podUID="48dd08f0-9733-48b1-bb45-cd93e4bb0564" containerName="main" containerID="cri-o://b431916a3c55afd0dbe6bb6e169a8f98091f38581a2b1918e8c3c97d6c3b7a56" gracePeriod=45 Oct 30 14:40:50.372561 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:50.371494 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:40:50.645698 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:50.645670 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-75443e/rate-test-model-75443e-kserve-6b64d597fc-74z2f" Oct 30 14:40:50.746145 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:50.746113 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48dd08f0-9733-48b1-bb45-cd93e4bb0564-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48dd08f0-9733-48b1-bb45-cd93e4bb0564-model-cache\") pod \"48dd08f0-9733-48b1-bb45-cd93e4bb0564\" (UID: \"48dd08f0-9733-48b1-bb45-cd93e4bb0564\") " Oct 30 14:40:50.746544 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:50.746210 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48dd08f0-9733-48b1-bb45-cd93e4bb0564-home\" (UniqueName: \"kubernetes.io/empty-dir/48dd08f0-9733-48b1-bb45-cd93e4bb0564-home\") pod \"48dd08f0-9733-48b1-bb45-cd93e4bb0564\" (UID: \"48dd08f0-9733-48b1-bb45-cd93e4bb0564\") " Oct 30 14:40:50.746544 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:50.746260 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48dd08f0-9733-48b1-bb45-cd93e4bb0564-tls-certs\" (UniqueName: \"kubernetes.io/secret/48dd08f0-9733-48b1-bb45-cd93e4bb0564-tls-certs\") pod \"48dd08f0-9733-48b1-bb45-cd93e4bb0564\" (UID: \"48dd08f0-9733-48b1-bb45-cd93e4bb0564\") " Oct 30 14:40:50.746544 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:50.746325 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48dd08f0-9733-48b1-bb45-cd93e4bb0564-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48dd08f0-9733-48b1-bb45-cd93e4bb0564-tmp-dir\") pod \"48dd08f0-9733-48b1-bb45-cd93e4bb0564\" (UID: \"48dd08f0-9733-48b1-bb45-cd93e4bb0564\") " Oct 30 14:40:50.746544 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:50.746359 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48dd08f0-9733-48b1-bb45-cd93e4bb0564-dshm\" (UniqueName: \"kubernetes.io/empty-dir/48dd08f0-9733-48b1-bb45-cd93e4bb0564-dshm\") pod \"48dd08f0-9733-48b1-bb45-cd93e4bb0564\" (UID: \"48dd08f0-9733-48b1-bb45-cd93e4bb0564\") " Oct 30 14:40:50.746544 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:50.746379 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48dd08f0-9733-48b1-bb45-cd93e4bb0564-model-cache" pod "48dd08f0-9733-48b1-bb45-cd93e4bb0564" (UID: "48dd08f0-9733-48b1-bb45-cd93e4bb0564"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:40:50.746544 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:50.746493 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48dd08f0-9733-48b1-bb45-cd93e4bb0564-kube-api-access-rb8jb\" (UniqueName: \"kubernetes.io/projected/48dd08f0-9733-48b1-bb45-cd93e4bb0564-kube-api-access-rb8jb\") pod \"48dd08f0-9733-48b1-bb45-cd93e4bb0564\" (UID: \"48dd08f0-9733-48b1-bb45-cd93e4bb0564\") " Oct 30 14:40:50.746544 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:50.746495 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48dd08f0-9733-48b1-bb45-cd93e4bb0564-home" pod "48dd08f0-9733-48b1-bb45-cd93e4bb0564" (UID: "48dd08f0-9733-48b1-bb45-cd93e4bb0564"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:40:50.746895 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:50.746546 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48dd08f0-9733-48b1-bb45-cd93e4bb0564-tmp-dir" pod "48dd08f0-9733-48b1-bb45-cd93e4bb0564" (UID: "48dd08f0-9733-48b1-bb45-cd93e4bb0564"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:40:50.747131 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:50.747107 4982 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48dd08f0-9733-48b1-bb45-cd93e4bb0564-model-cache\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:40:50.747223 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:50.747135 4982 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48dd08f0-9733-48b1-bb45-cd93e4bb0564-home\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:40:50.747223 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:50.747146 4982 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48dd08f0-9733-48b1-bb45-cd93e4bb0564-tmp-dir\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:40:50.748882 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:50.748855 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48dd08f0-9733-48b1-bb45-cd93e4bb0564-dshm" pod "48dd08f0-9733-48b1-bb45-cd93e4bb0564" (UID: "48dd08f0-9733-48b1-bb45-cd93e4bb0564"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:40:50.749002 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:50.748979 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48dd08f0-9733-48b1-bb45-cd93e4bb0564-tls-certs" pod "48dd08f0-9733-48b1-bb45-cd93e4bb0564" (UID: "48dd08f0-9733-48b1-bb45-cd93e4bb0564"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:40:50.749054 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:50.749016 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48dd08f0-9733-48b1-bb45-cd93e4bb0564-kube-api-access-rb8jb" pod "48dd08f0-9733-48b1-bb45-cd93e4bb0564" (UID: "48dd08f0-9733-48b1-bb45-cd93e4bb0564"). InnerVolumeSpecName "kube-api-access-rb8jb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:40:50.847279 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:50.847246 4982 generic.go:356] "Generic (PLEG): container finished" podID="48dd08f0-9733-48b1-bb45-cd93e4bb0564" containerID="b431916a3c55afd0dbe6bb6e169a8f98091f38581a2b1918e8c3c97d6c3b7a56" exitCode=0 Oct 30 14:40:50.847472 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:50.847283 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-75443e/rate-test-model-75443e-kserve-6b64d597fc-74z2f" event={"ID":"48dd08f0-9733-48b1-bb45-cd93e4bb0564","Type":"ContainerDied","Data":"b431916a3c55afd0dbe6bb6e169a8f98091f38581a2b1918e8c3c97d6c3b7a56"} Oct 30 14:40:50.847472 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:50.847317 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-75443e/rate-test-model-75443e-kserve-6b64d597fc-74z2f" Oct 30 14:40:50.847472 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:50.847332 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-75443e/rate-test-model-75443e-kserve-6b64d597fc-74z2f" event={"ID":"48dd08f0-9733-48b1-bb45-cd93e4bb0564","Type":"ContainerDied","Data":"9f489910d087911c7cdc09b25f5d991058af6764be72892ab8f407ea1f2b38b6"} Oct 30 14:40:50.847472 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:50.847351 4982 scope.go:122] "RemoveContainer" containerID="b431916a3c55afd0dbe6bb6e169a8f98091f38581a2b1918e8c3c97d6c3b7a56" Oct 30 14:40:50.848363 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:50.848328 4982 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48dd08f0-9733-48b1-bb45-cd93e4bb0564-tls-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:40:50.848363 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:50.848365 4982 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48dd08f0-9733-48b1-bb45-cd93e4bb0564-dshm\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:40:50.848558 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:50.848379 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb8jb\" (UniqueName: \"kubernetes.io/projected/48dd08f0-9733-48b1-bb45-cd93e4bb0564-kube-api-access-rb8jb\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:40:50.849530 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:50.849501 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2" event={"ID":"f8c75528-f183-4379-841e-e413ac8d9c35","Type":"ContainerStarted","Data":"c66d067be4b524c1eceba229db9944d94415e0dfeda879690abcb03637b658bb"} Oct 30 14:40:50.849695 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:50.849551 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2" event={"ID":"f8c75528-f183-4379-841e-e413ac8d9c35","Type":"ContainerStarted","Data":"1184e2c852511f0baa2dc2ae1095a5e9da783022012781039ecddbaa99d7b9a9"} Oct 30 14:40:50.870840 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:50.870818 4982 scope.go:122] "RemoveContainer" containerID="b431916a3c55afd0dbe6bb6e169a8f98091f38581a2b1918e8c3c97d6c3b7a56" Oct 30 14:40:50.871100 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:40:50.871079 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b431916a3c55afd0dbe6bb6e169a8f98091f38581a2b1918e8c3c97d6c3b7a56\": container with ID starting with b431916a3c55afd0dbe6bb6e169a8f98091f38581a2b1918e8c3c97d6c3b7a56 not found: ID does not exist" containerID="b431916a3c55afd0dbe6bb6e169a8f98091f38581a2b1918e8c3c97d6c3b7a56" Oct 30 14:40:50.871172 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:50.871107 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b431916a3c55afd0dbe6bb6e169a8f98091f38581a2b1918e8c3c97d6c3b7a56"} err="failed to get container status \"b431916a3c55afd0dbe6bb6e169a8f98091f38581a2b1918e8c3c97d6c3b7a56\": rpc error: code = NotFound desc = could not find container \"b431916a3c55afd0dbe6bb6e169a8f98091f38581a2b1918e8c3c97d6c3b7a56\": container with ID starting with b431916a3c55afd0dbe6bb6e169a8f98091f38581a2b1918e8c3c97d6c3b7a56 not found: ID does not exist" Oct 30 14:40:50.907650 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:50.907557 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-75443e/rate-test-model-75443e-kserve-6b64d597fc-74z2f"] Oct 30 14:40:50.915356 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:50.915328 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w6-75443e/rate-test-model-75443e-kserve-6b64d597fc-74z2f"] Oct 30 14:40:50.960923 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:50.960874 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2" podStartSLOduration=1.960861855 podStartE2EDuration="1.960861855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:40:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:40:50.956761864 +0000 UTC m=+3120.700895904" watchObservedRunningTime="2026-10-30 14:40:50.960861855 +0000 UTC m=+3120.704995895" Oct 30 14:40:50.978458 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:50.978435 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2" Oct 30 14:40:51.472563 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:51.472531 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:40:51.653229 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:51.653198 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:40:51.992665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:51.990224 4982 patch_prober.go:29] interesting pod/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 30 14:40:51.992665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:51.990296 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2" podUID="f8c75528-f183-4379-841e-e413ac8d9c35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 30 14:40:52.112919 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:52.112879 4982 scope.go:122] "RemoveContainer" containerID="62d6b2d46350e915035a9b986c1d050601ca058ad60be62bdfe6a5e76d1fbfc5" Oct 30 14:40:52.125223 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:52.125201 4982 scope.go:122] "RemoveContainer" containerID="5c8230a326978d3ba66850dd85582972fef91e3d19c02f33fbc67b0a248d4752" Oct 30 14:40:52.134545 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:52.134522 4982 scope.go:122] "RemoveContainer" containerID="6a4279d54171dd88b81fe4be414dd635f445d2d497e6e33cb423e0e2e1c1dbdd" Oct 30 14:40:52.416704 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:52.415735 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48dd08f0-9733-48b1-bb45-cd93e4bb0564" path="/var/lib/kubelet/pods/48dd08f0-9733-48b1-bb45-cd93e4bb0564/volumes" Oct 30 14:40:52.673073 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:52.672997 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:40:52.978698 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:52.978615 4982 patch_prober.go:29] interesting pod/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 30 14:40:52.978857 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:52.978707 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2" podUID="f8c75528-f183-4379-841e-e413ac8d9c35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 30 14:40:53.268114 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:53.268040 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:40:53.422140 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:53.422106 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2927fd-55db459cf-lqzdk"] Oct 30 14:40:53.422781 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:53.422755 4982 cpu_manager.go:414] "containerMap: removing container" podUID="48dd08f0-9733-48b1-bb45-cd93e4bb0564" containerName="main" Oct 30 14:40:53.422781 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:53.422773 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48dd08f0-9733-48b1-bb45-cd93e4bb0564" containerName="main" Oct 30 14:40:53.422981 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:53.422874 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="48dd08f0-9733-48b1-bb45-cd93e4bb0564" containerName="main" Oct 30 14:40:53.426484 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:53.426461 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2927fd-55db459cf-lqzdk" Oct 30 14:40:53.427035 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:53.427005 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:40:53.449438 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:53.449409 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2927fd-55db459cf-lqzdk"] Oct 30 14:40:53.457319 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:53.457297 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-2927fd\"" Oct 30 14:40:53.457446 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:53.457386 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-2927fd-dockercfg-w45cv\"" Oct 30 14:40:53.539450 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:53.513782 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/920bf174-09b0-4966-b029-4e200373359b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2927fd-55db459cf-lqzdk\" (UID: \"920bf174-09b0-4966-b029-4e200373359b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2927fd-55db459cf-lqzdk" Oct 30 14:40:53.539450 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:53.514077 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swkp5\" (UniqueName: \"kubernetes.io/projected/920bf174-09b0-4966-b029-4e200373359b-kube-api-access-swkp5\") pod \"payload-pre-processing-e2e-ipp-a-2927fd-55db459cf-lqzdk\" (UID: \"920bf174-09b0-4966-b029-4e200373359b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2927fd-55db459cf-lqzdk" Oct 30 14:40:53.628664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:53.586118 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2927fd-844ddcc4bd-wbrs7"] Oct 30 14:40:53.648428 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:53.617554 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/920bf174-09b0-4966-b029-4e200373359b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2927fd-55db459cf-lqzdk\" (UID: \"920bf174-09b0-4966-b029-4e200373359b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2927fd-55db459cf-lqzdk" Oct 30 14:40:53.648428 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:53.638989 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swkp5\" (UniqueName: \"kubernetes.io/projected/920bf174-09b0-4966-b029-4e200373359b-kube-api-access-swkp5\") pod \"payload-pre-processing-e2e-ipp-a-2927fd-55db459cf-lqzdk\" (UID: \"920bf174-09b0-4966-b029-4e200373359b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2927fd-55db459cf-lqzdk" Oct 30 14:40:53.648428 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:53.646442 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/920bf174-09b0-4966-b029-4e200373359b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2927fd-55db459cf-lqzdk\" (UID: \"920bf174-09b0-4966-b029-4e200373359b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2927fd-55db459cf-lqzdk" Oct 30 14:40:53.695920 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:53.695875 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2927fd-844ddcc4bd-wbrs7"] Oct 30 14:40:53.696236 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:53.696218 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2927fd-844ddcc4bd-wbrs7" Oct 30 14:40:53.712881 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:53.707731 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swkp5\" (UniqueName: \"kubernetes.io/projected/920bf174-09b0-4966-b029-4e200373359b-kube-api-access-swkp5\") pod \"payload-pre-processing-e2e-ipp-a-2927fd-55db459cf-lqzdk\" (UID: \"920bf174-09b0-4966-b029-4e200373359b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2927fd-55db459cf-lqzdk" Oct 30 14:40:53.819711 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:53.805363 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2927fd-55db459cf-lqzdk" Oct 30 14:40:53.819711 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:53.806043 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac0599bf-47df-45a3-96a3-55c24b5a2afd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2927fd-844ddcc4bd-wbrs7\" (UID: \"ac0599bf-47df-45a3-96a3-55c24b5a2afd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2927fd-844ddcc4bd-wbrs7" Oct 30 14:40:53.819711 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:53.806089 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzczq\" (UniqueName: \"kubernetes.io/projected/ac0599bf-47df-45a3-96a3-55c24b5a2afd-kube-api-access-zzczq\") pod \"payload-processing-e2e-ipp-a-2927fd-844ddcc4bd-wbrs7\" (UID: \"ac0599bf-47df-45a3-96a3-55c24b5a2afd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2927fd-844ddcc4bd-wbrs7" Oct 30 14:40:53.921668 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:53.908754 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac0599bf-47df-45a3-96a3-55c24b5a2afd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2927fd-844ddcc4bd-wbrs7\" (UID: \"ac0599bf-47df-45a3-96a3-55c24b5a2afd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2927fd-844ddcc4bd-wbrs7" Oct 30 14:40:53.921668 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:53.908813 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzczq\" (UniqueName: \"kubernetes.io/projected/ac0599bf-47df-45a3-96a3-55c24b5a2afd-kube-api-access-zzczq\") pod \"payload-processing-e2e-ipp-a-2927fd-844ddcc4bd-wbrs7\" (UID: \"ac0599bf-47df-45a3-96a3-55c24b5a2afd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2927fd-844ddcc4bd-wbrs7" Oct 30 14:40:53.922568 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:53.922531 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac0599bf-47df-45a3-96a3-55c24b5a2afd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2927fd-844ddcc4bd-wbrs7\" (UID: \"ac0599bf-47df-45a3-96a3-55c24b5a2afd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2927fd-844ddcc4bd-wbrs7" Oct 30 14:40:53.938841 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:53.938805 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzczq\" (UniqueName: \"kubernetes.io/projected/ac0599bf-47df-45a3-96a3-55c24b5a2afd-kube-api-access-zzczq\") pod \"payload-processing-e2e-ipp-a-2927fd-844ddcc4bd-wbrs7\" (UID: \"ac0599bf-47df-45a3-96a3-55c24b5a2afd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2927fd-844ddcc4bd-wbrs7" Oct 30 14:40:53.978516 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:53.978483 4982 patch_prober.go:29] interesting pod/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 30 14:40:53.978688 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:53.978549 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2" podUID="f8c75528-f183-4379-841e-e413ac8d9c35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 30 14:40:54.127852 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:54.127762 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2927fd-844ddcc4bd-wbrs7" Oct 30 14:40:54.214770 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:54.214736 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2927fd-55db459cf-lqzdk"] Oct 30 14:40:54.554718 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:54.551522 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2927fd-844ddcc4bd-wbrs7"] Oct 30 14:40:54.982471 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:54.982435 4982 patch_prober.go:29] interesting pod/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 30 14:40:54.982801 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:54.982758 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2" podUID="f8c75528-f183-4379-841e-e413ac8d9c35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 30 14:40:55.044607 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:55.042364 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2927fd-844ddcc4bd-wbrs7" event={"ID":"ac0599bf-47df-45a3-96a3-55c24b5a2afd","Type":"ContainerStarted","Data":"a062f00c57ee37f4201c3e5880bf0acdb49de6b18714fc39f8c4943372efe3ac"} Oct 30 14:40:55.044607 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:55.042440 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2927fd-844ddcc4bd-wbrs7" event={"ID":"ac0599bf-47df-45a3-96a3-55c24b5a2afd","Type":"ContainerStarted","Data":"adc5310d3551a031cbcae553e1182a3fa18feb60fa64a70eec3c460eefec9fb7"} Oct 30 14:40:55.044607 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:55.043631 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2927fd-844ddcc4bd-wbrs7" Oct 30 14:40:55.048301 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:55.047459 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2927fd-55db459cf-lqzdk" event={"ID":"920bf174-09b0-4966-b029-4e200373359b","Type":"ContainerStarted","Data":"3036a0f174d32eab2eddbdf5612262798ac7fd61ebd51ca95b8736d1ae0909d5"} Oct 30 14:40:55.048301 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:55.047518 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2927fd-55db459cf-lqzdk" event={"ID":"920bf174-09b0-4966-b029-4e200373359b","Type":"ContainerStarted","Data":"9ce41e9e4fb2daebd4ad159fa4b968cb6121302b6dcaf2d12da4a1b54cd9beeb"} Oct 30 14:40:55.048301 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:55.048245 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2927fd-55db459cf-lqzdk" Oct 30 14:40:55.107784 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:55.107660 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-2927fd-844ddcc4bd-wbrs7" podStartSLOduration=2.107623026 podStartE2EDuration="2.107623026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:40:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:40:55.105049267 +0000 UTC m=+3124.849183399" watchObservedRunningTime="2026-10-30 14:40:55.107623026 +0000 UTC m=+3124.851757068" Oct 30 14:40:55.498720 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:55.491081 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:40:55.978880 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:55.978845 4982 patch_prober.go:29] interesting pod/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 30 14:40:55.979280 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:55.978915 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2" podUID="f8c75528-f183-4379-841e-e413ac8d9c35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 30 14:40:56.978766 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:56.978733 4982 patch_prober.go:29] interesting pod/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 30 14:40:56.978929 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:56.978784 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2" podUID="f8c75528-f183-4379-841e-e413ac8d9c35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 30 14:40:57.182609 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:57.182559 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:40:57.376916 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:57.376884 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:40:57.417928 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:57.417872 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2927fd-55db459cf-lqzdk" podStartSLOduration=4.417855363 podStartE2EDuration="4.417855363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:40:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:40:55.204200416 +0000 UTC m=+3124.948334458" watchObservedRunningTime="2026-10-30 14:40:57.417855363 +0000 UTC m=+3127.161989405" Oct 30 14:40:57.420952 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:57.420921 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-33c209-6f4597cc68-xwdkx"] Oct 30 14:40:57.460769 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:57.460734 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-33c209-bffd664d7-kcz62"] Oct 30 14:40:57.627454 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:57.627354 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:40:57.792290 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:57.792248 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:40:57.995334 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:57.995243 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2" Oct 30 14:40:57.996903 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:57.996880 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2" Oct 30 14:40:57.998263 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:57.998242 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2" Oct 30 14:40:58.039971 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:40:58.039931 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:40:58.611006 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:40:58.610957 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:41:00.546008 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:00.545964 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:41:00.695117 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:00.695076 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:41:01.425500 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:01.425458 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w6-0efe43/rate-test-model-0efe43-kserve-6f8954844f-d5kvb" podUID="8da898db-f45c-4c84-a91b-15129bfeffc9" containerName="main" containerID="cri-o://63ea54cacf799cc9c64d7ada08e7660834365319cd484bc26b4d27a11e321d18" gracePeriod=45 Oct 30 14:41:01.723268 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:01.723238 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-0efe43/rate-test-model-0efe43-kserve-6f8954844f-d5kvb" Oct 30 14:41:01.770663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:01.769352 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8da898db-f45c-4c84-a91b-15129bfeffc9-kube-api-access-cltsx\" (UniqueName: \"kubernetes.io/projected/8da898db-f45c-4c84-a91b-15129bfeffc9-kube-api-access-cltsx\") pod \"8da898db-f45c-4c84-a91b-15129bfeffc9\" (UID: \"8da898db-f45c-4c84-a91b-15129bfeffc9\") " Oct 30 14:41:01.770663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:01.769612 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8da898db-f45c-4c84-a91b-15129bfeffc9-home\" (UniqueName: \"kubernetes.io/empty-dir/8da898db-f45c-4c84-a91b-15129bfeffc9-home\") pod \"8da898db-f45c-4c84-a91b-15129bfeffc9\" (UID: \"8da898db-f45c-4c84-a91b-15129bfeffc9\") " Oct 30 14:41:01.770663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:01.769765 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8da898db-f45c-4c84-a91b-15129bfeffc9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8da898db-f45c-4c84-a91b-15129bfeffc9-dshm\") pod \"8da898db-f45c-4c84-a91b-15129bfeffc9\" (UID: \"8da898db-f45c-4c84-a91b-15129bfeffc9\") " Oct 30 14:41:01.770663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:01.769847 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8da898db-f45c-4c84-a91b-15129bfeffc9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8da898db-f45c-4c84-a91b-15129bfeffc9-model-cache\") pod \"8da898db-f45c-4c84-a91b-15129bfeffc9\" (UID: \"8da898db-f45c-4c84-a91b-15129bfeffc9\") " Oct 30 14:41:01.770663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:01.769962 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8da898db-f45c-4c84-a91b-15129bfeffc9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8da898db-f45c-4c84-a91b-15129bfeffc9-tmp-dir\") pod \"8da898db-f45c-4c84-a91b-15129bfeffc9\" (UID: \"8da898db-f45c-4c84-a91b-15129bfeffc9\") " Oct 30 14:41:01.770663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:01.769985 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8da898db-f45c-4c84-a91b-15129bfeffc9-tls-certs\" (UniqueName: \"kubernetes.io/secret/8da898db-f45c-4c84-a91b-15129bfeffc9-tls-certs\") pod \"8da898db-f45c-4c84-a91b-15129bfeffc9\" (UID: \"8da898db-f45c-4c84-a91b-15129bfeffc9\") " Oct 30 14:41:01.779879 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:01.779692 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8da898db-f45c-4c84-a91b-15129bfeffc9-model-cache" pod "8da898db-f45c-4c84-a91b-15129bfeffc9" (UID: "8da898db-f45c-4c84-a91b-15129bfeffc9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:41:01.780170 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:01.780132 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8da898db-f45c-4c84-a91b-15129bfeffc9-tmp-dir" pod "8da898db-f45c-4c84-a91b-15129bfeffc9" (UID: "8da898db-f45c-4c84-a91b-15129bfeffc9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:41:01.783726 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:01.783676 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8da898db-f45c-4c84-a91b-15129bfeffc9-home" pod "8da898db-f45c-4c84-a91b-15129bfeffc9" (UID: "8da898db-f45c-4c84-a91b-15129bfeffc9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:41:01.798585 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:01.796980 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8da898db-f45c-4c84-a91b-15129bfeffc9-tls-certs" pod "8da898db-f45c-4c84-a91b-15129bfeffc9" (UID: "8da898db-f45c-4c84-a91b-15129bfeffc9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:41:01.812672 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:01.803164 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8da898db-f45c-4c84-a91b-15129bfeffc9-kube-api-access-cltsx" pod "8da898db-f45c-4c84-a91b-15129bfeffc9" (UID: "8da898db-f45c-4c84-a91b-15129bfeffc9"). InnerVolumeSpecName "kube-api-access-cltsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:41:01.821284 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:01.821082 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8da898db-f45c-4c84-a91b-15129bfeffc9-dshm" pod "8da898db-f45c-4c84-a91b-15129bfeffc9" (UID: "8da898db-f45c-4c84-a91b-15129bfeffc9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:41:01.871621 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:01.871588 4982 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8da898db-f45c-4c84-a91b-15129bfeffc9-tmp-dir\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:41:01.871621 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:01.871617 4982 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8da898db-f45c-4c84-a91b-15129bfeffc9-tls-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:41:01.871621 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:01.871626 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cltsx\" (UniqueName: \"kubernetes.io/projected/8da898db-f45c-4c84-a91b-15129bfeffc9-kube-api-access-cltsx\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:41:01.871879 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:01.871633 4982 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8da898db-f45c-4c84-a91b-15129bfeffc9-home\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:41:01.871879 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:01.871679 4982 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8da898db-f45c-4c84-a91b-15129bfeffc9-dshm\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:41:01.871879 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:01.871687 4982 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8da898db-f45c-4c84-a91b-15129bfeffc9-model-cache\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:41:02.111380 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:02.111344 4982 generic.go:356] "Generic (PLEG): container finished" podID="8da898db-f45c-4c84-a91b-15129bfeffc9" containerID="63ea54cacf799cc9c64d7ada08e7660834365319cd484bc26b4d27a11e321d18" exitCode=0 Oct 30 14:41:02.111535 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:02.111383 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-0efe43/rate-test-model-0efe43-kserve-6f8954844f-d5kvb" Oct 30 14:41:02.111535 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:02.111421 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-0efe43/rate-test-model-0efe43-kserve-6f8954844f-d5kvb" event={"ID":"8da898db-f45c-4c84-a91b-15129bfeffc9","Type":"ContainerDied","Data":"63ea54cacf799cc9c64d7ada08e7660834365319cd484bc26b4d27a11e321d18"} Oct 30 14:41:02.111535 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:02.111474 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-0efe43/rate-test-model-0efe43-kserve-6f8954844f-d5kvb" event={"ID":"8da898db-f45c-4c84-a91b-15129bfeffc9","Type":"ContainerDied","Data":"fb7b7e36c2a03aafb1f7727eb4d774d7f56c2db4fdad1adf822c71e7efebb333"} Oct 30 14:41:02.111535 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:02.111496 4982 scope.go:122] "RemoveContainer" containerID="63ea54cacf799cc9c64d7ada08e7660834365319cd484bc26b4d27a11e321d18" Oct 30 14:41:02.123218 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:02.123190 4982 scope.go:122] "RemoveContainer" containerID="63ea54cacf799cc9c64d7ada08e7660834365319cd484bc26b4d27a11e321d18" Oct 30 14:41:02.123508 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:41:02.123483 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63ea54cacf799cc9c64d7ada08e7660834365319cd484bc26b4d27a11e321d18\": container with ID starting with 63ea54cacf799cc9c64d7ada08e7660834365319cd484bc26b4d27a11e321d18 not found: ID does not exist" containerID="63ea54cacf799cc9c64d7ada08e7660834365319cd484bc26b4d27a11e321d18" Oct 30 14:41:02.123602 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:02.123519 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63ea54cacf799cc9c64d7ada08e7660834365319cd484bc26b4d27a11e321d18"} err="failed to get container status \"63ea54cacf799cc9c64d7ada08e7660834365319cd484bc26b4d27a11e321d18\": rpc error: code = NotFound desc = could not find container \"63ea54cacf799cc9c64d7ada08e7660834365319cd484bc26b4d27a11e321d18\": container with ID starting with 63ea54cacf799cc9c64d7ada08e7660834365319cd484bc26b4d27a11e321d18 not found: ID does not exist" Oct 30 14:41:02.174107 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:02.174075 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-0efe43/rate-test-model-0efe43-kserve-6f8954844f-d5kvb"] Oct 30 14:41:02.187003 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:02.186982 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w6-0efe43/rate-test-model-0efe43-kserve-6f8954844f-d5kvb"] Oct 30 14:41:02.401697 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:02.401612 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8da898db-f45c-4c84-a91b-15129bfeffc9" path="/var/lib/kubelet/pods/8da898db-f45c-4c84-a91b-15129bfeffc9/volumes" Oct 30 14:41:02.421260 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:02.421222 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-33c209-6f4597cc68-xwdkx" podUID="97a44d13-3765-4192-be16-a736077a52ef" containerName="payload-processing" containerID="cri-o://861c805a7801b9aecc623a7279b75a1e2535804849e88807693dd9d2129f1959" gracePeriod=25 Oct 30 14:41:02.461915 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:02.461887 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c209-bffd664d7-kcz62" podUID="18484bef-725d-4ac1-9b31-70195e0eeb38" containerName="payload-pre-processing" containerID="cri-o://8d2395ada5069df0c0df8ced363f31b6defc6fdc1ed5c5902b1efd2b799f0ce9" gracePeriod=25 Oct 30 14:41:02.616342 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:02.616265 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w6-0efe43/rate-test-model-0efe43-kserve-6f8954844f-d5kvb" podUID="8da898db-f45c-4c84-a91b-15129bfeffc9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.203:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:41:02.633753 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:02.632904 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:41:02.796538 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:02.796375 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-33c209-6f4597cc68-xwdkx" Oct 30 14:41:02.859780 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:02.859750 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c209-bffd664d7-kcz62" Oct 30 14:41:02.891362 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:02.890854 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97a44d13-3765-4192-be16-a736077a52ef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97a44d13-3765-4192-be16-a736077a52ef-plugins-config-volume\") pod \"97a44d13-3765-4192-be16-a736077a52ef\" (UID: \"97a44d13-3765-4192-be16-a736077a52ef\") " Oct 30 14:41:02.891362 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:02.890914 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97a44d13-3765-4192-be16-a736077a52ef-kube-api-access-kz8p5\" (UniqueName: \"kubernetes.io/projected/97a44d13-3765-4192-be16-a736077a52ef-kube-api-access-kz8p5\") pod \"97a44d13-3765-4192-be16-a736077a52ef\" (UID: \"97a44d13-3765-4192-be16-a736077a52ef\") " Oct 30 14:41:02.891362 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:02.891265 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97a44d13-3765-4192-be16-a736077a52ef-plugins-config-volume" pod "97a44d13-3765-4192-be16-a736077a52ef" (UID: "97a44d13-3765-4192-be16-a736077a52ef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:41:02.893173 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:02.893146 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97a44d13-3765-4192-be16-a736077a52ef-kube-api-access-kz8p5" pod "97a44d13-3765-4192-be16-a736077a52ef" (UID: "97a44d13-3765-4192-be16-a736077a52ef"). InnerVolumeSpecName "kube-api-access-kz8p5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:41:02.992704 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:02.992587 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18484bef-725d-4ac1-9b31-70195e0eeb38-kube-api-access-pgtt8\" (UniqueName: \"kubernetes.io/projected/18484bef-725d-4ac1-9b31-70195e0eeb38-kube-api-access-pgtt8\") pod \"18484bef-725d-4ac1-9b31-70195e0eeb38\" (UID: \"18484bef-725d-4ac1-9b31-70195e0eeb38\") " Oct 30 14:41:02.992704 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:02.992659 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18484bef-725d-4ac1-9b31-70195e0eeb38-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18484bef-725d-4ac1-9b31-70195e0eeb38-plugins-config-volume\") pod \"18484bef-725d-4ac1-9b31-70195e0eeb38\" (UID: \"18484bef-725d-4ac1-9b31-70195e0eeb38\") " Oct 30 14:41:02.993141 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:02.993112 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18484bef-725d-4ac1-9b31-70195e0eeb38-plugins-config-volume" pod "18484bef-725d-4ac1-9b31-70195e0eeb38" (UID: "18484bef-725d-4ac1-9b31-70195e0eeb38"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:41:02.993201 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:02.993175 4982 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97a44d13-3765-4192-be16-a736077a52ef-plugins-config-volume\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:41:02.993201 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:02.993191 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kz8p5\" (UniqueName: \"kubernetes.io/projected/97a44d13-3765-4192-be16-a736077a52ef-kube-api-access-kz8p5\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:41:02.994948 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:02.994917 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18484bef-725d-4ac1-9b31-70195e0eeb38-kube-api-access-pgtt8" pod "18484bef-725d-4ac1-9b31-70195e0eeb38" (UID: "18484bef-725d-4ac1-9b31-70195e0eeb38"). InnerVolumeSpecName "kube-api-access-pgtt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:41:03.094022 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:03.093988 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgtt8\" (UniqueName: \"kubernetes.io/projected/18484bef-725d-4ac1-9b31-70195e0eeb38-kube-api-access-pgtt8\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:41:03.094022 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:03.094017 4982 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18484bef-725d-4ac1-9b31-70195e0eeb38-plugins-config-volume\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:41:03.122529 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:03.122499 4982 generic.go:356] "Generic (PLEG): container finished" podID="97a44d13-3765-4192-be16-a736077a52ef" containerID="861c805a7801b9aecc623a7279b75a1e2535804849e88807693dd9d2129f1959" exitCode=0 Oct 30 14:41:03.122723 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:03.122563 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-33c209-6f4597cc68-xwdkx" Oct 30 14:41:03.122723 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:03.122593 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-33c209-6f4597cc68-xwdkx" event={"ID":"97a44d13-3765-4192-be16-a736077a52ef","Type":"ContainerDied","Data":"861c805a7801b9aecc623a7279b75a1e2535804849e88807693dd9d2129f1959"} Oct 30 14:41:03.122723 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:03.122699 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-33c209-6f4597cc68-xwdkx" event={"ID":"97a44d13-3765-4192-be16-a736077a52ef","Type":"ContainerDied","Data":"c15e8308eb2721639b795a992ead7493d0be3a7896ea9c1eff6193de1a4f535c"} Oct 30 14:41:03.122909 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:03.122731 4982 scope.go:122] "RemoveContainer" containerID="861c805a7801b9aecc623a7279b75a1e2535804849e88807693dd9d2129f1959" Oct 30 14:41:03.124427 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:03.124401 4982 generic.go:356] "Generic (PLEG): container finished" podID="18484bef-725d-4ac1-9b31-70195e0eeb38" containerID="8d2395ada5069df0c0df8ced363f31b6defc6fdc1ed5c5902b1efd2b799f0ce9" exitCode=0 Oct 30 14:41:03.124560 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:03.124441 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c209-bffd664d7-kcz62" Oct 30 14:41:03.124560 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:03.124506 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c209-bffd664d7-kcz62" event={"ID":"18484bef-725d-4ac1-9b31-70195e0eeb38","Type":"ContainerDied","Data":"8d2395ada5069df0c0df8ced363f31b6defc6fdc1ed5c5902b1efd2b799f0ce9"} Oct 30 14:41:03.124560 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:03.124546 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c209-bffd664d7-kcz62" event={"ID":"18484bef-725d-4ac1-9b31-70195e0eeb38","Type":"ContainerDied","Data":"ed5d14211087a0d2aa6daf23e91980140fc4f954b9b4ba1f22599847cf40e0ba"} Oct 30 14:41:03.134386 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:03.134369 4982 scope.go:122] "RemoveContainer" containerID="861c805a7801b9aecc623a7279b75a1e2535804849e88807693dd9d2129f1959" Oct 30 14:41:03.134727 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:41:03.134708 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"861c805a7801b9aecc623a7279b75a1e2535804849e88807693dd9d2129f1959\": container with ID starting with 861c805a7801b9aecc623a7279b75a1e2535804849e88807693dd9d2129f1959 not found: ID does not exist" containerID="861c805a7801b9aecc623a7279b75a1e2535804849e88807693dd9d2129f1959" Oct 30 14:41:03.134794 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:03.134734 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"861c805a7801b9aecc623a7279b75a1e2535804849e88807693dd9d2129f1959"} err="failed to get container status \"861c805a7801b9aecc623a7279b75a1e2535804849e88807693dd9d2129f1959\": rpc error: code = NotFound desc = could not find container \"861c805a7801b9aecc623a7279b75a1e2535804849e88807693dd9d2129f1959\": container with ID starting with 861c805a7801b9aecc623a7279b75a1e2535804849e88807693dd9d2129f1959 not found: ID does not exist" Oct 30 14:41:03.134794 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:03.134753 4982 scope.go:122] "RemoveContainer" containerID="8d2395ada5069df0c0df8ced363f31b6defc6fdc1ed5c5902b1efd2b799f0ce9" Oct 30 14:41:03.144415 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:03.144400 4982 scope.go:122] "RemoveContainer" containerID="8d2395ada5069df0c0df8ced363f31b6defc6fdc1ed5c5902b1efd2b799f0ce9" Oct 30 14:41:03.144760 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:41:03.144739 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d2395ada5069df0c0df8ced363f31b6defc6fdc1ed5c5902b1efd2b799f0ce9\": container with ID starting with 8d2395ada5069df0c0df8ced363f31b6defc6fdc1ed5c5902b1efd2b799f0ce9 not found: ID does not exist" containerID="8d2395ada5069df0c0df8ced363f31b6defc6fdc1ed5c5902b1efd2b799f0ce9" Oct 30 14:41:03.144798 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:03.144766 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d2395ada5069df0c0df8ced363f31b6defc6fdc1ed5c5902b1efd2b799f0ce9"} err="failed to get container status \"8d2395ada5069df0c0df8ced363f31b6defc6fdc1ed5c5902b1efd2b799f0ce9\": rpc error: code = NotFound desc = could not find container \"8d2395ada5069df0c0df8ced363f31b6defc6fdc1ed5c5902b1efd2b799f0ce9\": container with ID starting with 8d2395ada5069df0c0df8ced363f31b6defc6fdc1ed5c5902b1efd2b799f0ce9 not found: ID does not exist" Oct 30 14:41:03.183553 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:03.183529 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-33c209-bffd664d7-kcz62"] Oct 30 14:41:03.196795 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:03.196768 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-33c209-bffd664d7-kcz62"] Oct 30 14:41:03.237649 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:03.237613 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-33c209-6f4597cc68-xwdkx"] Oct 30 14:41:03.242607 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:03.242577 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-33c209-6f4597cc68-xwdkx"] Oct 30 14:41:04.404521 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:04.404485 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18484bef-725d-4ac1-9b31-70195e0eeb38" path="/var/lib/kubelet/pods/18484bef-725d-4ac1-9b31-70195e0eeb38/volumes" Oct 30 14:41:04.405137 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:04.405115 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97a44d13-3765-4192-be16-a736077a52ef" path="/var/lib/kubelet/pods/97a44d13-3765-4192-be16-a736077a52ef/volumes" Oct 30 14:41:04.670492 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:04.670408 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:41:06.039631 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:06.039600 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9346a31-qnmmw"] Oct 30 14:41:06.043670 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:06.043632 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9346a31-qnmmw"] Oct 30 14:41:06.401747 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:06.401712 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4acdc09c-aeff-4e7d-a67f-b8fe04ce5736" path="/var/lib/kubelet/pods/4acdc09c-aeff-4e7d-a67f-b8fe04ce5736/volumes" Oct 30 14:41:06.477710 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:06.477672 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6176c3-645458f68c-q8b98"] Oct 30 14:41:06.479316 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:06.479284 4982 cpu_manager.go:414] "containerMap: removing container" podUID="8da898db-f45c-4c84-a91b-15129bfeffc9" containerName="main" Oct 30 14:41:06.479316 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:06.479314 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8da898db-f45c-4c84-a91b-15129bfeffc9" containerName="main" Oct 30 14:41:06.479510 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:06.479339 4982 cpu_manager.go:414] "containerMap: removing container" podUID="18484bef-725d-4ac1-9b31-70195e0eeb38" containerName="payload-pre-processing" Oct 30 14:41:06.479510 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:06.479347 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18484bef-725d-4ac1-9b31-70195e0eeb38" containerName="payload-pre-processing" Oct 30 14:41:06.479510 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:06.479382 4982 cpu_manager.go:414] "containerMap: removing container" podUID="97a44d13-3765-4192-be16-a736077a52ef" containerName="payload-processing" Oct 30 14:41:06.479510 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:06.479390 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97a44d13-3765-4192-be16-a736077a52ef" containerName="payload-processing" Oct 30 14:41:06.479771 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:06.479755 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="8da898db-f45c-4c84-a91b-15129bfeffc9" containerName="main" Oct 30 14:41:06.479836 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:06.479773 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="18484bef-725d-4ac1-9b31-70195e0eeb38" containerName="payload-pre-processing" Oct 30 14:41:06.479836 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:06.479795 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="97a44d13-3765-4192-be16-a736077a52ef" containerName="payload-processing" Oct 30 14:41:06.511111 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:06.511080 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6176c3-645458f68c-q8b98"] Oct 30 14:41:06.512224 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:06.511197 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6176c3-645458f68c-q8b98" Oct 30 14:41:06.529818 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:06.529794 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-6176c3-serving-cert\"" Oct 30 14:41:06.609098 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:06.609063 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:41:06.644185 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:06.644150 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bd7ef80-e959-407e-8b8e-145600cdbc39-maas-api-tls\") pod \"maas-api-e2e-ipp-a-6176c3-645458f68c-q8b98\" (UID: \"3bd7ef80-e959-407e-8b8e-145600cdbc39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6176c3-645458f68c-q8b98" Oct 30 14:41:06.644335 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:06.644189 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bd7ef80-e959-407e-8b8e-145600cdbc39-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-6176c3-645458f68c-q8b98\" (UID: \"3bd7ef80-e959-407e-8b8e-145600cdbc39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6176c3-645458f68c-q8b98" Oct 30 14:41:06.644471 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:06.644386 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm9mz\" (UniqueName: \"kubernetes.io/projected/3bd7ef80-e959-407e-8b8e-145600cdbc39-kube-api-access-gm9mz\") pod \"maas-api-e2e-ipp-a-6176c3-645458f68c-q8b98\" (UID: \"3bd7ef80-e959-407e-8b8e-145600cdbc39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6176c3-645458f68c-q8b98" Oct 30 14:41:06.644471 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:06.644442 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bd7ef80-e959-407e-8b8e-145600cdbc39-metrics-certs\") pod \"maas-api-e2e-ipp-a-6176c3-645458f68c-q8b98\" (UID: \"3bd7ef80-e959-407e-8b8e-145600cdbc39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6176c3-645458f68c-q8b98" Oct 30 14:41:06.745603 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:06.745520 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm9mz\" (UniqueName: \"kubernetes.io/projected/3bd7ef80-e959-407e-8b8e-145600cdbc39-kube-api-access-gm9mz\") pod \"maas-api-e2e-ipp-a-6176c3-645458f68c-q8b98\" (UID: \"3bd7ef80-e959-407e-8b8e-145600cdbc39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6176c3-645458f68c-q8b98" Oct 30 14:41:06.745603 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:06.745569 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bd7ef80-e959-407e-8b8e-145600cdbc39-metrics-certs\") pod \"maas-api-e2e-ipp-a-6176c3-645458f68c-q8b98\" (UID: \"3bd7ef80-e959-407e-8b8e-145600cdbc39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6176c3-645458f68c-q8b98" Oct 30 14:41:06.745827 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:06.745719 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bd7ef80-e959-407e-8b8e-145600cdbc39-maas-api-tls\") pod \"maas-api-e2e-ipp-a-6176c3-645458f68c-q8b98\" (UID: \"3bd7ef80-e959-407e-8b8e-145600cdbc39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6176c3-645458f68c-q8b98" Oct 30 14:41:06.745827 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:06.745745 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bd7ef80-e959-407e-8b8e-145600cdbc39-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-6176c3-645458f68c-q8b98\" (UID: \"3bd7ef80-e959-407e-8b8e-145600cdbc39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6176c3-645458f68c-q8b98" Oct 30 14:41:06.746444 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:06.746419 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bd7ef80-e959-407e-8b8e-145600cdbc39-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-6176c3-645458f68c-q8b98\" (UID: \"3bd7ef80-e959-407e-8b8e-145600cdbc39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6176c3-645458f68c-q8b98" Oct 30 14:41:06.748763 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:06.748740 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bd7ef80-e959-407e-8b8e-145600cdbc39-maas-api-tls\") pod \"maas-api-e2e-ipp-a-6176c3-645458f68c-q8b98\" (UID: \"3bd7ef80-e959-407e-8b8e-145600cdbc39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6176c3-645458f68c-q8b98" Oct 30 14:41:06.749221 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:06.749197 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bd7ef80-e959-407e-8b8e-145600cdbc39-metrics-certs\") pod \"maas-api-e2e-ipp-a-6176c3-645458f68c-q8b98\" (UID: \"3bd7ef80-e959-407e-8b8e-145600cdbc39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6176c3-645458f68c-q8b98" Oct 30 14:41:06.761859 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:06.761832 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:41:06.772837 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:06.772805 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm9mz\" (UniqueName: \"kubernetes.io/projected/3bd7ef80-e959-407e-8b8e-145600cdbc39-kube-api-access-gm9mz\") pod \"maas-api-e2e-ipp-a-6176c3-645458f68c-q8b98\" (UID: \"3bd7ef80-e959-407e-8b8e-145600cdbc39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6176c3-645458f68c-q8b98" Oct 30 14:41:06.827804 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:06.827770 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6176c3-645458f68c-q8b98" Oct 30 14:41:07.068713 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:07.068685 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2927fd-844ddcc4bd-wbrs7" Oct 30 14:41:07.069087 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:07.068756 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2927fd-55db459cf-lqzdk" Oct 30 14:41:07.145679 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:07.145650 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6176c3-645458f68c-q8b98"] Oct 30 14:41:07.146923 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:41:07.146899 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bd7ef80_e959_407e_8b8e_145600cdbc39.slice/crio-ccbe4ffe794161be7977a33fd6b9b3fbbaa32293cc6f26eca0318073a2a2d83a WatchSource:0}: Error finding container ccbe4ffe794161be7977a33fd6b9b3fbbaa32293cc6f26eca0318073a2a2d83a: Status 404 returned error can't find the container with id ccbe4ffe794161be7977a33fd6b9b3fbbaa32293cc6f26eca0318073a2a2d83a Oct 30 14:41:07.162009 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:07.161970 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6176c3-645458f68c-q8b98" event={"ID":"3bd7ef80-e959-407e-8b8e-145600cdbc39","Type":"ContainerStarted","Data":"ccbe4ffe794161be7977a33fd6b9b3fbbaa32293cc6f26eca0318073a2a2d83a"} Oct 30 14:41:08.170874 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:08.170836 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6176c3-645458f68c-q8b98" event={"ID":"3bd7ef80-e959-407e-8b8e-145600cdbc39","Type":"ContainerStarted","Data":"bde418952e69bd7dd22846190fa299be2c90428bf8f281150e477f42d04f8a5e"} Oct 30 14:41:08.171365 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:08.171054 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6176c3-645458f68c-q8b98" Oct 30 14:41:08.227169 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:08.227109 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6176c3-645458f68c-q8b98" podStartSLOduration=1.813784361 podStartE2EDuration="2.227092585s" totalImagesPullingTime="413.308224ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:41:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:41:08.224116721 +0000 UTC m=+3137.968250768" watchObservedRunningTime="2026-10-30 14:41:08.227092585 +0000 UTC m=+3137.971226629" Oct 30 14:41:08.814124 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:08.814079 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:41:10.931567 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:10.931521 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:41:12.528473 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:12.528442 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:41:12.827594 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:12.827533 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:41:13.148119 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:13.148026 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:41:13.312052 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:13.312011 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:41:14.187879 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:14.187844 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6176c3-645458f68c-q8b98" Oct 30 14:41:14.923965 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:14.923929 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:41:16.075296 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:16.075258 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6f920ce-qgqnn"] Oct 30 14:41:16.088754 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:16.088715 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6f920ce-qgqnn"] Oct 30 14:41:16.400846 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:16.400753 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c013f868-ccac-4409-b5cc-ce01b6dd954a" path="/var/lib/kubelet/pods/c013f868-ccac-4409-b5cc-ce01b6dd954a/volumes" Oct 30 14:41:16.800612 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:16.800515 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:41:17.023661 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:17.023615 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:41:17.479126 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:17.479091 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:41:18.699796 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:18.699754 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:41:18.739921 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:18.739880 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3b95dd-858549854c-z49d4"] Oct 30 14:41:18.745632 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:18.745614 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b95dd-858549854c-z49d4" Oct 30 14:41:18.764751 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:18.764726 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-3b95dd\"" Oct 30 14:41:18.782345 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:18.782319 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3b95dd-858549854c-z49d4"] Oct 30 14:41:18.825623 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:18.825582 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe8f23d0-2fda-4393-967d-c21cbbc8afe9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3b95dd-858549854c-z49d4\" (UID: \"fe8f23d0-2fda-4393-967d-c21cbbc8afe9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b95dd-858549854c-z49d4" Oct 30 14:41:18.825804 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:18.825727 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssxqs\" (UniqueName: \"kubernetes.io/projected/fe8f23d0-2fda-4393-967d-c21cbbc8afe9-kube-api-access-ssxqs\") pod \"payload-pre-processing-e2e-ipp-b-3b95dd-858549854c-z49d4\" (UID: \"fe8f23d0-2fda-4393-967d-c21cbbc8afe9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b95dd-858549854c-z49d4" Oct 30 14:41:18.861023 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:18.860994 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3b95dd-5ccb7d956-4mkxk"] Oct 30 14:41:18.865564 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:18.865546 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3b95dd-5ccb7d956-4mkxk" Oct 30 14:41:18.895685 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:18.895635 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-3b95dd-dockercfg-rpbq6\"" Oct 30 14:41:18.900517 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:18.900473 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3b95dd-5ccb7d956-4mkxk"] Oct 30 14:41:19.038651 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:19.013380 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a53fc31f-d9a7-45d9-8564-1e9b695c9bbf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3b95dd-5ccb7d956-4mkxk\" (UID: \"a53fc31f-d9a7-45d9-8564-1e9b695c9bbf\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3b95dd-5ccb7d956-4mkxk" Oct 30 14:41:19.038651 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:19.013487 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe8f23d0-2fda-4393-967d-c21cbbc8afe9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3b95dd-858549854c-z49d4\" (UID: \"fe8f23d0-2fda-4393-967d-c21cbbc8afe9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b95dd-858549854c-z49d4" Oct 30 14:41:19.038651 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:19.013669 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssxqs\" (UniqueName: \"kubernetes.io/projected/fe8f23d0-2fda-4393-967d-c21cbbc8afe9-kube-api-access-ssxqs\") pod \"payload-pre-processing-e2e-ipp-b-3b95dd-858549854c-z49d4\" (UID: \"fe8f23d0-2fda-4393-967d-c21cbbc8afe9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b95dd-858549854c-z49d4" Oct 30 14:41:19.038651 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:19.013717 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjfzd\" (UniqueName: \"kubernetes.io/projected/a53fc31f-d9a7-45d9-8564-1e9b695c9bbf-kube-api-access-rjfzd\") pod \"payload-processing-e2e-ipp-b-3b95dd-5ccb7d956-4mkxk\" (UID: \"a53fc31f-d9a7-45d9-8564-1e9b695c9bbf\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3b95dd-5ccb7d956-4mkxk" Oct 30 14:41:19.038651 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:19.015193 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe8f23d0-2fda-4393-967d-c21cbbc8afe9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3b95dd-858549854c-z49d4\" (UID: \"fe8f23d0-2fda-4393-967d-c21cbbc8afe9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b95dd-858549854c-z49d4" Oct 30 14:41:19.042938 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:19.041365 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:41:19.090363 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:19.089755 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssxqs\" (UniqueName: \"kubernetes.io/projected/fe8f23d0-2fda-4393-967d-c21cbbc8afe9-kube-api-access-ssxqs\") pod \"payload-pre-processing-e2e-ipp-b-3b95dd-858549854c-z49d4\" (UID: \"fe8f23d0-2fda-4393-967d-c21cbbc8afe9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b95dd-858549854c-z49d4" Oct 30 14:41:19.115495 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:19.115456 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjfzd\" (UniqueName: \"kubernetes.io/projected/a53fc31f-d9a7-45d9-8564-1e9b695c9bbf-kube-api-access-rjfzd\") pod \"payload-processing-e2e-ipp-b-3b95dd-5ccb7d956-4mkxk\" (UID: \"a53fc31f-d9a7-45d9-8564-1e9b695c9bbf\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3b95dd-5ccb7d956-4mkxk" Oct 30 14:41:19.115724 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:19.115707 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a53fc31f-d9a7-45d9-8564-1e9b695c9bbf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3b95dd-5ccb7d956-4mkxk\" (UID: \"a53fc31f-d9a7-45d9-8564-1e9b695c9bbf\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3b95dd-5ccb7d956-4mkxk" Oct 30 14:41:19.116505 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:19.116485 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a53fc31f-d9a7-45d9-8564-1e9b695c9bbf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3b95dd-5ccb7d956-4mkxk\" (UID: \"a53fc31f-d9a7-45d9-8564-1e9b695c9bbf\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3b95dd-5ccb7d956-4mkxk" Oct 30 14:41:19.145438 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:19.145405 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjfzd\" (UniqueName: \"kubernetes.io/projected/a53fc31f-d9a7-45d9-8564-1e9b695c9bbf-kube-api-access-rjfzd\") pod \"payload-processing-e2e-ipp-b-3b95dd-5ccb7d956-4mkxk\" (UID: \"a53fc31f-d9a7-45d9-8564-1e9b695c9bbf\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3b95dd-5ccb7d956-4mkxk" Oct 30 14:41:19.179394 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:19.179373 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3b95dd-5ccb7d956-4mkxk" Oct 30 14:41:19.360648 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:19.360619 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b95dd-858549854c-z49d4" Oct 30 14:41:19.499961 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:41:19.499928 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda53fc31f_d9a7_45d9_8564_1e9b695c9bbf.slice/crio-10848c7a6d332b8aed6a2211925367237a8eb58015bdbc13b081114478a88286 WatchSource:0}: Error finding container 10848c7a6d332b8aed6a2211925367237a8eb58015bdbc13b081114478a88286: Status 404 returned error can't find the container with id 10848c7a6d332b8aed6a2211925367237a8eb58015bdbc13b081114478a88286 Oct 30 14:41:19.500304 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:19.500277 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3b95dd-5ccb7d956-4mkxk"] Oct 30 14:41:19.688905 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:19.688873 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3b95dd-858549854c-z49d4"] Oct 30 14:41:19.690970 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:41:19.690940 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe8f23d0_2fda_4393_967d_c21cbbc8afe9.slice/crio-00dc3839f9ca016064d64885daafaa5d6a61ec114a1f8a14900dbdea60849ea0 WatchSource:0}: Error finding container 00dc3839f9ca016064d64885daafaa5d6a61ec114a1f8a14900dbdea60849ea0: Status 404 returned error can't find the container with id 00dc3839f9ca016064d64885daafaa5d6a61ec114a1f8a14900dbdea60849ea0 Oct 30 14:41:19.785297 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:19.785265 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:41:20.283502 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:20.283464 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b95dd-858549854c-z49d4" event={"ID":"fe8f23d0-2fda-4393-967d-c21cbbc8afe9","Type":"ContainerStarted","Data":"54469c4a0663a338a6a17c7bc8093d3c756a622e43d43fb387efdfa902ad2e87"} Oct 30 14:41:20.283727 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:20.283537 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b95dd-858549854c-z49d4" Oct 30 14:41:20.283727 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:20.283564 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b95dd-858549854c-z49d4" event={"ID":"fe8f23d0-2fda-4393-967d-c21cbbc8afe9","Type":"ContainerStarted","Data":"00dc3839f9ca016064d64885daafaa5d6a61ec114a1f8a14900dbdea60849ea0"} Oct 30 14:41:20.285218 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:20.285188 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3b95dd-5ccb7d956-4mkxk" event={"ID":"a53fc31f-d9a7-45d9-8564-1e9b695c9bbf","Type":"ContainerStarted","Data":"c30a0521c45fb6929029a53be0fa71eeef9b5d563e600b3fb079364f3623f4e8"} Oct 30 14:41:20.285351 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:20.285234 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3b95dd-5ccb7d956-4mkxk" Oct 30 14:41:20.285351 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:20.285249 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3b95dd-5ccb7d956-4mkxk" event={"ID":"a53fc31f-d9a7-45d9-8564-1e9b695c9bbf","Type":"ContainerStarted","Data":"10848c7a6d332b8aed6a2211925367237a8eb58015bdbc13b081114478a88286"} Oct 30 14:41:20.345975 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:20.345928 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b95dd-858549854c-z49d4" podStartSLOduration=2.345914314 podStartE2EDuration="2.345914314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:41:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:41:20.328383816 +0000 UTC m=+3150.072517856" watchObservedRunningTime="2026-10-30 14:41:20.345914314 +0000 UTC m=+3150.090048351" Oct 30 14:41:20.376812 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:20.376771 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-3b95dd-5ccb7d956-4mkxk" podStartSLOduration=2.376758365 podStartE2EDuration="2.376758365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:41:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:41:20.373795803 +0000 UTC m=+3150.117929843" watchObservedRunningTime="2026-10-30 14:41:20.376758365 +0000 UTC m=+3150.120892471" Oct 30 14:41:21.957482 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:21.957439 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:41:24.197052 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:24.196961 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:41:25.632973 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:25.632940 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:41:25.844108 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:25.844075 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:41:26.187600 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:26.187568 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-460b0b-55d85f88fb-7rkhm"] Oct 30 14:41:26.192393 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:26.192372 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-460b0b-55d85f88fb-7rkhm" Oct 30 14:41:26.218784 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:26.218760 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-460b0b-55d85f88fb-7rkhm"] Oct 30 14:41:26.227056 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:26.227004 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-460b0b\"" Oct 30 14:41:26.229103 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:26.227708 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-460b0b-dockercfg-8z7s6\"" Oct 30 14:41:26.339673 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:26.339611 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d600402-b61d-48af-b02e-d38d9adf5b3b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-460b0b-55d85f88fb-7rkhm\" (UID: \"9d600402-b61d-48af-b02e-d38d9adf5b3b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-460b0b-55d85f88fb-7rkhm" Oct 30 14:41:26.339847 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:26.339758 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6zss\" (UniqueName: \"kubernetes.io/projected/9d600402-b61d-48af-b02e-d38d9adf5b3b-kube-api-access-n6zss\") pod \"payload-processing-e2e-ipp-b-460b0b-55d85f88fb-7rkhm\" (UID: \"9d600402-b61d-48af-b02e-d38d9adf5b3b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-460b0b-55d85f88fb-7rkhm" Oct 30 14:41:26.440906 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:26.440836 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6zss\" (UniqueName: \"kubernetes.io/projected/9d600402-b61d-48af-b02e-d38d9adf5b3b-kube-api-access-n6zss\") pod \"payload-processing-e2e-ipp-b-460b0b-55d85f88fb-7rkhm\" (UID: \"9d600402-b61d-48af-b02e-d38d9adf5b3b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-460b0b-55d85f88fb-7rkhm" Oct 30 14:41:26.441067 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:26.441047 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d600402-b61d-48af-b02e-d38d9adf5b3b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-460b0b-55d85f88fb-7rkhm\" (UID: \"9d600402-b61d-48af-b02e-d38d9adf5b3b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-460b0b-55d85f88fb-7rkhm" Oct 30 14:41:26.441861 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:26.441836 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d600402-b61d-48af-b02e-d38d9adf5b3b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-460b0b-55d85f88fb-7rkhm\" (UID: \"9d600402-b61d-48af-b02e-d38d9adf5b3b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-460b0b-55d85f88fb-7rkhm" Oct 30 14:41:26.467877 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:26.467851 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6zss\" (UniqueName: \"kubernetes.io/projected/9d600402-b61d-48af-b02e-d38d9adf5b3b-kube-api-access-n6zss\") pod \"payload-processing-e2e-ipp-b-460b0b-55d85f88fb-7rkhm\" (UID: \"9d600402-b61d-48af-b02e-d38d9adf5b3b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-460b0b-55d85f88fb-7rkhm" Oct 30 14:41:26.508866 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:26.508839 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-460b0b-55d85f88fb-7rkhm" Oct 30 14:41:26.828389 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:26.828348 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-460b0b-55d85f88fb-7rkhm"] Oct 30 14:41:26.828910 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:41:26.828385 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d600402_b61d_48af_b02e_d38d9adf5b3b.slice/crio-a5b6e5021ff16ccd8a6155a58608feae849ff103bfc7dcd2af47ecfe304f98da WatchSource:0}: Error finding container a5b6e5021ff16ccd8a6155a58608feae849ff103bfc7dcd2af47ecfe304f98da: Status 404 returned error can't find the container with id a5b6e5021ff16ccd8a6155a58608feae849ff103bfc7dcd2af47ecfe304f98da Oct 30 14:41:27.155205 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:27.155124 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:41:27.352888 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:27.352845 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-460b0b-55d85f88fb-7rkhm" event={"ID":"9d600402-b61d-48af-b02e-d38d9adf5b3b","Type":"ContainerStarted","Data":"71f3f2b588eac942975d20be6b5fccb32c43c3197ddcfe004adc0c438434ae6e"} Oct 30 14:41:27.353080 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:27.352918 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-460b0b-55d85f88fb-7rkhm" event={"ID":"9d600402-b61d-48af-b02e-d38d9adf5b3b","Type":"ContainerStarted","Data":"a5b6e5021ff16ccd8a6155a58608feae849ff103bfc7dcd2af47ecfe304f98da"} Oct 30 14:41:27.353080 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:27.352949 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-460b0b-55d85f88fb-7rkhm" Oct 30 14:41:27.405106 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:27.405051 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-460b0b-55d85f88fb-7rkhm" podStartSLOduration=1.40503343 podStartE2EDuration="1.40503343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:41:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:41:27.399976823 +0000 UTC m=+3157.144110877" watchObservedRunningTime="2026-10-30 14:41:27.40503343 +0000 UTC m=+3157.149167476" Oct 30 14:41:28.014114 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:41:28.014077 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:41:28.991517 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:28.991478 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:41:31.033169 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:31.033135 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:41:31.296163 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:31.296081 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3b95dd-5ccb7d956-4mkxk" Oct 30 14:41:31.296163 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:31.296161 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b95dd-858549854c-z49d4" Oct 30 14:41:33.020054 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:33.020016 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:41:35.040108 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:35.040045 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:41:35.844664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:35.841420 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6"] Oct 30 14:41:35.848343 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:35.848307 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6" Oct 30 14:41:35.895676 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:35.878663 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-e50e63-openshift-default-dockercfg-n5qtw\"" Oct 30 14:41:35.909707 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:35.903788 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6"] Oct 30 14:41:36.036992 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:36.036082 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6921ba6-c364-4314-8c37-b7a32e2b3be6-istio-data\") pod \"e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6\" (UID: \"f6921ba6-c364-4314-8c37-b7a32e2b3be6\") " pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6" Oct 30 14:41:36.036992 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:36.036133 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6921ba6-c364-4314-8c37-b7a32e2b3be6-istio-envoy\") pod \"e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6\" (UID: \"f6921ba6-c364-4314-8c37-b7a32e2b3be6\") " pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6" Oct 30 14:41:36.036992 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:36.036209 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6921ba6-c364-4314-8c37-b7a32e2b3be6-workload-socket\") pod \"e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6\" (UID: \"f6921ba6-c364-4314-8c37-b7a32e2b3be6\") " pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6" Oct 30 14:41:36.036992 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:36.036274 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6921ba6-c364-4314-8c37-b7a32e2b3be6-istio-podinfo\") pod \"e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6\" (UID: \"f6921ba6-c364-4314-8c37-b7a32e2b3be6\") " pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6" Oct 30 14:41:36.036992 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:36.036326 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6921ba6-c364-4314-8c37-b7a32e2b3be6-workload-certs\") pod \"e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6\" (UID: \"f6921ba6-c364-4314-8c37-b7a32e2b3be6\") " pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6" Oct 30 14:41:36.036992 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:36.036384 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6921ba6-c364-4314-8c37-b7a32e2b3be6-credential-socket\") pod \"e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6\" (UID: \"f6921ba6-c364-4314-8c37-b7a32e2b3be6\") " pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6" Oct 30 14:41:36.036992 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:36.036428 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6921ba6-c364-4314-8c37-b7a32e2b3be6-istiod-ca-cert\") pod \"e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6\" (UID: \"f6921ba6-c364-4314-8c37-b7a32e2b3be6\") " pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6" Oct 30 14:41:36.036992 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:36.036517 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6921ba6-c364-4314-8c37-b7a32e2b3be6-istio-token\") pod \"e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6\" (UID: \"f6921ba6-c364-4314-8c37-b7a32e2b3be6\") " pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6" Oct 30 14:41:36.036992 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:36.036565 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgmdb\" (UniqueName: \"kubernetes.io/projected/f6921ba6-c364-4314-8c37-b7a32e2b3be6-kube-api-access-cgmdb\") pod \"e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6\" (UID: \"f6921ba6-c364-4314-8c37-b7a32e2b3be6\") " pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6" Oct 30 14:41:36.137806 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:36.137714 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6921ba6-c364-4314-8c37-b7a32e2b3be6-credential-socket\") pod \"e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6\" (UID: \"f6921ba6-c364-4314-8c37-b7a32e2b3be6\") " pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6" Oct 30 14:41:36.137806 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:36.137768 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6921ba6-c364-4314-8c37-b7a32e2b3be6-istiod-ca-cert\") pod \"e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6\" (UID: \"f6921ba6-c364-4314-8c37-b7a32e2b3be6\") " pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6" Oct 30 14:41:36.138392 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:36.137845 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6921ba6-c364-4314-8c37-b7a32e2b3be6-istio-token\") pod \"e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6\" (UID: \"f6921ba6-c364-4314-8c37-b7a32e2b3be6\") " pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6" Oct 30 14:41:36.138392 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:36.137889 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgmdb\" (UniqueName: \"kubernetes.io/projected/f6921ba6-c364-4314-8c37-b7a32e2b3be6-kube-api-access-cgmdb\") pod \"e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6\" (UID: \"f6921ba6-c364-4314-8c37-b7a32e2b3be6\") " pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6" Oct 30 14:41:36.138392 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:36.137995 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6921ba6-c364-4314-8c37-b7a32e2b3be6-istio-data\") pod \"e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6\" (UID: \"f6921ba6-c364-4314-8c37-b7a32e2b3be6\") " pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6" Oct 30 14:41:36.138392 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:36.138023 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6921ba6-c364-4314-8c37-b7a32e2b3be6-istio-envoy\") pod \"e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6\" (UID: \"f6921ba6-c364-4314-8c37-b7a32e2b3be6\") " pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6" Oct 30 14:41:36.138392 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:36.138093 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6921ba6-c364-4314-8c37-b7a32e2b3be6-workload-socket\") pod \"e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6\" (UID: \"f6921ba6-c364-4314-8c37-b7a32e2b3be6\") " pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6" Oct 30 14:41:36.138392 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:36.138157 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6921ba6-c364-4314-8c37-b7a32e2b3be6-istio-podinfo\") pod \"e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6\" (UID: \"f6921ba6-c364-4314-8c37-b7a32e2b3be6\") " pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6" Oct 30 14:41:36.138392 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:36.138174 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6921ba6-c364-4314-8c37-b7a32e2b3be6-credential-socket\") pod \"e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6\" (UID: \"f6921ba6-c364-4314-8c37-b7a32e2b3be6\") " pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6" Oct 30 14:41:36.138392 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:36.138196 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6921ba6-c364-4314-8c37-b7a32e2b3be6-workload-certs\") pod \"e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6\" (UID: \"f6921ba6-c364-4314-8c37-b7a32e2b3be6\") " pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6" Oct 30 14:41:36.138392 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:36.138342 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6921ba6-c364-4314-8c37-b7a32e2b3be6-istio-data\") pod \"e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6\" (UID: \"f6921ba6-c364-4314-8c37-b7a32e2b3be6\") " pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6" Oct 30 14:41:36.138848 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:36.138524 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6921ba6-c364-4314-8c37-b7a32e2b3be6-workload-certs\") pod \"e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6\" (UID: \"f6921ba6-c364-4314-8c37-b7a32e2b3be6\") " pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6" Oct 30 14:41:36.138848 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:36.138570 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6921ba6-c364-4314-8c37-b7a32e2b3be6-istiod-ca-cert\") pod \"e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6\" (UID: \"f6921ba6-c364-4314-8c37-b7a32e2b3be6\") " pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6" Oct 30 14:41:36.138848 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:36.138573 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6921ba6-c364-4314-8c37-b7a32e2b3be6-workload-socket\") pod \"e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6\" (UID: \"f6921ba6-c364-4314-8c37-b7a32e2b3be6\") " pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6" Oct 30 14:41:36.140423 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:36.140395 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6921ba6-c364-4314-8c37-b7a32e2b3be6-istio-envoy\") pod \"e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6\" (UID: \"f6921ba6-c364-4314-8c37-b7a32e2b3be6\") " pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6" Oct 30 14:41:36.140608 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:36.140589 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6921ba6-c364-4314-8c37-b7a32e2b3be6-istio-podinfo\") pod \"e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6\" (UID: \"f6921ba6-c364-4314-8c37-b7a32e2b3be6\") " pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6" Oct 30 14:41:36.164254 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:36.164229 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6921ba6-c364-4314-8c37-b7a32e2b3be6-istio-token\") pod \"e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6\" (UID: \"f6921ba6-c364-4314-8c37-b7a32e2b3be6\") " pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6" Oct 30 14:41:36.164405 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:36.164386 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgmdb\" (UniqueName: \"kubernetes.io/projected/f6921ba6-c364-4314-8c37-b7a32e2b3be6-kube-api-access-cgmdb\") pod \"e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6\" (UID: \"f6921ba6-c364-4314-8c37-b7a32e2b3be6\") " pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6" Oct 30 14:41:36.256658 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:36.256611 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6" Oct 30 14:41:36.655795 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:36.655763 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6"] Oct 30 14:41:36.656440 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:41:36.656415 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6921ba6_c364_4314_8c37_b7a32e2b3be6.slice/crio-3a7a04fe49703bd0105f561e9746efd7368f439dda72bd0d1bad99a2ea490535 WatchSource:0}: Error finding container 3a7a04fe49703bd0105f561e9746efd7368f439dda72bd0d1bad99a2ea490535: Status 404 returned error can't find the container with id 3a7a04fe49703bd0105f561e9746efd7368f439dda72bd0d1bad99a2ea490535 Oct 30 14:41:36.659279 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:36.659249 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:41:36.659352 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:36.659314 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:41:36.659352 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:36.659346 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:41:36.946376 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:36.946243 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:41:37.546662 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:37.546226 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6" event={"ID":"f6921ba6-c364-4314-8c37-b7a32e2b3be6","Type":"ContainerStarted","Data":"b0cee75778a4ca2eada123d92a17cf36a2091f7329210fa6d3e582dca109c0c0"} Oct 30 14:41:37.546662 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:37.546307 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6" event={"ID":"f6921ba6-c364-4314-8c37-b7a32e2b3be6","Type":"ContainerStarted","Data":"3a7a04fe49703bd0105f561e9746efd7368f439dda72bd0d1bad99a2ea490535"} Oct 30 14:41:37.658631 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:37.653161 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6" podStartSLOduration=2.6531418 podStartE2EDuration="2.6531418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:41:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:41:37.648420241 +0000 UTC m=+3167.392554287" watchObservedRunningTime="2026-10-30 14:41:37.6531418 +0000 UTC m=+3167.397275840" Oct 30 14:41:38.258662 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:38.258010 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6" Oct 30 14:41:38.266675 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:38.265812 4982 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 30 14:41:38.266675 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:38.265885 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6" podUID="f6921ba6-c364-4314-8c37-b7a32e2b3be6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 30 14:41:38.363810 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:38.363776 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-460b0b-55d85f88fb-7rkhm" Oct 30 14:41:38.945979 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:38.945933 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:41:39.258016 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:39.257934 4982 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 30 14:41:39.258235 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:39.258214 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6" podUID="f6921ba6-c364-4314-8c37-b7a32e2b3be6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 30 14:41:40.257386 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:40.257344 4982 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 30 14:41:40.257808 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:40.257417 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6" podUID="f6921ba6-c364-4314-8c37-b7a32e2b3be6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 30 14:41:40.770400 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:40.770363 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:41:41.043596 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:41.043513 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b4e0be4-jd99f"] Oct 30 14:41:41.055850 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:41.055822 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b4e0be4-jd99f"] Oct 30 14:41:41.257960 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:41.257927 4982 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 30 14:41:41.258343 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:41.257982 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6" podUID="f6921ba6-c364-4314-8c37-b7a32e2b3be6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 30 14:41:42.257614 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:42.257575 4982 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 30 14:41:42.257812 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:42.257630 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6" podUID="f6921ba6-c364-4314-8c37-b7a32e2b3be6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 30 14:41:42.415843 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:42.415807 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a34762f-5688-47e3-b7d4-2cc93b375358" path="/var/lib/kubelet/pods/3a34762f-5688-47e3-b7d4-2cc93b375358/volumes" Oct 30 14:41:42.667467 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:42.667427 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:41:43.423612 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:43.423584 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6" Oct 30 14:41:43.424360 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:43.424338 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6" Oct 30 14:41:43.426955 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:43.426932 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6" Oct 30 14:41:44.636026 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:44.635991 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:41:45.755952 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:45.755921 4982 generic.go:356] "Generic (PLEG): container finished" podID="7cbc37dd-65f1-40f7-8b5a-0ab581c99f86" containerID="42d6a9828625667b4d322e348b24ce5bae9dd7647f0ad389a0c84dcc9275d6ff" exitCode=0 Oct 30 14:41:45.756438 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:45.755999 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5" event={"ID":"7cbc37dd-65f1-40f7-8b5a-0ab581c99f86","Type":"ContainerDied","Data":"42d6a9828625667b4d322e348b24ce5bae9dd7647f0ad389a0c84dcc9275d6ff"} Oct 30 14:41:45.756438 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:45.756377 4982 scope.go:122] "RemoveContainer" containerID="42d6a9828625667b4d322e348b24ce5bae9dd7647f0ad389a0c84dcc9275d6ff" Oct 30 14:41:45.758278 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:45.758241 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:41:45.758437 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:45.758332 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:41:45.758437 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:45.758384 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:41:46.356691 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:46.356651 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5" Oct 30 14:41:46.615717 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:46.612604 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:41:46.765828 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:46.765796 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5" event={"ID":"7cbc37dd-65f1-40f7-8b5a-0ab581c99f86","Type":"ContainerStarted","Data":"f7ce5a0b43bb323ea8cf13a6ff78e3d2bc40c02006661b9b0dee4a64fd62d478"} Oct 30 14:41:46.860596 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:46.860556 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:41:47.251212 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:47.251178 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5" Oct 30 14:41:47.252336 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:47.252300 4982 patch_prober.go:29] interesting pod/e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 30 14:41:47.252477 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:47.252369 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5" podUID="7cbc37dd-65f1-40f7-8b5a-0ab581c99f86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 30 14:41:48.251254 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:48.251220 4982 patch_prober.go:29] interesting pod/e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 30 14:41:48.251675 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:48.251271 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5" podUID="7cbc37dd-65f1-40f7-8b5a-0ab581c99f86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 30 14:41:48.489665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:48.486198 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:41:49.251313 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:49.251270 4982 patch_prober.go:29] interesting pod/e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 30 14:41:49.251753 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:49.251344 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5" podUID="7cbc37dd-65f1-40f7-8b5a-0ab581c99f86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 30 14:41:50.251144 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:50.251109 4982 patch_prober.go:29] interesting pod/e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 30 14:41:50.251331 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:50.251162 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5" podUID="7cbc37dd-65f1-40f7-8b5a-0ab581c99f86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 30 14:41:51.117340 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:51.117305 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:41:51.251149 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:51.251115 4982 patch_prober.go:29] interesting pod/e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 30 14:41:51.251332 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:51.251169 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5" podUID="7cbc37dd-65f1-40f7-8b5a-0ab581c99f86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 30 14:41:51.273812 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:51.273776 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:41:52.118296 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:52.118261 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-318d85fc-22c2x"] Oct 30 14:41:52.133358 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:52.133320 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-318d85fc-22c2x"] Oct 30 14:41:52.256107 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:52.256081 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5" Oct 30 14:41:52.256437 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:52.256412 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5" Oct 30 14:41:52.257079 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:52.257057 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5" Oct 30 14:41:52.434899 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:52.434830 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7a27530-b0f4-446b-91c3-cd597557e224" path="/var/lib/kubelet/pods/b7a27530-b0f4-446b-91c3-cd597557e224/volumes" Oct 30 14:41:53.518377 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:53.518346 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:41:55.487668 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:55.486882 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:41:56.882449 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:56.882414 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:41:57.062593 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:57.062555 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:41:57.296785 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:57.296694 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-33c209-5b7b659959-97sdt"] Oct 30 14:41:57.302931 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:57.302903 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c209-5b7b659959-97sdt" Oct 30 14:41:57.313031 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:57.313002 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-33c209-5b7b659959-97sdt"] Oct 30 14:41:57.332015 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:57.329791 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-33c209\"" Oct 30 14:41:57.437941 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:57.437910 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4f5d962-a1e9-4f43-85c7-e4984032b6da-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-33c209-5b7b659959-97sdt\" (UID: \"b4f5d962-a1e9-4f43-85c7-e4984032b6da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c209-5b7b659959-97sdt" Oct 30 14:41:57.438147 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:57.438037 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srknv\" (UniqueName: \"kubernetes.io/projected/b4f5d962-a1e9-4f43-85c7-e4984032b6da-kube-api-access-srknv\") pod \"payload-pre-processing-e2e-ipp-a-33c209-5b7b659959-97sdt\" (UID: \"b4f5d962-a1e9-4f43-85c7-e4984032b6da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c209-5b7b659959-97sdt" Oct 30 14:41:57.540051 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:57.540011 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4f5d962-a1e9-4f43-85c7-e4984032b6da-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-33c209-5b7b659959-97sdt\" (UID: \"b4f5d962-a1e9-4f43-85c7-e4984032b6da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c209-5b7b659959-97sdt" Oct 30 14:41:57.540233 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:57.540145 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srknv\" (UniqueName: \"kubernetes.io/projected/b4f5d962-a1e9-4f43-85c7-e4984032b6da-kube-api-access-srknv\") pod \"payload-pre-processing-e2e-ipp-a-33c209-5b7b659959-97sdt\" (UID: \"b4f5d962-a1e9-4f43-85c7-e4984032b6da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c209-5b7b659959-97sdt" Oct 30 14:41:57.540921 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:57.540894 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4f5d962-a1e9-4f43-85c7-e4984032b6da-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-33c209-5b7b659959-97sdt\" (UID: \"b4f5d962-a1e9-4f43-85c7-e4984032b6da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c209-5b7b659959-97sdt" Oct 30 14:41:57.566418 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:57.566389 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srknv\" (UniqueName: \"kubernetes.io/projected/b4f5d962-a1e9-4f43-85c7-e4984032b6da-kube-api-access-srknv\") pod \"payload-pre-processing-e2e-ipp-a-33c209-5b7b659959-97sdt\" (UID: \"b4f5d962-a1e9-4f43-85c7-e4984032b6da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c209-5b7b659959-97sdt" Oct 30 14:41:57.645619 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:57.645592 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c209-5b7b659959-97sdt" Oct 30 14:41:57.966721 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:57.966684 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-33c209-5b7b659959-97sdt"] Oct 30 14:41:57.967209 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:41:57.966977 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4f5d962_a1e9_4f43_85c7_e4984032b6da.slice/crio-0213b8126cb6e0a6f42ebe68bf3405e39d4702fb31351ec667cdc199c4bee834 WatchSource:0}: Error finding container 0213b8126cb6e0a6f42ebe68bf3405e39d4702fb31351ec667cdc199c4bee834: Status 404 returned error can't find the container with id 0213b8126cb6e0a6f42ebe68bf3405e39d4702fb31351ec667cdc199c4bee834 Oct 30 14:41:58.015839 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:41:58.015799 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:41:58.162315 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:58.162284 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:41:58.897021 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:58.896984 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c209-5b7b659959-97sdt" event={"ID":"b4f5d962-a1e9-4f43-85c7-e4984032b6da","Type":"ContainerStarted","Data":"60c09878a089e13239e42a1668f9b98145aff1d02970eb8d8a4aa0fc54ebdc4a"} Oct 30 14:41:58.897223 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:58.897045 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c209-5b7b659959-97sdt" event={"ID":"b4f5d962-a1e9-4f43-85c7-e4984032b6da","Type":"ContainerStarted","Data":"0213b8126cb6e0a6f42ebe68bf3405e39d4702fb31351ec667cdc199c4bee834"} Oct 30 14:41:58.897223 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:58.897073 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c209-5b7b659959-97sdt" Oct 30 14:41:58.946319 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:41:58.946259 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c209-5b7b659959-97sdt" podStartSLOduration=1.946240688 podStartE2EDuration="1.946240688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:41:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:41:58.944793381 +0000 UTC m=+3188.688927423" watchObservedRunningTime="2026-10-30 14:41:58.946240688 +0000 UTC m=+3188.690374729" Oct 30 14:42:00.938744 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:00.938713 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:42:02.758464 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:02.758429 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:42:04.751134 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:04.751051 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:42:05.790893 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:05.790860 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:42:05.921675 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:05.921630 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a1854b-6bd77d5846-984lk"] Oct 30 14:42:06.092067 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:06.092033 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:42:06.245208 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:06.245169 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:42:06.929020 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:06.928983 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:42:08.950946 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:08.950914 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:42:09.908900 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:09.908867 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c209-5b7b659959-97sdt" Oct 30 14:42:10.898343 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:10.898309 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:42:10.922298 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:10.922240 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-a1854b-6bd77d5846-984lk" podUID="3d7690af-86af-4404-a598-852078e649fa" containerName="payload-processing" containerID="cri-o://e209456e2bdca4070b2f92abb353d330e2f4930962be0dcafb028f0e46f4f802" gracePeriod=25 Oct 30 14:42:11.213500 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:11.213472 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a1854b-6bd77d5846-984lk" Oct 30 14:42:11.249578 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:11.249549 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d7690af-86af-4404-a598-852078e649fa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d7690af-86af-4404-a598-852078e649fa-plugins-config-volume\") pod \"3d7690af-86af-4404-a598-852078e649fa\" (UID: \"3d7690af-86af-4404-a598-852078e649fa\") " Oct 30 14:42:11.249745 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:11.249590 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d7690af-86af-4404-a598-852078e649fa-kube-api-access-4gv4t\" (UniqueName: \"kubernetes.io/projected/3d7690af-86af-4404-a598-852078e649fa-kube-api-access-4gv4t\") pod \"3d7690af-86af-4404-a598-852078e649fa\" (UID: \"3d7690af-86af-4404-a598-852078e649fa\") " Oct 30 14:42:11.249984 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:11.249958 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d7690af-86af-4404-a598-852078e649fa-plugins-config-volume" pod "3d7690af-86af-4404-a598-852078e649fa" (UID: "3d7690af-86af-4404-a598-852078e649fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:42:11.250261 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:11.250239 4982 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d7690af-86af-4404-a598-852078e649fa-plugins-config-volume\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:42:11.251667 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:11.251628 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d7690af-86af-4404-a598-852078e649fa-kube-api-access-4gv4t" pod "3d7690af-86af-4404-a598-852078e649fa" (UID: "3d7690af-86af-4404-a598-852078e649fa"). InnerVolumeSpecName "kube-api-access-4gv4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:42:11.352144 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:11.352111 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gv4t\" (UniqueName: \"kubernetes.io/projected/3d7690af-86af-4404-a598-852078e649fa-kube-api-access-4gv4t\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:42:12.019208 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:12.019173 4982 generic.go:356] "Generic (PLEG): container finished" podID="3d7690af-86af-4404-a598-852078e649fa" containerID="e209456e2bdca4070b2f92abb353d330e2f4930962be0dcafb028f0e46f4f802" exitCode=0 Oct 30 14:42:12.019841 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:12.019233 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a1854b-6bd77d5846-984lk" Oct 30 14:42:12.019987 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:12.019255 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a1854b-6bd77d5846-984lk" event={"ID":"3d7690af-86af-4404-a598-852078e649fa","Type":"ContainerDied","Data":"e209456e2bdca4070b2f92abb353d330e2f4930962be0dcafb028f0e46f4f802"} Oct 30 14:42:12.020051 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:12.019977 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a1854b-6bd77d5846-984lk" event={"ID":"3d7690af-86af-4404-a598-852078e649fa","Type":"ContainerDied","Data":"d9090736e3bafc8f783548acc3bb3aacfa001cd8b1913945fc4bc36846a63c35"} Oct 30 14:42:12.020051 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:12.020023 4982 scope.go:122] "RemoveContainer" containerID="e209456e2bdca4070b2f92abb353d330e2f4930962be0dcafb028f0e46f4f802" Oct 30 14:42:12.036840 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:12.036820 4982 scope.go:122] "RemoveContainer" containerID="e209456e2bdca4070b2f92abb353d330e2f4930962be0dcafb028f0e46f4f802" Oct 30 14:42:12.037107 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:42:12.037083 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e209456e2bdca4070b2f92abb353d330e2f4930962be0dcafb028f0e46f4f802\": container with ID starting with e209456e2bdca4070b2f92abb353d330e2f4930962be0dcafb028f0e46f4f802 not found: ID does not exist" containerID="e209456e2bdca4070b2f92abb353d330e2f4930962be0dcafb028f0e46f4f802" Oct 30 14:42:12.037188 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:12.037117 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e209456e2bdca4070b2f92abb353d330e2f4930962be0dcafb028f0e46f4f802"} err="failed to get container status \"e209456e2bdca4070b2f92abb353d330e2f4930962be0dcafb028f0e46f4f802\": rpc error: code = NotFound desc = could not find container \"e209456e2bdca4070b2f92abb353d330e2f4930962be0dcafb028f0e46f4f802\": container with ID starting with e209456e2bdca4070b2f92abb353d330e2f4930962be0dcafb028f0e46f4f802 not found: ID does not exist" Oct 30 14:42:12.080004 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:12.079979 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a1854b-6bd77d5846-984lk"] Oct 30 14:42:12.095113 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:12.095084 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a1854b-6bd77d5846-984lk"] Oct 30 14:42:12.402217 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:12.402187 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d7690af-86af-4404-a598-852078e649fa" path="/var/lib/kubelet/pods/3d7690af-86af-4404-a598-852078e649fa/volumes" Oct 30 14:42:13.099001 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:13.098961 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:42:15.164693 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:15.164654 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:42:15.398214 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:15.398179 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6176c3-749f86c99b-wfgdk"] Oct 30 14:42:15.695657 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:15.695612 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:42:15.845924 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:15.845888 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:42:16.043767 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:16.043677 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd12b3e9-qblsv"] Oct 30 14:42:16.048338 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:16.048272 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd12b3e9-qblsv"] Oct 30 14:42:16.402523 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:16.402484 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84bd46f0-7339-4f38-83cb-d728845bda61" path="/var/lib/kubelet/pods/84bd46f0-7339-4f38-83cb-d728845bda61/volumes" Oct 30 14:42:17.282327 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:17.282295 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:42:19.455712 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:19.455620 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:42:19.611039 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:19.611004 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:42:19.783701 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:19.783612 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:42:20.399441 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:20.399396 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-6176c3-749f86c99b-wfgdk" podUID="ad63625b-56af-46f1-be5c-f65fb08d97af" containerName="payload-processing" containerID="cri-o://bc0cc2e0d9f6062385c53865e955fb7f3212f8c84280b80846b520b3e5f2d742" gracePeriod=25 Oct 30 14:42:20.686820 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:20.686474 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6176c3-749f86c99b-wfgdk" Oct 30 14:42:20.783222 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:20.783181 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad63625b-56af-46f1-be5c-f65fb08d97af-kube-api-access-gp45d\" (UniqueName: \"kubernetes.io/projected/ad63625b-56af-46f1-be5c-f65fb08d97af-kube-api-access-gp45d\") pod \"ad63625b-56af-46f1-be5c-f65fb08d97af\" (UID: \"ad63625b-56af-46f1-be5c-f65fb08d97af\") " Oct 30 14:42:20.783425 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:20.783354 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad63625b-56af-46f1-be5c-f65fb08d97af-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad63625b-56af-46f1-be5c-f65fb08d97af-plugins-config-volume\") pod \"ad63625b-56af-46f1-be5c-f65fb08d97af\" (UID: \"ad63625b-56af-46f1-be5c-f65fb08d97af\") " Oct 30 14:42:20.784662 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:20.784618 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad63625b-56af-46f1-be5c-f65fb08d97af-plugins-config-volume" pod "ad63625b-56af-46f1-be5c-f65fb08d97af" (UID: "ad63625b-56af-46f1-be5c-f65fb08d97af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:42:20.789118 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:20.789087 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad63625b-56af-46f1-be5c-f65fb08d97af-kube-api-access-gp45d" pod "ad63625b-56af-46f1-be5c-f65fb08d97af" (UID: "ad63625b-56af-46f1-be5c-f65fb08d97af"). InnerVolumeSpecName "kube-api-access-gp45d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:42:20.885408 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:20.885365 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gp45d\" (UniqueName: \"kubernetes.io/projected/ad63625b-56af-46f1-be5c-f65fb08d97af-kube-api-access-gp45d\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:42:20.885408 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:20.885402 4982 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad63625b-56af-46f1-be5c-f65fb08d97af-plugins-config-volume\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:42:20.915272 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:20.915175 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:42:21.094739 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:21.094702 4982 generic.go:356] "Generic (PLEG): container finished" podID="ad63625b-56af-46f1-be5c-f65fb08d97af" containerID="bc0cc2e0d9f6062385c53865e955fb7f3212f8c84280b80846b520b3e5f2d742" exitCode=0 Oct 30 14:42:21.094943 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:21.094891 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6176c3-749f86c99b-wfgdk" event={"ID":"ad63625b-56af-46f1-be5c-f65fb08d97af","Type":"ContainerDied","Data":"bc0cc2e0d9f6062385c53865e955fb7f3212f8c84280b80846b520b3e5f2d742"} Oct 30 14:42:21.094943 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:21.094934 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6176c3-749f86c99b-wfgdk" event={"ID":"ad63625b-56af-46f1-be5c-f65fb08d97af","Type":"ContainerDied","Data":"cef274e931ee9a50160048c631610837ee3ebf324005c20a47545d0f9dd2ae52"} Oct 30 14:42:21.095088 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:21.094956 4982 scope.go:122] "RemoveContainer" containerID="bc0cc2e0d9f6062385c53865e955fb7f3212f8c84280b80846b520b3e5f2d742" Oct 30 14:42:21.095146 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:21.095088 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6176c3-749f86c99b-wfgdk" Oct 30 14:42:21.115722 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:21.115697 4982 scope.go:122] "RemoveContainer" containerID="bc0cc2e0d9f6062385c53865e955fb7f3212f8c84280b80846b520b3e5f2d742" Oct 30 14:42:21.116028 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:42:21.116004 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc0cc2e0d9f6062385c53865e955fb7f3212f8c84280b80846b520b3e5f2d742\": container with ID starting with bc0cc2e0d9f6062385c53865e955fb7f3212f8c84280b80846b520b3e5f2d742 not found: ID does not exist" containerID="bc0cc2e0d9f6062385c53865e955fb7f3212f8c84280b80846b520b3e5f2d742" Oct 30 14:42:21.116131 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:21.116036 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc0cc2e0d9f6062385c53865e955fb7f3212f8c84280b80846b520b3e5f2d742"} err="failed to get container status \"bc0cc2e0d9f6062385c53865e955fb7f3212f8c84280b80846b520b3e5f2d742\": rpc error: code = NotFound desc = could not find container \"bc0cc2e0d9f6062385c53865e955fb7f3212f8c84280b80846b520b3e5f2d742\": container with ID starting with bc0cc2e0d9f6062385c53865e955fb7f3212f8c84280b80846b520b3e5f2d742 not found: ID does not exist" Oct 30 14:42:21.163138 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:21.163113 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6176c3-749f86c99b-wfgdk"] Oct 30 14:42:21.177720 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:21.177667 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6176c3-749f86c99b-wfgdk"] Oct 30 14:42:21.334173 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:21.334138 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp"] Oct 30 14:42:21.335449 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:21.335420 4982 cpu_manager.go:414] "containerMap: removing container" podUID="3d7690af-86af-4404-a598-852078e649fa" containerName="payload-processing" Oct 30 14:42:21.335599 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:21.335586 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d7690af-86af-4404-a598-852078e649fa" containerName="payload-processing" Oct 30 14:42:21.335719 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:21.335707 4982 cpu_manager.go:414] "containerMap: removing container" podUID="ad63625b-56af-46f1-be5c-f65fb08d97af" containerName="payload-processing" Oct 30 14:42:21.335808 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:21.335798 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad63625b-56af-46f1-be5c-f65fb08d97af" containerName="payload-processing" Oct 30 14:42:21.336104 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:21.336090 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d7690af-86af-4404-a598-852078e649fa" containerName="payload-processing" Oct 30 14:42:21.337071 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:21.337054 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad63625b-56af-46f1-be5c-f65fb08d97af" containerName="payload-processing" Oct 30 14:42:21.342932 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:21.342914 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp" Oct 30 14:42:21.361460 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:21.361437 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-d05e21-openshift-default-dockercfg-6p687\"" Oct 30 14:42:21.365329 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:21.365299 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp"] Oct 30 14:42:21.424689 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:21.424657 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:42:21.508716 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:21.503671 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5d8a300-e5ec-462a-95b3-d6f1e220dc42-istio-podinfo\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp\" (UID: \"b5d8a300-e5ec-462a-95b3-d6f1e220dc42\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp" Oct 30 14:42:21.508716 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:21.503745 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5d8a300-e5ec-462a-95b3-d6f1e220dc42-istio-token\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp\" (UID: \"b5d8a300-e5ec-462a-95b3-d6f1e220dc42\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp" Oct 30 14:42:21.508716 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:21.503801 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d8a300-e5ec-462a-95b3-d6f1e220dc42-credential-socket\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp\" (UID: \"b5d8a300-e5ec-462a-95b3-d6f1e220dc42\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp" Oct 30 14:42:21.508716 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:21.503852 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5d8a300-e5ec-462a-95b3-d6f1e220dc42-workload-certs\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp\" (UID: \"b5d8a300-e5ec-462a-95b3-d6f1e220dc42\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp" Oct 30 14:42:21.508716 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:21.503890 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lg9p\" (UniqueName: \"kubernetes.io/projected/b5d8a300-e5ec-462a-95b3-d6f1e220dc42-kube-api-access-8lg9p\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp\" (UID: \"b5d8a300-e5ec-462a-95b3-d6f1e220dc42\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp" Oct 30 14:42:21.508716 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:21.503955 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d8a300-e5ec-462a-95b3-d6f1e220dc42-workload-socket\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp\" (UID: \"b5d8a300-e5ec-462a-95b3-d6f1e220dc42\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp" Oct 30 14:42:21.508716 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:21.504071 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5d8a300-e5ec-462a-95b3-d6f1e220dc42-istio-envoy\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp\" (UID: \"b5d8a300-e5ec-462a-95b3-d6f1e220dc42\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp" Oct 30 14:42:21.508716 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:21.504217 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5d8a300-e5ec-462a-95b3-d6f1e220dc42-istiod-ca-cert\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp\" (UID: \"b5d8a300-e5ec-462a-95b3-d6f1e220dc42\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp" Oct 30 14:42:21.508716 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:21.504249 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5d8a300-e5ec-462a-95b3-d6f1e220dc42-istio-data\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp\" (UID: \"b5d8a300-e5ec-462a-95b3-d6f1e220dc42\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp" Oct 30 14:42:21.624769 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:21.624527 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:42:21.659680 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:21.626366 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5d8a300-e5ec-462a-95b3-d6f1e220dc42-istio-envoy\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp\" (UID: \"b5d8a300-e5ec-462a-95b3-d6f1e220dc42\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp" Oct 30 14:42:21.660306 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:21.660267 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5d8a300-e5ec-462a-95b3-d6f1e220dc42-istiod-ca-cert\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp\" (UID: \"b5d8a300-e5ec-462a-95b3-d6f1e220dc42\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp" Oct 30 14:42:21.660539 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:21.660511 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5d8a300-e5ec-462a-95b3-d6f1e220dc42-istio-data\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp\" (UID: \"b5d8a300-e5ec-462a-95b3-d6f1e220dc42\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp" Oct 30 14:42:21.668940 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:21.668909 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5d8a300-e5ec-462a-95b3-d6f1e220dc42-istio-podinfo\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp\" (UID: \"b5d8a300-e5ec-462a-95b3-d6f1e220dc42\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp" Oct 30 14:42:21.669145 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:21.669127 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5d8a300-e5ec-462a-95b3-d6f1e220dc42-istio-token\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp\" (UID: \"b5d8a300-e5ec-462a-95b3-d6f1e220dc42\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp" Oct 30 14:42:21.675677 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:21.669269 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d8a300-e5ec-462a-95b3-d6f1e220dc42-credential-socket\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp\" (UID: \"b5d8a300-e5ec-462a-95b3-d6f1e220dc42\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp" Oct 30 14:42:21.675677 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:21.673753 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5d8a300-e5ec-462a-95b3-d6f1e220dc42-workload-certs\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp\" (UID: \"b5d8a300-e5ec-462a-95b3-d6f1e220dc42\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp" Oct 30 14:42:21.675677 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:21.673803 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lg9p\" (UniqueName: \"kubernetes.io/projected/b5d8a300-e5ec-462a-95b3-d6f1e220dc42-kube-api-access-8lg9p\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp\" (UID: \"b5d8a300-e5ec-462a-95b3-d6f1e220dc42\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp" Oct 30 14:42:21.675677 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:21.673863 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d8a300-e5ec-462a-95b3-d6f1e220dc42-workload-socket\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp\" (UID: \"b5d8a300-e5ec-462a-95b3-d6f1e220dc42\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp" Oct 30 14:42:21.675677 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:21.670962 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d8a300-e5ec-462a-95b3-d6f1e220dc42-credential-socket\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp\" (UID: \"b5d8a300-e5ec-462a-95b3-d6f1e220dc42\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp" Oct 30 14:42:21.675677 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:21.663993 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5d8a300-e5ec-462a-95b3-d6f1e220dc42-istio-data\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp\" (UID: \"b5d8a300-e5ec-462a-95b3-d6f1e220dc42\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp" Oct 30 14:42:21.675677 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:21.674287 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5d8a300-e5ec-462a-95b3-d6f1e220dc42-workload-certs\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp\" (UID: \"b5d8a300-e5ec-462a-95b3-d6f1e220dc42\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp" Oct 30 14:42:21.675677 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:21.663745 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5d8a300-e5ec-462a-95b3-d6f1e220dc42-istiod-ca-cert\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp\" (UID: \"b5d8a300-e5ec-462a-95b3-d6f1e220dc42\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp" Oct 30 14:42:21.675677 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:21.664577 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5d8a300-e5ec-462a-95b3-d6f1e220dc42-istio-envoy\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp\" (UID: \"b5d8a300-e5ec-462a-95b3-d6f1e220dc42\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp" Oct 30 14:42:21.675677 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:21.674834 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d8a300-e5ec-462a-95b3-d6f1e220dc42-workload-socket\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp\" (UID: \"b5d8a300-e5ec-462a-95b3-d6f1e220dc42\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp" Oct 30 14:42:21.701608 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:21.692300 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5d8a300-e5ec-462a-95b3-d6f1e220dc42-istio-podinfo\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp\" (UID: \"b5d8a300-e5ec-462a-95b3-d6f1e220dc42\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp" Oct 30 14:42:21.712558 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:21.712508 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lg9p\" (UniqueName: \"kubernetes.io/projected/b5d8a300-e5ec-462a-95b3-d6f1e220dc42-kube-api-access-8lg9p\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp\" (UID: \"b5d8a300-e5ec-462a-95b3-d6f1e220dc42\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp" Oct 30 14:42:21.713272 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:21.713228 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5d8a300-e5ec-462a-95b3-d6f1e220dc42-istio-token\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp\" (UID: \"b5d8a300-e5ec-462a-95b3-d6f1e220dc42\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp" Oct 30 14:42:21.788006 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:21.787903 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:42:21.961700 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:21.961662 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp" Oct 30 14:42:22.300140 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:42:22.300106 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5d8a300_e5ec_462a_95b3_d6f1e220dc42.slice/crio-34836d38fe63faa50ac1150b32ce716caf168f02fe2d1ae0df7344660f2be0ab WatchSource:0}: Error finding container 34836d38fe63faa50ac1150b32ce716caf168f02fe2d1ae0df7344660f2be0ab: Status 404 returned error can't find the container with id 34836d38fe63faa50ac1150b32ce716caf168f02fe2d1ae0df7344660f2be0ab Oct 30 14:42:22.300389 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:22.300360 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp"] Oct 30 14:42:22.303029 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:22.302994 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:42:22.303123 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:22.303059 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:42:22.303123 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:22.303088 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:42:22.405528 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:22.405496 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad63625b-56af-46f1-be5c-f65fb08d97af" path="/var/lib/kubelet/pods/ad63625b-56af-46f1-be5c-f65fb08d97af/volumes" Oct 30 14:42:23.124071 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:23.124036 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp" event={"ID":"b5d8a300-e5ec-462a-95b3-d6f1e220dc42","Type":"ContainerStarted","Data":"b8ce2f3cc2284ca2be475a1f1e9b3f16132f1c5a2534b8f9dc61223fc01176ad"} Oct 30 14:42:23.124488 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:23.124099 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp" event={"ID":"b5d8a300-e5ec-462a-95b3-d6f1e220dc42","Type":"ContainerStarted","Data":"34836d38fe63faa50ac1150b32ce716caf168f02fe2d1ae0df7344660f2be0ab"} Oct 30 14:42:23.187371 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:23.187319 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp" podStartSLOduration=2.187302229 podStartE2EDuration="2.187302229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:42:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:42:23.185295558 +0000 UTC m=+3212.929429590" watchObservedRunningTime="2026-10-30 14:42:23.187302229 +0000 UTC m=+3212.931436270" Oct 30 14:42:23.671443 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:23.671396 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:42:23.961882 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:23.961792 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp" Oct 30 14:42:23.967913 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:23.967884 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp" Oct 30 14:42:24.132100 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:24.132074 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp" Oct 30 14:42:24.133103 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:24.133084 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp" Oct 30 14:42:25.631137 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:25.631081 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:42:26.038774 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:26.038697 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c152ab3d-wpzgh"] Oct 30 14:42:26.051436 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:26.051406 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c152ab3d-wpzgh"] Oct 30 14:42:26.402131 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:26.402101 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3225123b-b2e3-4ba0-a1b7-f60d2509cd97" path="/var/lib/kubelet/pods/3225123b-b2e3-4ba0-a1b7-f60d2509cd97/volumes" Oct 30 14:42:27.644757 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:27.644728 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:42:28.012816 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:42:28.012714 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:42:30.084570 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:30.084531 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:42:32.236529 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:32.236492 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:42:34.110003 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:34.109960 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:42:35.753405 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:35.753367 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:42:35.898259 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:35.898229 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:42:36.269767 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:36.269733 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:42:37.446282 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:37.446245 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:42:37.596203 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:37.596169 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:42:37.600114 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:37.600070 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d05e21-5767b6546c-hmlcp"] Oct 30 14:42:37.607287 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:37.607262 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d05e21-5767b6546c-hmlcp" Oct 30 14:42:37.623957 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:37.623930 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d05e21-5767b6546c-hmlcp"] Oct 30 14:42:37.631087 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:37.631062 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-d05e21-dockercfg-rs44w\"" Oct 30 14:42:37.657816 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:37.657780 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-d05e21\"" Oct 30 14:42:37.731234 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:37.731131 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs28q\" (UniqueName: \"kubernetes.io/projected/1a79c636-bf50-4835-9896-916723340900-kube-api-access-bs28q\") pod \"payload-pre-processing-e2e-ipp-b-d05e21-5767b6546c-hmlcp\" (UID: \"1a79c636-bf50-4835-9896-916723340900\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d05e21-5767b6546c-hmlcp" Oct 30 14:42:37.732103 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:37.732076 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a79c636-bf50-4835-9896-916723340900-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d05e21-5767b6546c-hmlcp\" (UID: \"1a79c636-bf50-4835-9896-916723340900\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d05e21-5767b6546c-hmlcp" Oct 30 14:42:37.853695 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:37.850806 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a79c636-bf50-4835-9896-916723340900-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d05e21-5767b6546c-hmlcp\" (UID: \"1a79c636-bf50-4835-9896-916723340900\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d05e21-5767b6546c-hmlcp" Oct 30 14:42:37.853695 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:37.850929 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs28q\" (UniqueName: \"kubernetes.io/projected/1a79c636-bf50-4835-9896-916723340900-kube-api-access-bs28q\") pod \"payload-pre-processing-e2e-ipp-b-d05e21-5767b6546c-hmlcp\" (UID: \"1a79c636-bf50-4835-9896-916723340900\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d05e21-5767b6546c-hmlcp" Oct 30 14:42:37.853695 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:37.852107 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a79c636-bf50-4835-9896-916723340900-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d05e21-5767b6546c-hmlcp\" (UID: \"1a79c636-bf50-4835-9896-916723340900\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d05e21-5767b6546c-hmlcp" Oct 30 14:42:37.878430 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:37.878402 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs28q\" (UniqueName: \"kubernetes.io/projected/1a79c636-bf50-4835-9896-916723340900-kube-api-access-bs28q\") pod \"payload-pre-processing-e2e-ipp-b-d05e21-5767b6546c-hmlcp\" (UID: \"1a79c636-bf50-4835-9896-916723340900\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d05e21-5767b6546c-hmlcp" Oct 30 14:42:37.925745 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:37.925715 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d05e21-5767b6546c-hmlcp" Oct 30 14:42:38.039095 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:38.039061 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6c013df-vvrs6"] Oct 30 14:42:38.045209 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:38.045177 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6c013df-vvrs6"] Oct 30 14:42:38.235748 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:38.235720 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d05e21-5767b6546c-hmlcp"] Oct 30 14:42:38.236180 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:42:38.236156 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a79c636_bf50_4835_9896_916723340900.slice/crio-d02bd3f2d447d664546ed4aef35822cb719890b22731676743da5b9f0bcaa989 WatchSource:0}: Error finding container d02bd3f2d447d664546ed4aef35822cb719890b22731676743da5b9f0bcaa989: Status 404 returned error can't find the container with id d02bd3f2d447d664546ed4aef35822cb719890b22731676743da5b9f0bcaa989 Oct 30 14:42:38.246994 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:38.246961 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d05e21-5767b6546c-hmlcp" event={"ID":"1a79c636-bf50-4835-9896-916723340900","Type":"ContainerStarted","Data":"d02bd3f2d447d664546ed4aef35822cb719890b22731676743da5b9f0bcaa989"} Oct 30 14:42:38.330420 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:38.330397 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:42:38.405126 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:38.405087 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f307466f-7382-4469-b6cd-b19e87a6fa81" path="/var/lib/kubelet/pods/f307466f-7382-4469-b6cd-b19e87a6fa81/volumes" Oct 30 14:42:39.255539 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:39.255497 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d05e21-5767b6546c-hmlcp" event={"ID":"1a79c636-bf50-4835-9896-916723340900","Type":"ContainerStarted","Data":"c23c1d0a6b2b6d45c5eb528d449245a7ce5a5802168c9724fd4df9eeedb4170e"} Oct 30 14:42:39.256057 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:39.255566 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d05e21-5767b6546c-hmlcp" Oct 30 14:42:39.301861 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:39.301814 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d05e21-5767b6546c-hmlcp" podStartSLOduration=2.301801908 podStartE2EDuration="2.301801908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:42:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:42:39.29911235 +0000 UTC m=+3229.043246392" watchObservedRunningTime="2026-10-30 14:42:39.301801908 +0000 UTC m=+3229.045935950" Oct 30 14:42:39.818536 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:39.818500 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:42:40.334109 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:40.334079 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:42:42.264665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:42.264341 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:42:44.371388 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:44.371354 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:42:45.027008 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:45.026958 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:42:45.475916 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:45.475883 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c7c37b-7cf9g"] Oct 30 14:42:45.480964 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:45.480941 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c7c37b-7cf9g" Oct 30 14:42:45.502535 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:45.502508 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c7c37b-7cf9g"] Oct 30 14:42:45.511311 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:45.511285 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:42:45.511456 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:45.511407 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:42:45.631353 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:45.631312 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4005a383-48bb-48dd-84c0-70572307aa61-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-c7c37b-7cf9g\" (UID: \"4005a383-48bb-48dd-84c0-70572307aa61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c7c37b-7cf9g" Oct 30 14:42:45.631535 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:45.631456 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4005a383-48bb-48dd-84c0-70572307aa61-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-c7c37b-7cf9g\" (UID: \"4005a383-48bb-48dd-84c0-70572307aa61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c7c37b-7cf9g" Oct 30 14:42:45.631689 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:45.631666 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rh4k\" (UniqueName: \"kubernetes.io/projected/4005a383-48bb-48dd-84c0-70572307aa61-kube-api-access-5rh4k\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-c7c37b-7cf9g\" (UID: \"4005a383-48bb-48dd-84c0-70572307aa61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c7c37b-7cf9g" Oct 30 14:42:45.733185 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:45.733101 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rh4k\" (UniqueName: \"kubernetes.io/projected/4005a383-48bb-48dd-84c0-70572307aa61-kube-api-access-5rh4k\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-c7c37b-7cf9g\" (UID: \"4005a383-48bb-48dd-84c0-70572307aa61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c7c37b-7cf9g" Oct 30 14:42:45.733350 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:45.733220 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4005a383-48bb-48dd-84c0-70572307aa61-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-c7c37b-7cf9g\" (UID: \"4005a383-48bb-48dd-84c0-70572307aa61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c7c37b-7cf9g" Oct 30 14:42:45.733350 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:45.733273 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4005a383-48bb-48dd-84c0-70572307aa61-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-c7c37b-7cf9g\" (UID: \"4005a383-48bb-48dd-84c0-70572307aa61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c7c37b-7cf9g" Oct 30 14:42:45.733894 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:45.733871 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4005a383-48bb-48dd-84c0-70572307aa61-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-c7c37b-7cf9g\" (UID: \"4005a383-48bb-48dd-84c0-70572307aa61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c7c37b-7cf9g" Oct 30 14:42:45.735450 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:45.735427 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4005a383-48bb-48dd-84c0-70572307aa61-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-c7c37b-7cf9g\" (UID: \"4005a383-48bb-48dd-84c0-70572307aa61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c7c37b-7cf9g" Oct 30 14:42:45.758788 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:45.758764 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rh4k\" (UniqueName: \"kubernetes.io/projected/4005a383-48bb-48dd-84c0-70572307aa61-kube-api-access-5rh4k\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-c7c37b-7cf9g\" (UID: \"4005a383-48bb-48dd-84c0-70572307aa61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c7c37b-7cf9g" Oct 30 14:42:45.797086 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:45.797065 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c7c37b-7cf9g" Oct 30 14:42:46.115171 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:46.115137 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c7c37b-7cf9g"] Oct 30 14:42:46.116833 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:42:46.116798 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4005a383_48bb_48dd_84c0_70572307aa61.slice/crio-c4e7e5697b5fdf5ecc0a3b1098a46b89112ecdcd37834d83dc462d629113481b WatchSource:0}: Error finding container c4e7e5697b5fdf5ecc0a3b1098a46b89112ecdcd37834d83dc462d629113481b: Status 404 returned error can't find the container with id c4e7e5697b5fdf5ecc0a3b1098a46b89112ecdcd37834d83dc462d629113481b Oct 30 14:42:46.319719 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:46.319680 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c7c37b-7cf9g" event={"ID":"4005a383-48bb-48dd-84c0-70572307aa61","Type":"ContainerStarted","Data":"c4e7e5697b5fdf5ecc0a3b1098a46b89112ecdcd37834d83dc462d629113481b"} Oct 30 14:42:46.428786 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:46.428745 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:42:47.000788 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:47.000757 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:42:47.331826 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:47.331792 4982 generic.go:356] "Generic (PLEG): container finished" podID="4005a383-48bb-48dd-84c0-70572307aa61" containerID="ac9aeeb6af766fce81e3ca832637fb090e41d7229a4f2160126386801b259256" exitCode=0 Oct 30 14:42:47.331992 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:47.331873 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c7c37b-7cf9g" event={"ID":"4005a383-48bb-48dd-84c0-70572307aa61","Type":"ContainerDied","Data":"ac9aeeb6af766fce81e3ca832637fb090e41d7229a4f2160126386801b259256"} Oct 30 14:42:48.066075 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:48.066040 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf36da92-5cf7n"] Oct 30 14:42:48.078918 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:48.078893 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf36da92-5cf7n"] Oct 30 14:42:48.405270 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:48.404781 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2214368-f584-4b7f-b208-ea49e90e7829" path="/var/lib/kubelet/pods/d2214368-f584-4b7f-b208-ea49e90e7829/volumes" Oct 30 14:42:48.500137 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:48.500109 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c7c37b-7cf9g" Oct 30 14:42:48.568114 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:48.568080 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:42:48.669416 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:48.669335 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4005a383-48bb-48dd-84c0-70572307aa61-kube-api-access-5rh4k\" (UniqueName: \"kubernetes.io/projected/4005a383-48bb-48dd-84c0-70572307aa61-kube-api-access-5rh4k\") pod \"4005a383-48bb-48dd-84c0-70572307aa61\" (UID: \"4005a383-48bb-48dd-84c0-70572307aa61\") " Oct 30 14:42:48.669416 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:48.669413 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4005a383-48bb-48dd-84c0-70572307aa61-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4005a383-48bb-48dd-84c0-70572307aa61-maas-api-openshift-ca\") pod \"4005a383-48bb-48dd-84c0-70572307aa61\" (UID: \"4005a383-48bb-48dd-84c0-70572307aa61\") " Oct 30 14:42:48.669632 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:48.669448 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4005a383-48bb-48dd-84c0-70572307aa61-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4005a383-48bb-48dd-84c0-70572307aa61-maas-api-xks-ca\") pod \"4005a383-48bb-48dd-84c0-70572307aa61\" (UID: \"4005a383-48bb-48dd-84c0-70572307aa61\") " Oct 30 14:42:48.669827 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:48.669804 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4005a383-48bb-48dd-84c0-70572307aa61-maas-api-openshift-ca" pod "4005a383-48bb-48dd-84c0-70572307aa61" (UID: "4005a383-48bb-48dd-84c0-70572307aa61"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:42:48.670163 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:48.670144 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4005a383-48bb-48dd-84c0-70572307aa61-maas-api-openshift-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:42:48.671583 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:48.671555 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4005a383-48bb-48dd-84c0-70572307aa61-maas-api-xks-ca" pod "4005a383-48bb-48dd-84c0-70572307aa61" (UID: "4005a383-48bb-48dd-84c0-70572307aa61"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:42:48.671714 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:48.671611 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4005a383-48bb-48dd-84c0-70572307aa61-kube-api-access-5rh4k" pod "4005a383-48bb-48dd-84c0-70572307aa61" (UID: "4005a383-48bb-48dd-84c0-70572307aa61"). InnerVolumeSpecName "kube-api-access-5rh4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:42:48.771994 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:48.771963 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rh4k\" (UniqueName: \"kubernetes.io/projected/4005a383-48bb-48dd-84c0-70572307aa61-kube-api-access-5rh4k\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:42:48.771994 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:48.771991 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4005a383-48bb-48dd-84c0-70572307aa61-maas-api-xks-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:42:49.347451 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:49.347418 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c7c37b-7cf9g" Oct 30 14:42:49.347909 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:49.347452 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c7c37b-7cf9g" event={"ID":"4005a383-48bb-48dd-84c0-70572307aa61","Type":"ContainerDied","Data":"c4e7e5697b5fdf5ecc0a3b1098a46b89112ecdcd37834d83dc462d629113481b"} Oct 30 14:42:49.347909 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:49.347512 4982 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4e7e5697b5fdf5ecc0a3b1098a46b89112ecdcd37834d83dc462d629113481b" Oct 30 14:42:50.266760 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:50.266727 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d05e21-5767b6546c-hmlcp" Oct 30 14:42:50.646026 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:50.645992 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:42:52.225826 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:52.225790 4982 scope.go:122] "RemoveContainer" containerID="16998eb0ea67daf7a464871f494ed3dffc64f50884d77c06e41f350380c3583c" Oct 30 14:42:52.237026 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:52.237001 4982 scope.go:122] "RemoveContainer" containerID="b70bd0b1db12584542b28fd4c9ff8153ae0aef5652e98f2d66f5bcd4306c44aa" Oct 30 14:42:52.248953 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:52.248933 4982 scope.go:122] "RemoveContainer" containerID="feb1ac0485a55d78d89a6c7629dd62d0880d12b317623b049a412d447f808928" Oct 30 14:42:52.260107 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:52.260008 4982 scope.go:122] "RemoveContainer" containerID="d9cdcdb199292e201af22223558e46bea6481d9d29d1ca5f54b174365b8f9645" Oct 30 14:42:52.363794 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:52.363753 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:42:52.469874 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:52.467893 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d3a829-64f6b74c79-6wf64"] Oct 30 14:42:52.469874 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:52.468095 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d3a829-64f6b74c79-6wf64" podUID="baeb0f86-9313-4cb6-a075-8d3cdd5e9e53" containerName="payload-pre-processing" containerID="cri-o://c19558b4446c8e2437d863eae5610e60b2c444ba16663583fa24c8c5550a47cc" gracePeriod=30 Oct 30 14:42:52.707501 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:52.707386 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:42:52.779875 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:52.779848 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d3a829-64f6b74c79-6wf64" Oct 30 14:42:52.848828 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:52.843932 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/baeb0f86-9313-4cb6-a075-8d3cdd5e9e53-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baeb0f86-9313-4cb6-a075-8d3cdd5e9e53-plugins-config-volume\") pod \"baeb0f86-9313-4cb6-a075-8d3cdd5e9e53\" (UID: \"baeb0f86-9313-4cb6-a075-8d3cdd5e9e53\") " Oct 30 14:42:52.848828 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:52.844016 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baeb0f86-9313-4cb6-a075-8d3cdd5e9e53-kube-api-access-ll9hs\" (UniqueName: \"kubernetes.io/projected/baeb0f86-9313-4cb6-a075-8d3cdd5e9e53-kube-api-access-ll9hs\") pod \"baeb0f86-9313-4cb6-a075-8d3cdd5e9e53\" (UID: \"baeb0f86-9313-4cb6-a075-8d3cdd5e9e53\") " Oct 30 14:42:52.848828 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:52.845973 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baeb0f86-9313-4cb6-a075-8d3cdd5e9e53-plugins-config-volume" pod "baeb0f86-9313-4cb6-a075-8d3cdd5e9e53" (UID: "baeb0f86-9313-4cb6-a075-8d3cdd5e9e53"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:42:52.851036 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:52.849507 4982 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baeb0f86-9313-4cb6-a075-8d3cdd5e9e53-plugins-config-volume\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:42:52.855596 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:52.855571 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baeb0f86-9313-4cb6-a075-8d3cdd5e9e53-kube-api-access-ll9hs" pod "baeb0f86-9313-4cb6-a075-8d3cdd5e9e53" (UID: "baeb0f86-9313-4cb6-a075-8d3cdd5e9e53"). InnerVolumeSpecName "kube-api-access-ll9hs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:42:52.922666 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:52.914626 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:42:52.957703 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:52.954571 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ll9hs\" (UniqueName: \"kubernetes.io/projected/baeb0f86-9313-4cb6-a075-8d3cdd5e9e53-kube-api-access-ll9hs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:42:53.064538 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:53.064496 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:42:53.231197 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:53.230452 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:42:53.386990 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:53.386903 4982 generic.go:356] "Generic (PLEG): container finished" podID="baeb0f86-9313-4cb6-a075-8d3cdd5e9e53" containerID="c19558b4446c8e2437d863eae5610e60b2c444ba16663583fa24c8c5550a47cc" exitCode=0 Oct 30 14:42:53.386990 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:53.386960 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d3a829-64f6b74c79-6wf64" Oct 30 14:42:53.387237 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:53.386987 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d3a829-64f6b74c79-6wf64" event={"ID":"baeb0f86-9313-4cb6-a075-8d3cdd5e9e53","Type":"ContainerDied","Data":"c19558b4446c8e2437d863eae5610e60b2c444ba16663583fa24c8c5550a47cc"} Oct 30 14:42:53.387237 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:53.387080 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d3a829-64f6b74c79-6wf64" event={"ID":"baeb0f86-9313-4cb6-a075-8d3cdd5e9e53","Type":"ContainerDied","Data":"1876bd1a4db40e11e9c2e87ae3b5df48981798bdc825b8cd9df8ce7dc1224b12"} Oct 30 14:42:53.387237 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:53.387113 4982 scope.go:122] "RemoveContainer" containerID="c19558b4446c8e2437d863eae5610e60b2c444ba16663583fa24c8c5550a47cc" Oct 30 14:42:53.398461 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:53.398444 4982 scope.go:122] "RemoveContainer" containerID="c19558b4446c8e2437d863eae5610e60b2c444ba16663583fa24c8c5550a47cc" Oct 30 14:42:53.398764 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:42:53.398743 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c19558b4446c8e2437d863eae5610e60b2c444ba16663583fa24c8c5550a47cc\": container with ID starting with c19558b4446c8e2437d863eae5610e60b2c444ba16663583fa24c8c5550a47cc not found: ID does not exist" containerID="c19558b4446c8e2437d863eae5610e60b2c444ba16663583fa24c8c5550a47cc" Oct 30 14:42:53.398853 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:53.398772 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c19558b4446c8e2437d863eae5610e60b2c444ba16663583fa24c8c5550a47cc"} err="failed to get container status \"c19558b4446c8e2437d863eae5610e60b2c444ba16663583fa24c8c5550a47cc\": rpc error: code = NotFound desc = could not find container \"c19558b4446c8e2437d863eae5610e60b2c444ba16663583fa24c8c5550a47cc\": container with ID starting with c19558b4446c8e2437d863eae5610e60b2c444ba16663583fa24c8c5550a47cc not found: ID does not exist" Oct 30 14:42:53.452790 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:53.452761 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d3a829-64f6b74c79-6wf64"] Oct 30 14:42:53.465917 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:53.465890 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d3a829-64f6b74c79-6wf64"] Oct 30 14:42:53.744349 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:53.744253 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:42:54.401167 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:54.401053 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="baeb0f86-9313-4cb6-a075-8d3cdd5e9e53" path="/var/lib/kubelet/pods/baeb0f86-9313-4cb6-a075-8d3cdd5e9e53/volumes" Oct 30 14:42:55.462524 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:55.462488 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:42:56.340755 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:56.340721 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-c7c37b-67bb95b964-cqz6m"] Oct 30 14:42:56.453178 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:56.453148 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c7c37b-6b9c5c5999-kdfrr"] Oct 30 14:42:56.576459 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:56.576417 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:42:56.719088 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:56.718989 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:42:57.671155 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:57.671118 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:42:58.013345 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:42:58.013258 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:42:59.394407 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:59.394373 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2927fd-844ddcc4bd-wbrs7"] Oct 30 14:42:59.447514 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:59.447486 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2927fd-55db459cf-lqzdk"] Oct 30 14:42:59.583075 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:59.583037 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:42:59.736132 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:59.736038 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:42:59.892034 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:42:59.891999 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:00.165475 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:00.165440 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:01.341739 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:01.341698 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c7c37b-67bb95b964-cqz6m" podUID="fb2ea041-440f-42d9-8454-6ff8bc952a5e" containerName="payload-processing" containerID="cri-o://65cb85cfc52abff3dbf77c62f5b14356e5fb0146fc827d8bb7dac8349f6f0d08" gracePeriod=25 Oct 30 14:43:01.458462 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:01.458408 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c7c37b-6b9c5c5999-kdfrr" podUID="d4cdb56a-9a5e-4edc-8cec-ff32aa54de9d" containerName="payload-pre-processing" containerID="cri-o://d6606d65c58e04ee138c32d0eda581cf0d51ee5425d375ba18b84376cde310bb" gracePeriod=25 Oct 30 14:43:01.520274 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:01.520240 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:01.683315 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:01.683224 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:01.940696 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:01.940572 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:02.764840 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:02.764776 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:02.807929 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:02.807887 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c7c37b-6b9c5c5999-kdfrr" podUID="d4cdb56a-9a5e-4edc-8cec-ff32aa54de9d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:43:02.808484 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:02.808446 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c7c37b-67bb95b964-cqz6m" podUID="fb2ea041-440f-42d9-8454-6ff8bc952a5e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:43:04.395313 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:04.395274 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-2927fd-844ddcc4bd-wbrs7" podUID="ac0599bf-47df-45a3-96a3-55c24b5a2afd" containerName="payload-processing" containerID="cri-o://a062f00c57ee37f4201c3e5880bf0acdb49de6b18714fc39f8c4943372efe3ac" gracePeriod=25 Oct 30 14:43:04.448062 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:04.448028 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2927fd-55db459cf-lqzdk" podUID="920bf174-09b0-4966-b029-4e200373359b" containerName="payload-pre-processing" containerID="cri-o://3036a0f174d32eab2eddbdf5612262798ac7fd61ebd51ca95b8736d1ae0909d5" gracePeriod=25 Oct 30 14:43:04.528925 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:04.528899 4982 generic.go:356] "Generic (PLEG): container finished" podID="ac0599bf-47df-45a3-96a3-55c24b5a2afd" containerID="a062f00c57ee37f4201c3e5880bf0acdb49de6b18714fc39f8c4943372efe3ac" exitCode=0 Oct 30 14:43:04.529043 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:04.528970 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2927fd-844ddcc4bd-wbrs7" event={"ID":"ac0599bf-47df-45a3-96a3-55c24b5a2afd","Type":"ContainerDied","Data":"a062f00c57ee37f4201c3e5880bf0acdb49de6b18714fc39f8c4943372efe3ac"} Oct 30 14:43:04.748384 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:04.748351 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2927fd-844ddcc4bd-wbrs7" Oct 30 14:43:04.751977 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:04.751930 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:04.807287 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:04.807262 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2927fd-55db459cf-lqzdk" Oct 30 14:43:04.848227 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:04.848194 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac0599bf-47df-45a3-96a3-55c24b5a2afd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac0599bf-47df-45a3-96a3-55c24b5a2afd-plugins-config-volume\") pod \"ac0599bf-47df-45a3-96a3-55c24b5a2afd\" (UID: \"ac0599bf-47df-45a3-96a3-55c24b5a2afd\") " Oct 30 14:43:04.848408 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:04.848283 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac0599bf-47df-45a3-96a3-55c24b5a2afd-kube-api-access-zzczq\" (UniqueName: \"kubernetes.io/projected/ac0599bf-47df-45a3-96a3-55c24b5a2afd-kube-api-access-zzczq\") pod \"ac0599bf-47df-45a3-96a3-55c24b5a2afd\" (UID: \"ac0599bf-47df-45a3-96a3-55c24b5a2afd\") " Oct 30 14:43:04.848632 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:04.848610 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac0599bf-47df-45a3-96a3-55c24b5a2afd-plugins-config-volume" pod "ac0599bf-47df-45a3-96a3-55c24b5a2afd" (UID: "ac0599bf-47df-45a3-96a3-55c24b5a2afd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:43:04.848802 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:04.848786 4982 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac0599bf-47df-45a3-96a3-55c24b5a2afd-plugins-config-volume\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:43:04.850501 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:04.850468 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac0599bf-47df-45a3-96a3-55c24b5a2afd-kube-api-access-zzczq" pod "ac0599bf-47df-45a3-96a3-55c24b5a2afd" (UID: "ac0599bf-47df-45a3-96a3-55c24b5a2afd"). InnerVolumeSpecName "kube-api-access-zzczq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:43:04.950181 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:04.950140 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/920bf174-09b0-4966-b029-4e200373359b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/920bf174-09b0-4966-b029-4e200373359b-plugins-config-volume\") pod \"920bf174-09b0-4966-b029-4e200373359b\" (UID: \"920bf174-09b0-4966-b029-4e200373359b\") " Oct 30 14:43:04.950364 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:04.950274 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/920bf174-09b0-4966-b029-4e200373359b-kube-api-access-swkp5\" (UniqueName: \"kubernetes.io/projected/920bf174-09b0-4966-b029-4e200373359b-kube-api-access-swkp5\") pod \"920bf174-09b0-4966-b029-4e200373359b\" (UID: \"920bf174-09b0-4966-b029-4e200373359b\") " Oct 30 14:43:04.950575 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:04.950529 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/920bf174-09b0-4966-b029-4e200373359b-plugins-config-volume" pod "920bf174-09b0-4966-b029-4e200373359b" (UID: "920bf174-09b0-4966-b029-4e200373359b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:43:04.950894 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:04.950869 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzczq\" (UniqueName: \"kubernetes.io/projected/ac0599bf-47df-45a3-96a3-55c24b5a2afd-kube-api-access-zzczq\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:43:04.950894 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:04.950891 4982 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/920bf174-09b0-4966-b029-4e200373359b-plugins-config-volume\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:43:04.952933 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:04.952908 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/920bf174-09b0-4966-b029-4e200373359b-kube-api-access-swkp5" pod "920bf174-09b0-4966-b029-4e200373359b" (UID: "920bf174-09b0-4966-b029-4e200373359b"). InnerVolumeSpecName "kube-api-access-swkp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:43:05.052411 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:05.052346 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swkp5\" (UniqueName: \"kubernetes.io/projected/920bf174-09b0-4966-b029-4e200373359b-kube-api-access-swkp5\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:43:05.538810 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:05.538776 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2927fd-844ddcc4bd-wbrs7" Oct 30 14:43:05.538810 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:05.538788 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2927fd-844ddcc4bd-wbrs7" event={"ID":"ac0599bf-47df-45a3-96a3-55c24b5a2afd","Type":"ContainerDied","Data":"adc5310d3551a031cbcae553e1182a3fa18feb60fa64a70eec3c460eefec9fb7"} Oct 30 14:43:05.539377 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:05.538857 4982 scope.go:122] "RemoveContainer" containerID="a062f00c57ee37f4201c3e5880bf0acdb49de6b18714fc39f8c4943372efe3ac" Oct 30 14:43:05.540998 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:05.540978 4982 generic.go:356] "Generic (PLEG): container finished" podID="920bf174-09b0-4966-b029-4e200373359b" containerID="3036a0f174d32eab2eddbdf5612262798ac7fd61ebd51ca95b8736d1ae0909d5" exitCode=0 Oct 30 14:43:05.541139 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:05.541023 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2927fd-55db459cf-lqzdk" event={"ID":"920bf174-09b0-4966-b029-4e200373359b","Type":"ContainerDied","Data":"3036a0f174d32eab2eddbdf5612262798ac7fd61ebd51ca95b8736d1ae0909d5"} Oct 30 14:43:05.541235 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:05.541172 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2927fd-55db459cf-lqzdk" event={"ID":"920bf174-09b0-4966-b029-4e200373359b","Type":"ContainerDied","Data":"9ce41e9e4fb2daebd4ad159fa4b968cb6121302b6dcaf2d12da4a1b54cd9beeb"} Oct 30 14:43:05.541235 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:05.541030 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2927fd-55db459cf-lqzdk" Oct 30 14:43:05.550486 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:05.550473 4982 scope.go:122] "RemoveContainer" containerID="3036a0f174d32eab2eddbdf5612262798ac7fd61ebd51ca95b8736d1ae0909d5" Oct 30 14:43:05.560773 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:05.560744 4982 scope.go:122] "RemoveContainer" containerID="3036a0f174d32eab2eddbdf5612262798ac7fd61ebd51ca95b8736d1ae0909d5" Oct 30 14:43:05.561021 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:43:05.561004 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3036a0f174d32eab2eddbdf5612262798ac7fd61ebd51ca95b8736d1ae0909d5\": container with ID starting with 3036a0f174d32eab2eddbdf5612262798ac7fd61ebd51ca95b8736d1ae0909d5 not found: ID does not exist" containerID="3036a0f174d32eab2eddbdf5612262798ac7fd61ebd51ca95b8736d1ae0909d5" Oct 30 14:43:05.561084 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:05.561027 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3036a0f174d32eab2eddbdf5612262798ac7fd61ebd51ca95b8736d1ae0909d5"} err="failed to get container status \"3036a0f174d32eab2eddbdf5612262798ac7fd61ebd51ca95b8736d1ae0909d5\": rpc error: code = NotFound desc = could not find container \"3036a0f174d32eab2eddbdf5612262798ac7fd61ebd51ca95b8736d1ae0909d5\": container with ID starting with 3036a0f174d32eab2eddbdf5612262798ac7fd61ebd51ca95b8736d1ae0909d5 not found: ID does not exist" Oct 30 14:43:05.601798 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:05.601768 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2927fd-55db459cf-lqzdk"] Oct 30 14:43:05.615222 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:05.615194 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2927fd-55db459cf-lqzdk"] Oct 30 14:43:05.651966 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:05.651932 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2927fd-844ddcc4bd-wbrs7"] Oct 30 14:43:05.663413 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:05.663387 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2927fd-844ddcc4bd-wbrs7"] Oct 30 14:43:06.370495 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:06.370466 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:06.419022 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:06.418983 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="920bf174-09b0-4966-b029-4e200373359b" path="/var/lib/kubelet/pods/920bf174-09b0-4966-b029-4e200373359b/volumes" Oct 30 14:43:06.420012 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:06.419986 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac0599bf-47df-45a3-96a3-55c24b5a2afd" path="/var/lib/kubelet/pods/ac0599bf-47df-45a3-96a3-55c24b5a2afd/volumes" Oct 30 14:43:06.547681 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:06.545654 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:07.786902 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:07.786859 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:09.574368 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:09.574337 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:11.513074 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:11.513024 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:12.402442 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:12.402399 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-460b0b-qlpfr"] Oct 30 14:43:12.403382 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:12.403362 4982 cpu_manager.go:414] "containerMap: removing container" podUID="ac0599bf-47df-45a3-96a3-55c24b5a2afd" containerName="payload-processing" Oct 30 14:43:12.403452 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:12.403385 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac0599bf-47df-45a3-96a3-55c24b5a2afd" containerName="payload-processing" Oct 30 14:43:12.403452 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:12.403411 4982 cpu_manager.go:414] "containerMap: removing container" podUID="baeb0f86-9313-4cb6-a075-8d3cdd5e9e53" containerName="payload-pre-processing" Oct 30 14:43:12.403452 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:12.403419 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baeb0f86-9313-4cb6-a075-8d3cdd5e9e53" containerName="payload-pre-processing" Oct 30 14:43:12.403452 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:12.403436 4982 cpu_manager.go:414] "containerMap: removing container" podUID="4005a383-48bb-48dd-84c0-70572307aa61" containerName="revoke-keys" Oct 30 14:43:12.403452 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:12.403444 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4005a383-48bb-48dd-84c0-70572307aa61" containerName="revoke-keys" Oct 30 14:43:12.403612 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:12.403468 4982 cpu_manager.go:414] "containerMap: removing container" podUID="920bf174-09b0-4966-b029-4e200373359b" containerName="payload-pre-processing" Oct 30 14:43:12.403612 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:12.403476 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="920bf174-09b0-4966-b029-4e200373359b" containerName="payload-pre-processing" Oct 30 14:43:12.403709 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:12.403624 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac0599bf-47df-45a3-96a3-55c24b5a2afd" containerName="payload-processing" Oct 30 14:43:12.403709 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:12.403652 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="baeb0f86-9313-4cb6-a075-8d3cdd5e9e53" containerName="payload-pre-processing" Oct 30 14:43:12.403709 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:12.403661 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="4005a383-48bb-48dd-84c0-70572307aa61" containerName="revoke-keys" Oct 30 14:43:12.403709 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:12.403672 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="920bf174-09b0-4966-b029-4e200373359b" containerName="payload-pre-processing" Oct 30 14:43:12.409157 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:12.409137 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-460b0b-qlpfr" Oct 30 14:43:12.425860 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:12.425835 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-460b0b-qlpfr"] Oct 30 14:43:12.434351 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:12.434322 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:43:12.434468 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:12.434382 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:43:12.468233 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:12.468207 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5265f061-4aa3-439e-ad80-dd952cd256d7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-460b0b-qlpfr\" (UID: \"5265f061-4aa3-439e-ad80-dd952cd256d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-460b0b-qlpfr" Oct 30 14:43:12.468462 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:12.468434 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7r4g\" (UniqueName: \"kubernetes.io/projected/5265f061-4aa3-439e-ad80-dd952cd256d7-kube-api-access-v7r4g\") pod \"maas-api-revoke-keys-e2e-ipp-b-460b0b-qlpfr\" (UID: \"5265f061-4aa3-439e-ad80-dd952cd256d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-460b0b-qlpfr" Oct 30 14:43:12.468553 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:12.468498 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5265f061-4aa3-439e-ad80-dd952cd256d7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-460b0b-qlpfr\" (UID: \"5265f061-4aa3-439e-ad80-dd952cd256d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-460b0b-qlpfr" Oct 30 14:43:12.570139 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:12.570104 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5265f061-4aa3-439e-ad80-dd952cd256d7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-460b0b-qlpfr\" (UID: \"5265f061-4aa3-439e-ad80-dd952cd256d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-460b0b-qlpfr" Oct 30 14:43:12.570562 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:12.570228 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7r4g\" (UniqueName: \"kubernetes.io/projected/5265f061-4aa3-439e-ad80-dd952cd256d7-kube-api-access-v7r4g\") pod \"maas-api-revoke-keys-e2e-ipp-b-460b0b-qlpfr\" (UID: \"5265f061-4aa3-439e-ad80-dd952cd256d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-460b0b-qlpfr" Oct 30 14:43:12.570562 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:12.570281 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5265f061-4aa3-439e-ad80-dd952cd256d7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-460b0b-qlpfr\" (UID: \"5265f061-4aa3-439e-ad80-dd952cd256d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-460b0b-qlpfr" Oct 30 14:43:12.570822 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:12.570801 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5265f061-4aa3-439e-ad80-dd952cd256d7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-460b0b-qlpfr\" (UID: \"5265f061-4aa3-439e-ad80-dd952cd256d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-460b0b-qlpfr" Oct 30 14:43:12.572762 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:12.572736 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5265f061-4aa3-439e-ad80-dd952cd256d7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-460b0b-qlpfr\" (UID: \"5265f061-4aa3-439e-ad80-dd952cd256d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-460b0b-qlpfr" Oct 30 14:43:12.596156 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:12.596130 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7r4g\" (UniqueName: \"kubernetes.io/projected/5265f061-4aa3-439e-ad80-dd952cd256d7-kube-api-access-v7r4g\") pod \"maas-api-revoke-keys-e2e-ipp-b-460b0b-qlpfr\" (UID: \"5265f061-4aa3-439e-ad80-dd952cd256d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-460b0b-qlpfr" Oct 30 14:43:12.727148 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:12.727060 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-460b0b-qlpfr" Oct 30 14:43:12.808754 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:12.808264 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c7c37b-6b9c5c5999-kdfrr" podUID="d4cdb56a-9a5e-4edc-8cec-ff32aa54de9d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:43:12.810536 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:12.810472 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c7c37b-67bb95b964-cqz6m" podUID="fb2ea041-440f-42d9-8454-6ff8bc952a5e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:43:12.999293 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:12.999216 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:13.040404 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:13.040374 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-460b0b-qlpfr"] Oct 30 14:43:13.041503 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:43:13.041476 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5265f061_4aa3_439e_ad80_dd952cd256d7.slice/crio-7f05f30bc64a99fedf81ecc24ac26bd505b7a0442c04d7ee192b40b5e4cdcd52 WatchSource:0}: Error finding container 7f05f30bc64a99fedf81ecc24ac26bd505b7a0442c04d7ee192b40b5e4cdcd52: Status 404 returned error can't find the container with id 7f05f30bc64a99fedf81ecc24ac26bd505b7a0442c04d7ee192b40b5e4cdcd52 Oct 30 14:43:13.195972 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:13.195940 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:13.623651 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:13.623051 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-460b0b-qlpfr" event={"ID":"5265f061-4aa3-439e-ad80-dd952cd256d7","Type":"ContainerStarted","Data":"7f05f30bc64a99fedf81ecc24ac26bd505b7a0442c04d7ee192b40b5e4cdcd52"} Oct 30 14:43:14.453825 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:14.453751 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:14.632726 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:14.632690 4982 generic.go:356] "Generic (PLEG): container finished" podID="5265f061-4aa3-439e-ad80-dd952cd256d7" containerID="3268e201a8fef1d71f2aa8aa6e1852040a8432144c145298c5afdbcb6067335a" exitCode=0 Oct 30 14:43:14.633295 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:14.632780 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-460b0b-qlpfr" event={"ID":"5265f061-4aa3-439e-ad80-dd952cd256d7","Type":"ContainerDied","Data":"3268e201a8fef1d71f2aa8aa6e1852040a8432144c145298c5afdbcb6067335a"} Oct 30 14:43:15.815011 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:15.814984 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-460b0b-qlpfr" Oct 30 14:43:15.952002 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:15.951924 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5265f061-4aa3-439e-ad80-dd952cd256d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5265f061-4aa3-439e-ad80-dd952cd256d7-maas-api-xks-ca\") pod \"5265f061-4aa3-439e-ad80-dd952cd256d7\" (UID: \"5265f061-4aa3-439e-ad80-dd952cd256d7\") " Oct 30 14:43:15.952162 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:15.952063 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5265f061-4aa3-439e-ad80-dd952cd256d7-kube-api-access-v7r4g\" (UniqueName: \"kubernetes.io/projected/5265f061-4aa3-439e-ad80-dd952cd256d7-kube-api-access-v7r4g\") pod \"5265f061-4aa3-439e-ad80-dd952cd256d7\" (UID: \"5265f061-4aa3-439e-ad80-dd952cd256d7\") " Oct 30 14:43:15.952223 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:15.952175 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5265f061-4aa3-439e-ad80-dd952cd256d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5265f061-4aa3-439e-ad80-dd952cd256d7-maas-api-openshift-ca\") pod \"5265f061-4aa3-439e-ad80-dd952cd256d7\" (UID: \"5265f061-4aa3-439e-ad80-dd952cd256d7\") " Oct 30 14:43:15.952567 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:15.952541 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5265f061-4aa3-439e-ad80-dd952cd256d7-maas-api-openshift-ca" pod "5265f061-4aa3-439e-ad80-dd952cd256d7" (UID: "5265f061-4aa3-439e-ad80-dd952cd256d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:43:15.952794 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:15.952776 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5265f061-4aa3-439e-ad80-dd952cd256d7-maas-api-openshift-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:43:15.954107 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:15.954086 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5265f061-4aa3-439e-ad80-dd952cd256d7-maas-api-xks-ca" pod "5265f061-4aa3-439e-ad80-dd952cd256d7" (UID: "5265f061-4aa3-439e-ad80-dd952cd256d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:43:15.954259 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:15.954237 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5265f061-4aa3-439e-ad80-dd952cd256d7-kube-api-access-v7r4g" pod "5265f061-4aa3-439e-ad80-dd952cd256d7" (UID: "5265f061-4aa3-439e-ad80-dd952cd256d7"). InnerVolumeSpecName "kube-api-access-v7r4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:43:16.053631 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:16.053601 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5265f061-4aa3-439e-ad80-dd952cd256d7-maas-api-xks-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:43:16.053631 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:16.053628 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7r4g\" (UniqueName: \"kubernetes.io/projected/5265f061-4aa3-439e-ad80-dd952cd256d7-kube-api-access-v7r4g\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:43:16.311057 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:16.311021 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:16.650613 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:16.650542 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-460b0b-qlpfr" Oct 30 14:43:16.650613 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:16.650582 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-460b0b-qlpfr" event={"ID":"5265f061-4aa3-439e-ad80-dd952cd256d7","Type":"ContainerDied","Data":"7f05f30bc64a99fedf81ecc24ac26bd505b7a0442c04d7ee192b40b5e4cdcd52"} Oct 30 14:43:16.650830 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:16.650631 4982 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f05f30bc64a99fedf81ecc24ac26bd505b7a0442c04d7ee192b40b5e4cdcd52" Oct 30 14:43:18.137469 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:18.137437 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:18.286530 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:18.286459 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:19.726531 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:19.726499 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:20.393509 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:20.393474 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:20.551898 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:20.551849 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:20.564264 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:20.564233 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c7c37b-7cf9g"] Oct 30 14:43:20.576722 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:20.576693 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c7c37b-7cf9g"] Oct 30 14:43:20.705335 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:20.705238 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:21.069093 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:21.069060 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:21.661217 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:21.661191 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c7c37b-67bb95b964-cqz6m" Oct 30 14:43:21.722352 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:21.721705 4982 generic.go:356] "Generic (PLEG): container finished" podID="d4cdb56a-9a5e-4edc-8cec-ff32aa54de9d" containerID="d6606d65c58e04ee138c32d0eda581cf0d51ee5425d375ba18b84376cde310bb" exitCode=0 Oct 30 14:43:21.722352 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:21.721743 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c7c37b-6b9c5c5999-kdfrr" event={"ID":"d4cdb56a-9a5e-4edc-8cec-ff32aa54de9d","Type":"ContainerDied","Data":"d6606d65c58e04ee138c32d0eda581cf0d51ee5425d375ba18b84376cde310bb"} Oct 30 14:43:21.724357 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:21.723577 4982 generic.go:356] "Generic (PLEG): container finished" podID="fb2ea041-440f-42d9-8454-6ff8bc952a5e" containerID="65cb85cfc52abff3dbf77c62f5b14356e5fb0146fc827d8bb7dac8349f6f0d08" exitCode=0 Oct 30 14:43:21.724357 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:21.723629 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c7c37b-67bb95b964-cqz6m" event={"ID":"fb2ea041-440f-42d9-8454-6ff8bc952a5e","Type":"ContainerDied","Data":"65cb85cfc52abff3dbf77c62f5b14356e5fb0146fc827d8bb7dac8349f6f0d08"} Oct 30 14:43:21.724357 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:21.723697 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c7c37b-67bb95b964-cqz6m" event={"ID":"fb2ea041-440f-42d9-8454-6ff8bc952a5e","Type":"ContainerDied","Data":"d0774ddfba81cb8074ecabd6d93f1b16982d65442726b56374d2075628ffea05"} Oct 30 14:43:21.724357 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:21.723731 4982 scope.go:122] "RemoveContainer" containerID="65cb85cfc52abff3dbf77c62f5b14356e5fb0146fc827d8bb7dac8349f6f0d08" Oct 30 14:43:21.724357 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:21.723883 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c7c37b-67bb95b964-cqz6m" Oct 30 14:43:21.744875 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:21.743852 4982 scope.go:122] "RemoveContainer" containerID="65cb85cfc52abff3dbf77c62f5b14356e5fb0146fc827d8bb7dac8349f6f0d08" Oct 30 14:43:21.744875 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:43:21.744185 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65cb85cfc52abff3dbf77c62f5b14356e5fb0146fc827d8bb7dac8349f6f0d08\": container with ID starting with 65cb85cfc52abff3dbf77c62f5b14356e5fb0146fc827d8bb7dac8349f6f0d08 not found: ID does not exist" containerID="65cb85cfc52abff3dbf77c62f5b14356e5fb0146fc827d8bb7dac8349f6f0d08" Oct 30 14:43:21.744875 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:21.744220 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65cb85cfc52abff3dbf77c62f5b14356e5fb0146fc827d8bb7dac8349f6f0d08"} err="failed to get container status \"65cb85cfc52abff3dbf77c62f5b14356e5fb0146fc827d8bb7dac8349f6f0d08\": rpc error: code = NotFound desc = could not find container \"65cb85cfc52abff3dbf77c62f5b14356e5fb0146fc827d8bb7dac8349f6f0d08\": container with ID starting with 65cb85cfc52abff3dbf77c62f5b14356e5fb0146fc827d8bb7dac8349f6f0d08 not found: ID does not exist" Oct 30 14:43:21.744875 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:21.744353 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb2ea041-440f-42d9-8454-6ff8bc952a5e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb2ea041-440f-42d9-8454-6ff8bc952a5e-plugins-config-volume\") pod \"fb2ea041-440f-42d9-8454-6ff8bc952a5e\" (UID: \"fb2ea041-440f-42d9-8454-6ff8bc952a5e\") " Oct 30 14:43:21.744875 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:21.744415 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb2ea041-440f-42d9-8454-6ff8bc952a5e-kube-api-access-bzbxv\" (UniqueName: \"kubernetes.io/projected/fb2ea041-440f-42d9-8454-6ff8bc952a5e-kube-api-access-bzbxv\") pod \"fb2ea041-440f-42d9-8454-6ff8bc952a5e\" (UID: \"fb2ea041-440f-42d9-8454-6ff8bc952a5e\") " Oct 30 14:43:21.744875 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:21.744842 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb2ea041-440f-42d9-8454-6ff8bc952a5e-plugins-config-volume" pod "fb2ea041-440f-42d9-8454-6ff8bc952a5e" (UID: "fb2ea041-440f-42d9-8454-6ff8bc952a5e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:43:21.745285 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:21.745085 4982 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb2ea041-440f-42d9-8454-6ff8bc952a5e-plugins-config-volume\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:43:21.747819 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:21.747793 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb2ea041-440f-42d9-8454-6ff8bc952a5e-kube-api-access-bzbxv" pod "fb2ea041-440f-42d9-8454-6ff8bc952a5e" (UID: "fb2ea041-440f-42d9-8454-6ff8bc952a5e"). InnerVolumeSpecName "kube-api-access-bzbxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:43:21.775000 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:21.774980 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c7c37b-6b9c5c5999-kdfrr" Oct 30 14:43:21.845697 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:21.845670 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4cdb56a-9a5e-4edc-8cec-ff32aa54de9d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4cdb56a-9a5e-4edc-8cec-ff32aa54de9d-plugins-config-volume\") pod \"d4cdb56a-9a5e-4edc-8cec-ff32aa54de9d\" (UID: \"d4cdb56a-9a5e-4edc-8cec-ff32aa54de9d\") " Oct 30 14:43:21.845824 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:21.845810 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4cdb56a-9a5e-4edc-8cec-ff32aa54de9d-kube-api-access-7vm9t\" (UniqueName: \"kubernetes.io/projected/d4cdb56a-9a5e-4edc-8cec-ff32aa54de9d-kube-api-access-7vm9t\") pod \"d4cdb56a-9a5e-4edc-8cec-ff32aa54de9d\" (UID: \"d4cdb56a-9a5e-4edc-8cec-ff32aa54de9d\") " Oct 30 14:43:21.845971 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:21.845947 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4cdb56a-9a5e-4edc-8cec-ff32aa54de9d-plugins-config-volume" pod "d4cdb56a-9a5e-4edc-8cec-ff32aa54de9d" (UID: "d4cdb56a-9a5e-4edc-8cec-ff32aa54de9d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:43:21.846269 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:21.846254 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzbxv\" (UniqueName: \"kubernetes.io/projected/fb2ea041-440f-42d9-8454-6ff8bc952a5e-kube-api-access-bzbxv\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:43:21.846362 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:21.846270 4982 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4cdb56a-9a5e-4edc-8cec-ff32aa54de9d-plugins-config-volume\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:43:21.847874 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:21.847853 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4cdb56a-9a5e-4edc-8cec-ff32aa54de9d-kube-api-access-7vm9t" pod "d4cdb56a-9a5e-4edc-8cec-ff32aa54de9d" (UID: "d4cdb56a-9a5e-4edc-8cec-ff32aa54de9d"). InnerVolumeSpecName "kube-api-access-7vm9t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:43:21.947776 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:21.947712 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vm9t\" (UniqueName: \"kubernetes.io/projected/d4cdb56a-9a5e-4edc-8cec-ff32aa54de9d-kube-api-access-7vm9t\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:43:22.075608 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:22.075579 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-c7c37b-67bb95b964-cqz6m"] Oct 30 14:43:22.089255 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:22.089231 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-c7c37b-67bb95b964-cqz6m"] Oct 30 14:43:22.422287 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:22.420452 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4005a383-48bb-48dd-84c0-70572307aa61" path="/var/lib/kubelet/pods/4005a383-48bb-48dd-84c0-70572307aa61/volumes" Oct 30 14:43:22.422287 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:22.421065 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb2ea041-440f-42d9-8454-6ff8bc952a5e" path="/var/lib/kubelet/pods/fb2ea041-440f-42d9-8454-6ff8bc952a5e/volumes" Oct 30 14:43:22.728478 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:22.728400 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:22.734031 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:22.734005 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c7c37b-6b9c5c5999-kdfrr" Oct 30 14:43:22.734167 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:22.734039 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c7c37b-6b9c5c5999-kdfrr" event={"ID":"d4cdb56a-9a5e-4edc-8cec-ff32aa54de9d","Type":"ContainerDied","Data":"f3e9ed0d7231df82d4ddcc30f7b2f7eb9f2ba4253a8143d4f8df6e74438814b8"} Oct 30 14:43:22.734167 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:22.734102 4982 scope.go:122] "RemoveContainer" containerID="d6606d65c58e04ee138c32d0eda581cf0d51ee5425d375ba18b84376cde310bb" Oct 30 14:43:22.794514 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:22.794478 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c7c37b-6b9c5c5999-kdfrr"] Oct 30 14:43:22.809610 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:22.809576 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c7c37b-6b9c5c5999-kdfrr"] Oct 30 14:43:24.188804 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:24.188772 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:24.419110 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:24.419064 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4cdb56a-9a5e-4edc-8cec-ff32aa54de9d" path="/var/lib/kubelet/pods/d4cdb56a-9a5e-4edc-8cec-ff32aa54de9d/volumes" Oct 30 14:43:24.686619 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:24.686587 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:24.771320 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:24.771273 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3b95dd-5ccb7d956-4mkxk"] Oct 30 14:43:24.848716 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:24.845678 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3b95dd-858549854c-z49d4"] Oct 30 14:43:24.940713 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:24.940035 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:25.096176 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:25.096141 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:25.244417 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:25.244323 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:26.613114 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:26.613079 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:28.011679 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:43:28.011604 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:43:28.077477 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:28.077439 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:28.188253 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:28.188217 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-460b0b-5d5bdd49b-m8txx"] Oct 30 14:43:28.188585 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:28.188461 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-460b0b-5d5bdd49b-m8txx" podUID="311279d9-d231-4177-8401-9e9598ed9ef0" containerName="maas-api" containerID="cri-o://eb5f36da0b2602aa195ed56201e876135b37dae4ad114efdd728c0648b186b90" gracePeriod=30 Oct 30 14:43:28.276071 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:28.275997 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:28.363890 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:28.363852 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-460b0b-55d85f88fb-7rkhm"] Oct 30 14:43:28.364134 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:28.364109 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-460b0b-55d85f88fb-7rkhm" podUID="9d600402-b61d-48af-b02e-d38d9adf5b3b" containerName="payload-processing" containerID="cri-o://71f3f2b588eac942975d20be6b5fccb32c43c3197ddcfe004adc0c438434ae6e" gracePeriod=30 Oct 30 14:43:28.504336 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:28.504295 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:28.539223 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:28.530687 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27"] Oct 30 14:43:28.539223 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:28.531576 4982 cpu_manager.go:414] "containerMap: removing container" podUID="d4cdb56a-9a5e-4edc-8cec-ff32aa54de9d" containerName="payload-pre-processing" Oct 30 14:43:28.539223 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:28.531596 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4cdb56a-9a5e-4edc-8cec-ff32aa54de9d" containerName="payload-pre-processing" Oct 30 14:43:28.539223 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:28.531654 4982 cpu_manager.go:414] "containerMap: removing container" podUID="fb2ea041-440f-42d9-8454-6ff8bc952a5e" containerName="payload-processing" Oct 30 14:43:28.539223 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:28.531661 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb2ea041-440f-42d9-8454-6ff8bc952a5e" containerName="payload-processing" Oct 30 14:43:28.539223 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:28.531679 4982 cpu_manager.go:414] "containerMap: removing container" podUID="5265f061-4aa3-439e-ad80-dd952cd256d7" containerName="revoke-keys" Oct 30 14:43:28.539223 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:28.531686 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5265f061-4aa3-439e-ad80-dd952cd256d7" containerName="revoke-keys" Oct 30 14:43:28.539223 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:28.531799 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb2ea041-440f-42d9-8454-6ff8bc952a5e" containerName="payload-processing" Oct 30 14:43:28.539223 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:28.531811 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="5265f061-4aa3-439e-ad80-dd952cd256d7" containerName="revoke-keys" Oct 30 14:43:28.539223 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:28.531821 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4cdb56a-9a5e-4edc-8cec-ff32aa54de9d" containerName="payload-pre-processing" Oct 30 14:43:28.548785 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:28.544907 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27"] Oct 30 14:43:28.548785 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:28.545018 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27" Oct 30 14:43:28.564622 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:28.564595 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-90b406-openshift-default-dockercfg-mj4v6\"" Oct 30 14:43:28.627034 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:28.626761 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-460b0b-5d5bdd49b-m8txx" Oct 30 14:43:28.659662 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:28.651208 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:28.673667 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:28.670031 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/311279d9-d231-4177-8401-9e9598ed9ef0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/311279d9-d231-4177-8401-9e9598ed9ef0-metrics-certs\") pod \"311279d9-d231-4177-8401-9e9598ed9ef0\" (UID: \"311279d9-d231-4177-8401-9e9598ed9ef0\") " Oct 30 14:43:28.673667 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:28.670619 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/311279d9-d231-4177-8401-9e9598ed9ef0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/311279d9-d231-4177-8401-9e9598ed9ef0-maas-api-tls\") pod \"311279d9-d231-4177-8401-9e9598ed9ef0\" (UID: \"311279d9-d231-4177-8401-9e9598ed9ef0\") " Oct 30 14:43:28.673667 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:28.671191 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/311279d9-d231-4177-8401-9e9598ed9ef0-kube-api-access-nl92b\" (UniqueName: \"kubernetes.io/projected/311279d9-d231-4177-8401-9e9598ed9ef0-kube-api-access-nl92b\") pod \"311279d9-d231-4177-8401-9e9598ed9ef0\" (UID: \"311279d9-d231-4177-8401-9e9598ed9ef0\") " Oct 30 14:43:28.673667 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:28.671385 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/311279d9-d231-4177-8401-9e9598ed9ef0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/311279d9-d231-4177-8401-9e9598ed9ef0-openshift-service-ca\") pod \"311279d9-d231-4177-8401-9e9598ed9ef0\" (UID: \"311279d9-d231-4177-8401-9e9598ed9ef0\") " Oct 30 14:43:28.673667 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:28.671655 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b853a87-0adf-46f0-b45d-c17b69a68495-credential-socket\") pod \"e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27\" (UID: \"9b853a87-0adf-46f0-b45d-c17b69a68495\") " pod="openshift-ingress/e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27" Oct 30 14:43:28.673667 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:28.671737 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b853a87-0adf-46f0-b45d-c17b69a68495-istio-data\") pod \"e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27\" (UID: \"9b853a87-0adf-46f0-b45d-c17b69a68495\") " pod="openshift-ingress/e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27" Oct 30 14:43:28.673667 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:28.671775 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b853a87-0adf-46f0-b45d-c17b69a68495-workload-socket\") pod \"e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27\" (UID: \"9b853a87-0adf-46f0-b45d-c17b69a68495\") " pod="openshift-ingress/e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27" Oct 30 14:43:28.673667 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:28.671892 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b853a87-0adf-46f0-b45d-c17b69a68495-workload-certs\") pod \"e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27\" (UID: \"9b853a87-0adf-46f0-b45d-c17b69a68495\") " pod="openshift-ingress/e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27" Oct 30 14:43:28.673667 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:28.671921 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b853a87-0adf-46f0-b45d-c17b69a68495-istio-podinfo\") pod \"e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27\" (UID: \"9b853a87-0adf-46f0-b45d-c17b69a68495\") " pod="openshift-ingress/e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27" Oct 30 14:43:28.673667 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:28.671956 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b853a87-0adf-46f0-b45d-c17b69a68495-istio-envoy\") pod \"e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27\" (UID: \"9b853a87-0adf-46f0-b45d-c17b69a68495\") " pod="openshift-ingress/e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27" Oct 30 14:43:28.673667 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:28.672025 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b853a87-0adf-46f0-b45d-c17b69a68495-istio-token\") pod \"e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27\" (UID: \"9b853a87-0adf-46f0-b45d-c17b69a68495\") " pod="openshift-ingress/e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27" Oct 30 14:43:28.673667 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:28.672122 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr4qg\" (UniqueName: \"kubernetes.io/projected/9b853a87-0adf-46f0-b45d-c17b69a68495-kube-api-access-dr4qg\") pod \"e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27\" (UID: \"9b853a87-0adf-46f0-b45d-c17b69a68495\") " pod="openshift-ingress/e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27" Oct 30 14:43:28.673667 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:28.672190 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b853a87-0adf-46f0-b45d-c17b69a68495-istiod-ca-cert\") pod \"e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27\" (UID: \"9b853a87-0adf-46f0-b45d-c17b69a68495\") " pod="openshift-ingress/e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27" Oct 30 14:43:28.673667 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:28.673525 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/311279d9-d231-4177-8401-9e9598ed9ef0-openshift-service-ca" pod "311279d9-d231-4177-8401-9e9598ed9ef0" (UID: "311279d9-d231-4177-8401-9e9598ed9ef0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:43:28.706695 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:28.697951 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/311279d9-d231-4177-8401-9e9598ed9ef0-kube-api-access-nl92b" pod "311279d9-d231-4177-8401-9e9598ed9ef0" (UID: "311279d9-d231-4177-8401-9e9598ed9ef0"). InnerVolumeSpecName "kube-api-access-nl92b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:43:28.706695 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:28.706018 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/311279d9-d231-4177-8401-9e9598ed9ef0-metrics-certs" pod "311279d9-d231-4177-8401-9e9598ed9ef0" (UID: "311279d9-d231-4177-8401-9e9598ed9ef0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:43:28.706695 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:28.706064 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/311279d9-d231-4177-8401-9e9598ed9ef0-maas-api-tls" pod "311279d9-d231-4177-8401-9e9598ed9ef0" (UID: "311279d9-d231-4177-8401-9e9598ed9ef0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:43:28.754298 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:28.754268 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-460b0b-55d85f88fb-7rkhm" Oct 30 14:43:28.773691 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:28.773657 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b853a87-0adf-46f0-b45d-c17b69a68495-istio-token\") pod \"e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27\" (UID: \"9b853a87-0adf-46f0-b45d-c17b69a68495\") " pod="openshift-ingress/e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27" Oct 30 14:43:28.773838 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:28.773769 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr4qg\" (UniqueName: \"kubernetes.io/projected/9b853a87-0adf-46f0-b45d-c17b69a68495-kube-api-access-dr4qg\") pod \"e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27\" (UID: \"9b853a87-0adf-46f0-b45d-c17b69a68495\") " pod="openshift-ingress/e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27" Oct 30 14:43:28.773838 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:28.773828 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b853a87-0adf-46f0-b45d-c17b69a68495-istiod-ca-cert\") pod \"e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27\" (UID: \"9b853a87-0adf-46f0-b45d-c17b69a68495\") " pod="openshift-ingress/e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27" Oct 30 14:43:28.773952 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:28.773931 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b853a87-0adf-46f0-b45d-c17b69a68495-credential-socket\") pod \"e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27\" (UID: \"9b853a87-0adf-46f0-b45d-c17b69a68495\") " pod="openshift-ingress/e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27" Oct 30 14:43:28.774015 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:28.774000 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b853a87-0adf-46f0-b45d-c17b69a68495-istio-data\") pod \"e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27\" (UID: \"9b853a87-0adf-46f0-b45d-c17b69a68495\") " pod="openshift-ingress/e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27" Oct 30 14:43:28.774068 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:28.774034 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b853a87-0adf-46f0-b45d-c17b69a68495-workload-socket\") pod \"e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27\" (UID: \"9b853a87-0adf-46f0-b45d-c17b69a68495\") " pod="openshift-ingress/e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27" Oct 30 14:43:28.774159 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:28.774142 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b853a87-0adf-46f0-b45d-c17b69a68495-workload-certs\") pod \"e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27\" (UID: \"9b853a87-0adf-46f0-b45d-c17b69a68495\") " pod="openshift-ingress/e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27" Oct 30 14:43:28.774206 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:28.774175 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b853a87-0adf-46f0-b45d-c17b69a68495-istio-podinfo\") pod \"e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27\" (UID: \"9b853a87-0adf-46f0-b45d-c17b69a68495\") " pod="openshift-ingress/e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27" Oct 30 14:43:28.774243 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:28.774209 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b853a87-0adf-46f0-b45d-c17b69a68495-istio-envoy\") pod \"e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27\" (UID: \"9b853a87-0adf-46f0-b45d-c17b69a68495\") " pod="openshift-ingress/e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27" Oct 30 14:43:28.774317 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:28.774304 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/311279d9-d231-4177-8401-9e9598ed9ef0-maas-api-tls\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:43:28.774367 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:28.774325 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nl92b\" (UniqueName: \"kubernetes.io/projected/311279d9-d231-4177-8401-9e9598ed9ef0-kube-api-access-nl92b\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:43:28.774367 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:28.774340 4982 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/311279d9-d231-4177-8401-9e9598ed9ef0-openshift-service-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:43:28.774367 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:28.774355 4982 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/311279d9-d231-4177-8401-9e9598ed9ef0-metrics-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:43:28.774867 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:28.774804 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b853a87-0adf-46f0-b45d-c17b69a68495-istiod-ca-cert\") pod \"e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27\" (UID: \"9b853a87-0adf-46f0-b45d-c17b69a68495\") " pod="openshift-ingress/e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27" Oct 30 14:43:28.775351 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:28.775323 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b853a87-0adf-46f0-b45d-c17b69a68495-workload-socket\") pod \"e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27\" (UID: \"9b853a87-0adf-46f0-b45d-c17b69a68495\") " pod="openshift-ingress/e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27" Oct 30 14:43:28.775598 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:28.775578 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b853a87-0adf-46f0-b45d-c17b69a68495-credential-socket\") pod \"e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27\" (UID: \"9b853a87-0adf-46f0-b45d-c17b69a68495\") " pod="openshift-ingress/e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27" Oct 30 14:43:28.775851 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:28.775830 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b853a87-0adf-46f0-b45d-c17b69a68495-istio-data\") pod \"e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27\" (UID: \"9b853a87-0adf-46f0-b45d-c17b69a68495\") " pod="openshift-ingress/e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27" Oct 30 14:43:28.776110 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:28.776081 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b853a87-0adf-46f0-b45d-c17b69a68495-workload-certs\") pod \"e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27\" (UID: \"9b853a87-0adf-46f0-b45d-c17b69a68495\") " pod="openshift-ingress/e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27" Oct 30 14:43:28.778054 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:28.777495 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b853a87-0adf-46f0-b45d-c17b69a68495-istio-envoy\") pod \"e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27\" (UID: \"9b853a87-0adf-46f0-b45d-c17b69a68495\") " pod="openshift-ingress/e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27" Oct 30 14:43:28.785142 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:28.785115 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b853a87-0adf-46f0-b45d-c17b69a68495-istio-podinfo\") pod \"e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27\" (UID: \"9b853a87-0adf-46f0-b45d-c17b69a68495\") " pod="openshift-ingress/e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27" Oct 30 14:43:28.802407 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:28.802278 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b853a87-0adf-46f0-b45d-c17b69a68495-istio-token\") pod \"e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27\" (UID: \"9b853a87-0adf-46f0-b45d-c17b69a68495\") " pod="openshift-ingress/e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27" Oct 30 14:43:28.802679 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:28.802609 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr4qg\" (UniqueName: \"kubernetes.io/projected/9b853a87-0adf-46f0-b45d-c17b69a68495-kube-api-access-dr4qg\") pod \"e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27\" (UID: \"9b853a87-0adf-46f0-b45d-c17b69a68495\") " pod="openshift-ingress/e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27" Oct 30 14:43:28.826858 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:28.826776 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:28.909676 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:28.876381 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d600402-b61d-48af-b02e-d38d9adf5b3b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d600402-b61d-48af-b02e-d38d9adf5b3b-plugins-config-volume\") pod \"9d600402-b61d-48af-b02e-d38d9adf5b3b\" (UID: \"9d600402-b61d-48af-b02e-d38d9adf5b3b\") " Oct 30 14:43:28.927842 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:28.910292 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d600402-b61d-48af-b02e-d38d9adf5b3b-kube-api-access-n6zss\" (UniqueName: \"kubernetes.io/projected/9d600402-b61d-48af-b02e-d38d9adf5b3b-kube-api-access-n6zss\") pod \"9d600402-b61d-48af-b02e-d38d9adf5b3b\" (UID: \"9d600402-b61d-48af-b02e-d38d9adf5b3b\") " Oct 30 14:43:29.037754 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:28.895246 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d600402-b61d-48af-b02e-d38d9adf5b3b-plugins-config-volume" pod "9d600402-b61d-48af-b02e-d38d9adf5b3b" (UID: "9d600402-b61d-48af-b02e-d38d9adf5b3b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:43:29.038281 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:28.928389 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27" Oct 30 14:43:29.040746 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:29.040714 4982 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d600402-b61d-48af-b02e-d38d9adf5b3b-plugins-config-volume\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:43:29.063179 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:29.063139 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d600402-b61d-48af-b02e-d38d9adf5b3b-kube-api-access-n6zss" pod "9d600402-b61d-48af-b02e-d38d9adf5b3b" (UID: "9d600402-b61d-48af-b02e-d38d9adf5b3b"). InnerVolumeSpecName "kube-api-access-n6zss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:43:29.081902 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:29.079111 4982 generic.go:356] "Generic (PLEG): container finished" podID="9d600402-b61d-48af-b02e-d38d9adf5b3b" containerID="71f3f2b588eac942975d20be6b5fccb32c43c3197ddcfe004adc0c438434ae6e" exitCode=0 Oct 30 14:43:29.081902 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:29.079188 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-460b0b-55d85f88fb-7rkhm" Oct 30 14:43:29.081902 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:29.079293 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-460b0b-55d85f88fb-7rkhm" event={"ID":"9d600402-b61d-48af-b02e-d38d9adf5b3b","Type":"ContainerDied","Data":"71f3f2b588eac942975d20be6b5fccb32c43c3197ddcfe004adc0c438434ae6e"} Oct 30 14:43:29.081902 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:29.079359 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-460b0b-55d85f88fb-7rkhm" event={"ID":"9d600402-b61d-48af-b02e-d38d9adf5b3b","Type":"ContainerDied","Data":"a5b6e5021ff16ccd8a6155a58608feae849ff103bfc7dcd2af47ecfe304f98da"} Oct 30 14:43:29.081902 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:29.079384 4982 scope.go:122] "RemoveContainer" containerID="71f3f2b588eac942975d20be6b5fccb32c43c3197ddcfe004adc0c438434ae6e" Oct 30 14:43:29.082452 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:29.082427 4982 generic.go:356] "Generic (PLEG): container finished" podID="311279d9-d231-4177-8401-9e9598ed9ef0" containerID="eb5f36da0b2602aa195ed56201e876135b37dae4ad114efdd728c0648b186b90" exitCode=0 Oct 30 14:43:29.083028 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:29.082988 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-460b0b-5d5bdd49b-m8txx" event={"ID":"311279d9-d231-4177-8401-9e9598ed9ef0","Type":"ContainerDied","Data":"eb5f36da0b2602aa195ed56201e876135b37dae4ad114efdd728c0648b186b90"} Oct 30 14:43:29.083131 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:29.083060 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-460b0b-5d5bdd49b-m8txx" event={"ID":"311279d9-d231-4177-8401-9e9598ed9ef0","Type":"ContainerDied","Data":"3720d60dfeaecd0a2c611d171a537027d76ec593dfe0fd5f74fd95e051fdb9c6"} Oct 30 14:43:29.083190 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:29.083142 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-460b0b-5d5bdd49b-m8txx" Oct 30 14:43:29.090018 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:29.089995 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:29.100488 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:29.100470 4982 scope.go:122] "RemoveContainer" containerID="71f3f2b588eac942975d20be6b5fccb32c43c3197ddcfe004adc0c438434ae6e" Oct 30 14:43:29.100810 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:43:29.100788 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71f3f2b588eac942975d20be6b5fccb32c43c3197ddcfe004adc0c438434ae6e\": container with ID starting with 71f3f2b588eac942975d20be6b5fccb32c43c3197ddcfe004adc0c438434ae6e not found: ID does not exist" containerID="71f3f2b588eac942975d20be6b5fccb32c43c3197ddcfe004adc0c438434ae6e" Oct 30 14:43:29.100896 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:29.100815 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71f3f2b588eac942975d20be6b5fccb32c43c3197ddcfe004adc0c438434ae6e"} err="failed to get container status \"71f3f2b588eac942975d20be6b5fccb32c43c3197ddcfe004adc0c438434ae6e\": rpc error: code = NotFound desc = could not find container \"71f3f2b588eac942975d20be6b5fccb32c43c3197ddcfe004adc0c438434ae6e\": container with ID starting with 71f3f2b588eac942975d20be6b5fccb32c43c3197ddcfe004adc0c438434ae6e not found: ID does not exist" Oct 30 14:43:29.100896 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:29.100831 4982 scope.go:122] "RemoveContainer" containerID="eb5f36da0b2602aa195ed56201e876135b37dae4ad114efdd728c0648b186b90" Oct 30 14:43:29.112668 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:29.112620 4982 scope.go:122] "RemoveContainer" containerID="eb5f36da0b2602aa195ed56201e876135b37dae4ad114efdd728c0648b186b90" Oct 30 14:43:29.112982 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:43:29.112963 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb5f36da0b2602aa195ed56201e876135b37dae4ad114efdd728c0648b186b90\": container with ID starting with eb5f36da0b2602aa195ed56201e876135b37dae4ad114efdd728c0648b186b90 not found: ID does not exist" containerID="eb5f36da0b2602aa195ed56201e876135b37dae4ad114efdd728c0648b186b90" Oct 30 14:43:29.113047 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:29.112989 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb5f36da0b2602aa195ed56201e876135b37dae4ad114efdd728c0648b186b90"} err="failed to get container status \"eb5f36da0b2602aa195ed56201e876135b37dae4ad114efdd728c0648b186b90\": rpc error: code = NotFound desc = could not find container \"eb5f36da0b2602aa195ed56201e876135b37dae4ad114efdd728c0648b186b90\": container with ID starting with eb5f36da0b2602aa195ed56201e876135b37dae4ad114efdd728c0648b186b90 not found: ID does not exist" Oct 30 14:43:29.142093 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:29.142059 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6zss\" (UniqueName: \"kubernetes.io/projected/9d600402-b61d-48af-b02e-d38d9adf5b3b-kube-api-access-n6zss\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:43:29.159039 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:29.158942 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-460b0b-55d85f88fb-7rkhm"] Oct 30 14:43:29.165806 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:29.164912 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-460b0b-55d85f88fb-7rkhm"] Oct 30 14:43:29.208580 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:29.208548 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-460b0b-5d5bdd49b-m8txx"] Oct 30 14:43:29.214695 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:29.214671 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-460b0b-5d5bdd49b-m8txx"] Oct 30 14:43:29.249590 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:29.249557 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:29.330604 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:29.330511 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-460b0b-qlpfr"] Oct 30 14:43:29.337657 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:29.337586 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-460b0b-qlpfr"] Oct 30 14:43:29.407687 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:29.406251 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:29.439871 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:29.439843 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27"] Oct 30 14:43:29.443805 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:43:29.443773 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b853a87_0adf_46f0_b45d_c17b69a68495.slice/crio-8137093d76c5b51d2d6cdcfc7d27313f5dea91c5331556ce0f0a26d95dbb2749 WatchSource:0}: Error finding container 8137093d76c5b51d2d6cdcfc7d27313f5dea91c5331556ce0f0a26d95dbb2749: Status 404 returned error can't find the container with id 8137093d76c5b51d2d6cdcfc7d27313f5dea91c5331556ce0f0a26d95dbb2749 Oct 30 14:43:29.447597 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:29.447557 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:43:29.447724 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:29.447666 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:43:29.447724 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:29.447713 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:43:29.547720 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:29.545281 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2"] Oct 30 14:43:29.547720 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:29.545659 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2" podUID="f8c75528-f183-4379-841e-e413ac8d9c35" containerName="istio-proxy" containerID="cri-o://c66d067be4b524c1eceba229db9944d94415e0dfeda879690abcb03637b658bb" gracePeriod=30 Oct 30 14:43:29.605579 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:29.605501 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:29.772670 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:29.771780 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-3b95dd-5ccb7d956-4mkxk" podUID="a53fc31f-d9a7-45d9-8564-1e9b695c9bbf" containerName="payload-processing" containerID="cri-o://c30a0521c45fb6929029a53be0fa71eeef9b5d563e600b3fb079364f3623f4e8" gracePeriod=25 Oct 30 14:43:29.853321 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:29.853275 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b95dd-858549854c-z49d4" podUID="fe8f23d0-2fda-4393-967d-c21cbbc8afe9" containerName="payload-pre-processing" containerID="cri-o://54469c4a0663a338a6a17c7bc8093d3c756a622e43d43fb387efdfa902ad2e87" gracePeriod=25 Oct 30 14:43:29.855901 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:29.855834 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:30.107212 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:30.106959 4982 generic.go:356] "Generic (PLEG): container finished" podID="fe8f23d0-2fda-4393-967d-c21cbbc8afe9" containerID="54469c4a0663a338a6a17c7bc8093d3c756a622e43d43fb387efdfa902ad2e87" exitCode=0 Oct 30 14:43:30.107212 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:30.107080 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b95dd-858549854c-z49d4" event={"ID":"fe8f23d0-2fda-4393-967d-c21cbbc8afe9","Type":"ContainerDied","Data":"54469c4a0663a338a6a17c7bc8093d3c756a622e43d43fb387efdfa902ad2e87"} Oct 30 14:43:30.111121 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:30.110978 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27" event={"ID":"9b853a87-0adf-46f0-b45d-c17b69a68495","Type":"ContainerStarted","Data":"f40b22d496fc456deef3f35cbfb2bd4fce960f0bd73d35e1396e9f5ea8ab6e7a"} Oct 30 14:43:30.114022 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:30.113908 4982 generic.go:356] "Generic (PLEG): container finished" podID="a53fc31f-d9a7-45d9-8564-1e9b695c9bbf" containerID="c30a0521c45fb6929029a53be0fa71eeef9b5d563e600b3fb079364f3623f4e8" exitCode=0 Oct 30 14:43:30.115882 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:30.111033 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27" event={"ID":"9b853a87-0adf-46f0-b45d-c17b69a68495","Type":"ContainerStarted","Data":"8137093d76c5b51d2d6cdcfc7d27313f5dea91c5331556ce0f0a26d95dbb2749"} Oct 30 14:43:30.116686 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:30.116588 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3b95dd-5ccb7d956-4mkxk" event={"ID":"a53fc31f-d9a7-45d9-8564-1e9b695c9bbf","Type":"ContainerDied","Data":"c30a0521c45fb6929029a53be0fa71eeef9b5d563e600b3fb079364f3623f4e8"} Oct 30 14:43:30.150747 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:30.147306 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b95dd-858549854c-z49d4" Oct 30 14:43:30.168486 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:30.166517 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27" podStartSLOduration=2.16650115 podStartE2EDuration="2.16650115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:43:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:43:30.165096552 +0000 UTC m=+3279.909230595" watchObservedRunningTime="2026-10-30 14:43:30.16650115 +0000 UTC m=+3279.910635191" Oct 30 14:43:30.245658 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:30.242630 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3b95dd-5ccb7d956-4mkxk" Oct 30 14:43:30.271238 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:30.271210 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe8f23d0-2fda-4393-967d-c21cbbc8afe9-kube-api-access-ssxqs\" (UniqueName: \"kubernetes.io/projected/fe8f23d0-2fda-4393-967d-c21cbbc8afe9-kube-api-access-ssxqs\") pod \"fe8f23d0-2fda-4393-967d-c21cbbc8afe9\" (UID: \"fe8f23d0-2fda-4393-967d-c21cbbc8afe9\") " Oct 30 14:43:30.271419 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:30.271242 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe8f23d0-2fda-4393-967d-c21cbbc8afe9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe8f23d0-2fda-4393-967d-c21cbbc8afe9-plugins-config-volume\") pod \"fe8f23d0-2fda-4393-967d-c21cbbc8afe9\" (UID: \"fe8f23d0-2fda-4393-967d-c21cbbc8afe9\") " Oct 30 14:43:30.271848 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:30.271810 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe8f23d0-2fda-4393-967d-c21cbbc8afe9-plugins-config-volume" pod "fe8f23d0-2fda-4393-967d-c21cbbc8afe9" (UID: "fe8f23d0-2fda-4393-967d-c21cbbc8afe9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:43:30.272019 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:30.271982 4982 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe8f23d0-2fda-4393-967d-c21cbbc8afe9-plugins-config-volume\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:43:30.274083 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:30.274056 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe8f23d0-2fda-4393-967d-c21cbbc8afe9-kube-api-access-ssxqs" pod "fe8f23d0-2fda-4393-967d-c21cbbc8afe9" (UID: "fe8f23d0-2fda-4393-967d-c21cbbc8afe9"). InnerVolumeSpecName "kube-api-access-ssxqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:43:30.372732 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:30.372628 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a53fc31f-d9a7-45d9-8564-1e9b695c9bbf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a53fc31f-d9a7-45d9-8564-1e9b695c9bbf-plugins-config-volume\") pod \"a53fc31f-d9a7-45d9-8564-1e9b695c9bbf\" (UID: \"a53fc31f-d9a7-45d9-8564-1e9b695c9bbf\") " Oct 30 14:43:30.372914 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:30.372786 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a53fc31f-d9a7-45d9-8564-1e9b695c9bbf-kube-api-access-rjfzd\" (UniqueName: \"kubernetes.io/projected/a53fc31f-d9a7-45d9-8564-1e9b695c9bbf-kube-api-access-rjfzd\") pod \"a53fc31f-d9a7-45d9-8564-1e9b695c9bbf\" (UID: \"a53fc31f-d9a7-45d9-8564-1e9b695c9bbf\") " Oct 30 14:43:30.373371 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:30.373343 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssxqs\" (UniqueName: \"kubernetes.io/projected/fe8f23d0-2fda-4393-967d-c21cbbc8afe9-kube-api-access-ssxqs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:43:30.373480 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:30.373351 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a53fc31f-d9a7-45d9-8564-1e9b695c9bbf-plugins-config-volume" pod "a53fc31f-d9a7-45d9-8564-1e9b695c9bbf" (UID: "a53fc31f-d9a7-45d9-8564-1e9b695c9bbf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:43:30.382824 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:30.382798 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a53fc31f-d9a7-45d9-8564-1e9b695c9bbf-kube-api-access-rjfzd" pod "a53fc31f-d9a7-45d9-8564-1e9b695c9bbf" (UID: "a53fc31f-d9a7-45d9-8564-1e9b695c9bbf"). InnerVolumeSpecName "kube-api-access-rjfzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:43:30.401776 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:30.401746 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="311279d9-d231-4177-8401-9e9598ed9ef0" path="/var/lib/kubelet/pods/311279d9-d231-4177-8401-9e9598ed9ef0/volumes" Oct 30 14:43:30.402182 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:30.402165 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5265f061-4aa3-439e-ad80-dd952cd256d7" path="/var/lib/kubelet/pods/5265f061-4aa3-439e-ad80-dd952cd256d7/volumes" Oct 30 14:43:30.402544 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:30.402529 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d600402-b61d-48af-b02e-d38d9adf5b3b" path="/var/lib/kubelet/pods/9d600402-b61d-48af-b02e-d38d9adf5b3b/volumes" Oct 30 14:43:30.474287 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:30.474262 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjfzd\" (UniqueName: \"kubernetes.io/projected/a53fc31f-d9a7-45d9-8564-1e9b695c9bbf-kube-api-access-rjfzd\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:43:30.474287 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:30.474288 4982 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a53fc31f-d9a7-45d9-8564-1e9b695c9bbf-plugins-config-volume\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:43:30.736015 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:30.735083 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:30.890712 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:30.890674 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4be240-84fd4f496c-7z9qv"] Oct 30 14:43:30.890910 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:30.890882 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:30.891604 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:30.891579 4982 cpu_manager.go:414] "containerMap: removing container" podUID="fe8f23d0-2fda-4393-967d-c21cbbc8afe9" containerName="payload-pre-processing" Oct 30 14:43:30.891604 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:30.891605 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe8f23d0-2fda-4393-967d-c21cbbc8afe9" containerName="payload-pre-processing" Oct 30 14:43:30.891837 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:30.891626 4982 cpu_manager.go:414] "containerMap: removing container" podUID="a53fc31f-d9a7-45d9-8564-1e9b695c9bbf" containerName="payload-processing" Oct 30 14:43:30.891837 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:30.891634 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a53fc31f-d9a7-45d9-8564-1e9b695c9bbf" containerName="payload-processing" Oct 30 14:43:30.891837 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:30.891720 4982 cpu_manager.go:414] "containerMap: removing container" podUID="311279d9-d231-4177-8401-9e9598ed9ef0" containerName="maas-api" Oct 30 14:43:30.891837 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:30.891729 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="311279d9-d231-4177-8401-9e9598ed9ef0" containerName="maas-api" Oct 30 14:43:30.891837 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:30.891757 4982 cpu_manager.go:414] "containerMap: removing container" podUID="9d600402-b61d-48af-b02e-d38d9adf5b3b" containerName="payload-processing" Oct 30 14:43:30.891837 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:30.891765 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d600402-b61d-48af-b02e-d38d9adf5b3b" containerName="payload-processing" Oct 30 14:43:30.892162 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:30.891911 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d600402-b61d-48af-b02e-d38d9adf5b3b" containerName="payload-processing" Oct 30 14:43:30.892162 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:30.891930 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe8f23d0-2fda-4393-967d-c21cbbc8afe9" containerName="payload-pre-processing" Oct 30 14:43:30.892162 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:30.891943 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="a53fc31f-d9a7-45d9-8564-1e9b695c9bbf" containerName="payload-processing" Oct 30 14:43:30.892162 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:30.891964 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="311279d9-d231-4177-8401-9e9598ed9ef0" containerName="maas-api" Oct 30 14:43:30.897956 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:30.897928 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4be240-84fd4f496c-7z9qv" Oct 30 14:43:30.915972 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:30.915945 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-4be240\"" Oct 30 14:43:30.919093 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:30.919064 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4be240-84fd4f496c-7z9qv"] Oct 30 14:43:30.930658 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:30.930025 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27" Oct 30 14:43:30.936541 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:30.936507 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27" Oct 30 14:43:30.995711 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:30.992715 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb284e7d-7925-45d6-892e-622f297e32be-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4be240-84fd4f496c-7z9qv\" (UID: \"bb284e7d-7925-45d6-892e-622f297e32be\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4be240-84fd4f496c-7z9qv" Oct 30 14:43:30.995711 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:30.992900 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvt5n\" (UniqueName: \"kubernetes.io/projected/bb284e7d-7925-45d6-892e-622f297e32be-kube-api-access-pvt5n\") pod \"payload-pre-processing-e2e-ipp-b-4be240-84fd4f496c-7z9qv\" (UID: \"bb284e7d-7925-45d6-892e-622f297e32be\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4be240-84fd4f496c-7z9qv" Oct 30 14:43:30.998022 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:30.997988 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-4be240-dockercfg-hcvqg\"" Oct 30 14:43:31.106665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:31.093711 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4be240-58b8ddbdcb-n8p5d"] Oct 30 14:43:31.106665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:31.094677 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb284e7d-7925-45d6-892e-622f297e32be-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4be240-84fd4f496c-7z9qv\" (UID: \"bb284e7d-7925-45d6-892e-622f297e32be\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4be240-84fd4f496c-7z9qv" Oct 30 14:43:31.106665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:31.095773 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvt5n\" (UniqueName: \"kubernetes.io/projected/bb284e7d-7925-45d6-892e-622f297e32be-kube-api-access-pvt5n\") pod \"payload-pre-processing-e2e-ipp-b-4be240-84fd4f496c-7z9qv\" (UID: \"bb284e7d-7925-45d6-892e-622f297e32be\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4be240-84fd4f496c-7z9qv" Oct 30 14:43:31.106665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:31.095565 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb284e7d-7925-45d6-892e-622f297e32be-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4be240-84fd4f496c-7z9qv\" (UID: \"bb284e7d-7925-45d6-892e-622f297e32be\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4be240-84fd4f496c-7z9qv" Oct 30 14:43:31.106665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:31.102950 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4be240-58b8ddbdcb-n8p5d"] Oct 30 14:43:31.106665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:31.103049 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4be240-58b8ddbdcb-n8p5d" Oct 30 14:43:31.137847 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:31.136845 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3b95dd-5ccb7d956-4mkxk" event={"ID":"a53fc31f-d9a7-45d9-8564-1e9b695c9bbf","Type":"ContainerDied","Data":"10848c7a6d332b8aed6a2211925367237a8eb58015bdbc13b081114478a88286"} Oct 30 14:43:31.137847 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:31.136910 4982 scope.go:122] "RemoveContainer" containerID="c30a0521c45fb6929029a53be0fa71eeef9b5d563e600b3fb079364f3623f4e8" Oct 30 14:43:31.137847 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:31.137009 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3b95dd-5ccb7d956-4mkxk" Oct 30 14:43:31.155044 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:31.147743 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b95dd-858549854c-z49d4" Oct 30 14:43:31.155044 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:31.153889 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b95dd-858549854c-z49d4" event={"ID":"fe8f23d0-2fda-4393-967d-c21cbbc8afe9","Type":"ContainerDied","Data":"00dc3839f9ca016064d64885daafaa5d6a61ec114a1f8a14900dbdea60849ea0"} Oct 30 14:43:31.155044 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:31.153978 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27" Oct 30 14:43:31.159396 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:31.158731 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27" Oct 30 14:43:31.174197 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:31.171621 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvt5n\" (UniqueName: \"kubernetes.io/projected/bb284e7d-7925-45d6-892e-622f297e32be-kube-api-access-pvt5n\") pod \"payload-pre-processing-e2e-ipp-b-4be240-84fd4f496c-7z9qv\" (UID: \"bb284e7d-7925-45d6-892e-622f297e32be\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4be240-84fd4f496c-7z9qv" Oct 30 14:43:31.196459 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:31.196432 4982 scope.go:122] "RemoveContainer" containerID="54469c4a0663a338a6a17c7bc8093d3c756a622e43d43fb387efdfa902ad2e87" Oct 30 14:43:31.209307 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:31.209267 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3b95dd-858549854c-z49d4"] Oct 30 14:43:31.214569 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:31.214541 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3b95dd-858549854c-z49d4"] Oct 30 14:43:31.221504 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:31.221478 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4be240-84fd4f496c-7z9qv" Oct 30 14:43:31.259491 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:31.259431 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3b95dd-5ccb7d956-4mkxk"] Oct 30 14:43:31.269844 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:31.269790 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3b95dd-5ccb7d956-4mkxk"] Oct 30 14:43:31.305453 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:31.305423 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2d18448-7965-46d9-bc11-dd64afe3bad7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4be240-58b8ddbdcb-n8p5d\" (UID: \"e2d18448-7965-46d9-bc11-dd64afe3bad7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4be240-58b8ddbdcb-n8p5d" Oct 30 14:43:31.305602 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:31.305522 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq8bz\" (UniqueName: \"kubernetes.io/projected/e2d18448-7965-46d9-bc11-dd64afe3bad7-kube-api-access-pq8bz\") pod \"payload-processing-e2e-ipp-b-4be240-58b8ddbdcb-n8p5d\" (UID: \"e2d18448-7965-46d9-bc11-dd64afe3bad7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4be240-58b8ddbdcb-n8p5d" Oct 30 14:43:31.362966 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:31.362929 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:31.415742 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:31.407693 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2d18448-7965-46d9-bc11-dd64afe3bad7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4be240-58b8ddbdcb-n8p5d\" (UID: \"e2d18448-7965-46d9-bc11-dd64afe3bad7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4be240-58b8ddbdcb-n8p5d" Oct 30 14:43:31.415742 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:31.407840 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq8bz\" (UniqueName: \"kubernetes.io/projected/e2d18448-7965-46d9-bc11-dd64afe3bad7-kube-api-access-pq8bz\") pod \"payload-processing-e2e-ipp-b-4be240-58b8ddbdcb-n8p5d\" (UID: \"e2d18448-7965-46d9-bc11-dd64afe3bad7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4be240-58b8ddbdcb-n8p5d" Oct 30 14:43:31.415742 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:31.409105 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2d18448-7965-46d9-bc11-dd64afe3bad7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4be240-58b8ddbdcb-n8p5d\" (UID: \"e2d18448-7965-46d9-bc11-dd64afe3bad7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4be240-58b8ddbdcb-n8p5d" Oct 30 14:43:31.435813 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:31.435779 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq8bz\" (UniqueName: \"kubernetes.io/projected/e2d18448-7965-46d9-bc11-dd64afe3bad7-kube-api-access-pq8bz\") pod \"payload-processing-e2e-ipp-b-4be240-58b8ddbdcb-n8p5d\" (UID: \"e2d18448-7965-46d9-bc11-dd64afe3bad7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4be240-58b8ddbdcb-n8p5d" Oct 30 14:43:31.532838 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:31.532812 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4be240-84fd4f496c-7z9qv"] Oct 30 14:43:31.535176 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:43:31.535133 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb284e7d_7925_45d6_892e_622f297e32be.slice/crio-7902d29b33d06354568591335cfe50988e7e236d87e5b93d22007be780e4e55b WatchSource:0}: Error finding container 7902d29b33d06354568591335cfe50988e7e236d87e5b93d22007be780e4e55b: Status 404 returned error can't find the container with id 7902d29b33d06354568591335cfe50988e7e236d87e5b93d22007be780e4e55b Oct 30 14:43:31.729431 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:31.729402 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4be240-58b8ddbdcb-n8p5d" Oct 30 14:43:32.038027 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:32.037997 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4be240-58b8ddbdcb-n8p5d"] Oct 30 14:43:32.038626 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:43:32.038603 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2d18448_7965_46d9_bc11_dd64afe3bad7.slice/crio-ece2d5b0bcb5ea9ccb5a876ab24669b2f050ecf9c59ffcd31582aa08f1c8185a WatchSource:0}: Error finding container ece2d5b0bcb5ea9ccb5a876ab24669b2f050ecf9c59ffcd31582aa08f1c8185a: Status 404 returned error can't find the container with id ece2d5b0bcb5ea9ccb5a876ab24669b2f050ecf9c59ffcd31582aa08f1c8185a Oct 30 14:43:32.157573 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:32.157543 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4be240-84fd4f496c-7z9qv" event={"ID":"bb284e7d-7925-45d6-892e-622f297e32be","Type":"ContainerStarted","Data":"55aa5085e8d64cb01b208e53e9c668a10276aa2a23ff41d89f2c10c181fabe57"} Oct 30 14:43:32.158030 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:32.157598 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4be240-84fd4f496c-7z9qv" event={"ID":"bb284e7d-7925-45d6-892e-622f297e32be","Type":"ContainerStarted","Data":"7902d29b33d06354568591335cfe50988e7e236d87e5b93d22007be780e4e55b"} Oct 30 14:43:32.158030 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:32.157628 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4be240-84fd4f496c-7z9qv" Oct 30 14:43:32.159346 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:32.159317 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4be240-58b8ddbdcb-n8p5d" event={"ID":"e2d18448-7965-46d9-bc11-dd64afe3bad7","Type":"ContainerStarted","Data":"56e14dacc2dcc1dfc7ac5b261420d4a914e29551e4cde0ecf805e88fbf8135d6"} Oct 30 14:43:32.159482 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:32.159374 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-4be240-58b8ddbdcb-n8p5d" Oct 30 14:43:32.159482 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:32.159400 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4be240-58b8ddbdcb-n8p5d" event={"ID":"e2d18448-7965-46d9-bc11-dd64afe3bad7","Type":"ContainerStarted","Data":"ece2d5b0bcb5ea9ccb5a876ab24669b2f050ecf9c59ffcd31582aa08f1c8185a"} Oct 30 14:43:32.204195 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:32.204149 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4be240-84fd4f496c-7z9qv" podStartSLOduration=2.204136159 podStartE2EDuration="2.204136159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:43:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:43:32.20094929 +0000 UTC m=+3281.945083330" watchObservedRunningTime="2026-10-30 14:43:32.204136159 +0000 UTC m=+3281.948270200" Oct 30 14:43:32.245056 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:32.245008 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-4be240-58b8ddbdcb-n8p5d" podStartSLOduration=2.244993796 podStartE2EDuration="2.244993796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:43:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:43:32.243654009 +0000 UTC m=+3281.987788044" watchObservedRunningTime="2026-10-30 14:43:32.244993796 +0000 UTC m=+3281.989127839" Oct 30 14:43:32.402170 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:32.402136 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a53fc31f-d9a7-45d9-8564-1e9b695c9bbf" path="/var/lib/kubelet/pods/a53fc31f-d9a7-45d9-8564-1e9b695c9bbf/volumes" Oct 30 14:43:32.402670 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:32.402655 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe8f23d0-2fda-4393-967d-c21cbbc8afe9" path="/var/lib/kubelet/pods/fe8f23d0-2fda-4393-967d-c21cbbc8afe9/volumes" Oct 30 14:43:32.553367 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:32.553342 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:32.694867 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:32.694794 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:33.733966 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:33.733887 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:34.269609 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:34.269575 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:34.426187 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:34.426151 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:34.426563 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:34.426537 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-90b406-98b94bb7-4wb24"] Oct 30 14:43:34.433098 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:34.433072 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90b406-98b94bb7-4wb24" Oct 30 14:43:34.446904 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:34.446877 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57dd3cfb-792c-49c3-90c6-c58c34f9b5c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-90b406-98b94bb7-4wb24\" (UID: \"57dd3cfb-792c-49c3-90c6-c58c34f9b5c4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90b406-98b94bb7-4wb24" Oct 30 14:43:34.447052 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:34.446929 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h5md\" (UniqueName: \"kubernetes.io/projected/57dd3cfb-792c-49c3-90c6-c58c34f9b5c4-kube-api-access-6h5md\") pod \"payload-pre-processing-e2e-ipp-b-90b406-98b94bb7-4wb24\" (UID: \"57dd3cfb-792c-49c3-90c6-c58c34f9b5c4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90b406-98b94bb7-4wb24" Oct 30 14:43:34.451087 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:34.451061 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-90b406-98b94bb7-4wb24"] Oct 30 14:43:34.461683 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:34.460353 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-90b406-dockercfg-tb74p\"" Oct 30 14:43:34.461683 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:34.460464 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-90b406\"" Oct 30 14:43:34.576720 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:34.561027 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-90b406-548d97d98d-tjsf4"] Oct 30 14:43:34.576720 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:34.565404 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57dd3cfb-792c-49c3-90c6-c58c34f9b5c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-90b406-98b94bb7-4wb24\" (UID: \"57dd3cfb-792c-49c3-90c6-c58c34f9b5c4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90b406-98b94bb7-4wb24" Oct 30 14:43:34.576720 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:34.565525 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h5md\" (UniqueName: \"kubernetes.io/projected/57dd3cfb-792c-49c3-90c6-c58c34f9b5c4-kube-api-access-6h5md\") pod \"payload-pre-processing-e2e-ipp-b-90b406-98b94bb7-4wb24\" (UID: \"57dd3cfb-792c-49c3-90c6-c58c34f9b5c4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90b406-98b94bb7-4wb24" Oct 30 14:43:34.576720 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:34.566882 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57dd3cfb-792c-49c3-90c6-c58c34f9b5c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-90b406-98b94bb7-4wb24\" (UID: \"57dd3cfb-792c-49c3-90c6-c58c34f9b5c4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90b406-98b94bb7-4wb24" Oct 30 14:43:34.589906 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:34.589877 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-90b406-548d97d98d-tjsf4"] Oct 30 14:43:34.590209 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:34.590192 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-90b406-548d97d98d-tjsf4" Oct 30 14:43:34.671334 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:34.669678 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkg9s\" (UniqueName: \"kubernetes.io/projected/1ac28c86-bcac-404f-93e5-20f03557e069-kube-api-access-zkg9s\") pod \"payload-processing-e2e-ipp-b-90b406-548d97d98d-tjsf4\" (UID: \"1ac28c86-bcac-404f-93e5-20f03557e069\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-90b406-548d97d98d-tjsf4" Oct 30 14:43:34.671334 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:34.669788 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ac28c86-bcac-404f-93e5-20f03557e069-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-90b406-548d97d98d-tjsf4\" (UID: \"1ac28c86-bcac-404f-93e5-20f03557e069\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-90b406-548d97d98d-tjsf4" Oct 30 14:43:34.693366 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:34.693104 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h5md\" (UniqueName: \"kubernetes.io/projected/57dd3cfb-792c-49c3-90c6-c58c34f9b5c4-kube-api-access-6h5md\") pod \"payload-pre-processing-e2e-ipp-b-90b406-98b94bb7-4wb24\" (UID: \"57dd3cfb-792c-49c3-90c6-c58c34f9b5c4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90b406-98b94bb7-4wb24" Oct 30 14:43:34.747755 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:34.747723 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90b406-98b94bb7-4wb24" Oct 30 14:43:34.770933 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:34.770862 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkg9s\" (UniqueName: \"kubernetes.io/projected/1ac28c86-bcac-404f-93e5-20f03557e069-kube-api-access-zkg9s\") pod \"payload-processing-e2e-ipp-b-90b406-548d97d98d-tjsf4\" (UID: \"1ac28c86-bcac-404f-93e5-20f03557e069\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-90b406-548d97d98d-tjsf4" Oct 30 14:43:34.770933 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:34.770927 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ac28c86-bcac-404f-93e5-20f03557e069-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-90b406-548d97d98d-tjsf4\" (UID: \"1ac28c86-bcac-404f-93e5-20f03557e069\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-90b406-548d97d98d-tjsf4" Oct 30 14:43:34.771690 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:34.771575 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ac28c86-bcac-404f-93e5-20f03557e069-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-90b406-548d97d98d-tjsf4\" (UID: \"1ac28c86-bcac-404f-93e5-20f03557e069\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-90b406-548d97d98d-tjsf4" Oct 30 14:43:34.800842 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:34.800782 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkg9s\" (UniqueName: \"kubernetes.io/projected/1ac28c86-bcac-404f-93e5-20f03557e069-kube-api-access-zkg9s\") pod \"payload-processing-e2e-ipp-b-90b406-548d97d98d-tjsf4\" (UID: \"1ac28c86-bcac-404f-93e5-20f03557e069\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-90b406-548d97d98d-tjsf4" Oct 30 14:43:34.922633 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:34.922609 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2" Oct 30 14:43:34.974186 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:34.974156 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8c75528-f183-4379-841e-e413ac8d9c35-istio-token\" (UniqueName: \"kubernetes.io/projected/f8c75528-f183-4379-841e-e413ac8d9c35-istio-token\") pod \"f8c75528-f183-4379-841e-e413ac8d9c35\" (UID: \"f8c75528-f183-4379-841e-e413ac8d9c35\") " Oct 30 14:43:34.974369 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:34.974222 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8c75528-f183-4379-841e-e413ac8d9c35-kube-api-access-jg7px\" (UniqueName: \"kubernetes.io/projected/f8c75528-f183-4379-841e-e413ac8d9c35-kube-api-access-jg7px\") pod \"f8c75528-f183-4379-841e-e413ac8d9c35\" (UID: \"f8c75528-f183-4379-841e-e413ac8d9c35\") " Oct 30 14:43:34.974369 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:34.974291 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f8c75528-f183-4379-841e-e413ac8d9c35-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8c75528-f183-4379-841e-e413ac8d9c35-istio-podinfo\") pod \"f8c75528-f183-4379-841e-e413ac8d9c35\" (UID: \"f8c75528-f183-4379-841e-e413ac8d9c35\") " Oct 30 14:43:34.974369 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:34.974327 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8c75528-f183-4379-841e-e413ac8d9c35-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c75528-f183-4379-841e-e413ac8d9c35-workload-socket\") pod \"f8c75528-f183-4379-841e-e413ac8d9c35\" (UID: \"f8c75528-f183-4379-841e-e413ac8d9c35\") " Oct 30 14:43:34.974579 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:34.974397 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8c75528-f183-4379-841e-e413ac8d9c35-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8c75528-f183-4379-841e-e413ac8d9c35-workload-certs\") pod \"f8c75528-f183-4379-841e-e413ac8d9c35\" (UID: \"f8c75528-f183-4379-841e-e413ac8d9c35\") " Oct 30 14:43:34.974579 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:34.974424 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8c75528-f183-4379-841e-e413ac8d9c35-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c75528-f183-4379-841e-e413ac8d9c35-credential-socket\") pod \"f8c75528-f183-4379-841e-e413ac8d9c35\" (UID: \"f8c75528-f183-4379-841e-e413ac8d9c35\") " Oct 30 14:43:34.974579 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:34.974545 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8c75528-f183-4379-841e-e413ac8d9c35-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8c75528-f183-4379-841e-e413ac8d9c35-istiod-ca-cert\") pod \"f8c75528-f183-4379-841e-e413ac8d9c35\" (UID: \"f8c75528-f183-4379-841e-e413ac8d9c35\") " Oct 30 14:43:34.974794 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:34.974629 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8c75528-f183-4379-841e-e413ac8d9c35-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8c75528-f183-4379-841e-e413ac8d9c35-istio-data\") pod \"f8c75528-f183-4379-841e-e413ac8d9c35\" (UID: \"f8c75528-f183-4379-841e-e413ac8d9c35\") " Oct 30 14:43:34.974794 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:34.974688 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8c75528-f183-4379-841e-e413ac8d9c35-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8c75528-f183-4379-841e-e413ac8d9c35-istio-envoy\") pod \"f8c75528-f183-4379-841e-e413ac8d9c35\" (UID: \"f8c75528-f183-4379-841e-e413ac8d9c35\") " Oct 30 14:43:34.974794 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:34.974721 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8c75528-f183-4379-841e-e413ac8d9c35-workload-socket" pod "f8c75528-f183-4379-841e-e413ac8d9c35" (UID: "f8c75528-f183-4379-841e-e413ac8d9c35"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:43:34.975334 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:34.975311 4982 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c75528-f183-4379-841e-e413ac8d9c35-workload-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:43:34.975741 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:34.975693 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8c75528-f183-4379-841e-e413ac8d9c35-workload-certs" pod "f8c75528-f183-4379-841e-e413ac8d9c35" (UID: "f8c75528-f183-4379-841e-e413ac8d9c35"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:43:34.976140 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:34.976087 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8c75528-f183-4379-841e-e413ac8d9c35-istio-data" pod "f8c75528-f183-4379-841e-e413ac8d9c35" (UID: "f8c75528-f183-4379-841e-e413ac8d9c35"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:43:34.976271 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:34.975409 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8c75528-f183-4379-841e-e413ac8d9c35-istiod-ca-cert" pod "f8c75528-f183-4379-841e-e413ac8d9c35" (UID: "f8c75528-f183-4379-841e-e413ac8d9c35"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:43:34.977328 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:34.977301 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8c75528-f183-4379-841e-e413ac8d9c35-credential-socket" pod "f8c75528-f183-4379-841e-e413ac8d9c35" (UID: "f8c75528-f183-4379-841e-e413ac8d9c35"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:43:34.978416 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:34.978025 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8c75528-f183-4379-841e-e413ac8d9c35-kube-api-access-jg7px" pod "f8c75528-f183-4379-841e-e413ac8d9c35" (UID: "f8c75528-f183-4379-841e-e413ac8d9c35"). InnerVolumeSpecName "kube-api-access-jg7px". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:43:34.979827 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:34.979802 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8c75528-f183-4379-841e-e413ac8d9c35-istio-envoy" pod "f8c75528-f183-4379-841e-e413ac8d9c35" (UID: "f8c75528-f183-4379-841e-e413ac8d9c35"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:43:34.979989 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:34.979861 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8c75528-f183-4379-841e-e413ac8d9c35-istio-podinfo" pod "f8c75528-f183-4379-841e-e413ac8d9c35" (UID: "f8c75528-f183-4379-841e-e413ac8d9c35"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:43:34.979989 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:34.979886 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8c75528-f183-4379-841e-e413ac8d9c35-istio-token" pod "f8c75528-f183-4379-841e-e413ac8d9c35" (UID: "f8c75528-f183-4379-841e-e413ac8d9c35"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:43:35.035789 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:35.035761 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-90b406-548d97d98d-tjsf4" Oct 30 14:43:35.076421 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:35.076392 4982 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8c75528-f183-4379-841e-e413ac8d9c35-istio-data\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:43:35.076421 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:35.076419 4982 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8c75528-f183-4379-841e-e413ac8d9c35-istio-envoy\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:43:35.076421 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:35.076429 4982 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8c75528-f183-4379-841e-e413ac8d9c35-istio-token\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:43:35.076421 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:35.076438 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jg7px\" (UniqueName: \"kubernetes.io/projected/f8c75528-f183-4379-841e-e413ac8d9c35-kube-api-access-jg7px\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:43:35.077627 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:35.076448 4982 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8c75528-f183-4379-841e-e413ac8d9c35-istio-podinfo\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:43:35.077627 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:35.076457 4982 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8c75528-f183-4379-841e-e413ac8d9c35-workload-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:43:35.077627 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:35.076465 4982 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c75528-f183-4379-841e-e413ac8d9c35-credential-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:43:35.077627 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:35.076473 4982 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8c75528-f183-4379-841e-e413ac8d9c35-istiod-ca-cert\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:43:35.084179 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:35.084149 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-90b406-98b94bb7-4wb24"] Oct 30 14:43:35.084537 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:43:35.084511 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57dd3cfb_792c_49c3_90c6_c58c34f9b5c4.slice/crio-ad2047303164895dd1a4e5c34451a737505da973eabbb54ae95792ee47de2369 WatchSource:0}: Error finding container ad2047303164895dd1a4e5c34451a737505da973eabbb54ae95792ee47de2369: Status 404 returned error can't find the container with id ad2047303164895dd1a4e5c34451a737505da973eabbb54ae95792ee47de2369 Oct 30 14:43:35.189539 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:35.189504 4982 generic.go:356] "Generic (PLEG): container finished" podID="f8c75528-f183-4379-841e-e413ac8d9c35" containerID="c66d067be4b524c1eceba229db9944d94415e0dfeda879690abcb03637b658bb" exitCode=0 Oct 30 14:43:35.189712 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:35.189546 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2" event={"ID":"f8c75528-f183-4379-841e-e413ac8d9c35","Type":"ContainerDied","Data":"c66d067be4b524c1eceba229db9944d94415e0dfeda879690abcb03637b658bb"} Oct 30 14:43:35.189712 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:35.189566 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2" Oct 30 14:43:35.189712 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:35.189599 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2" event={"ID":"f8c75528-f183-4379-841e-e413ac8d9c35","Type":"ContainerDied","Data":"1184e2c852511f0baa2dc2ae1095a5e9da783022012781039ecddbaa99d7b9a9"} Oct 30 14:43:35.189712 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:35.189629 4982 scope.go:122] "RemoveContainer" containerID="c66d067be4b524c1eceba229db9944d94415e0dfeda879690abcb03637b658bb" Oct 30 14:43:35.191629 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:35.191485 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90b406-98b94bb7-4wb24" event={"ID":"57dd3cfb-792c-49c3-90c6-c58c34f9b5c4","Type":"ContainerStarted","Data":"ad2047303164895dd1a4e5c34451a737505da973eabbb54ae95792ee47de2369"} Oct 30 14:43:35.211736 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:35.211711 4982 scope.go:122] "RemoveContainer" containerID="c66d067be4b524c1eceba229db9944d94415e0dfeda879690abcb03637b658bb" Oct 30 14:43:35.212096 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:43:35.212070 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c66d067be4b524c1eceba229db9944d94415e0dfeda879690abcb03637b658bb\": container with ID starting with c66d067be4b524c1eceba229db9944d94415e0dfeda879690abcb03637b658bb not found: ID does not exist" containerID="c66d067be4b524c1eceba229db9944d94415e0dfeda879690abcb03637b658bb" Oct 30 14:43:35.212213 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:35.212105 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c66d067be4b524c1eceba229db9944d94415e0dfeda879690abcb03637b658bb"} err="failed to get container status \"c66d067be4b524c1eceba229db9944d94415e0dfeda879690abcb03637b658bb\": rpc error: code = NotFound desc = could not find container \"c66d067be4b524c1eceba229db9944d94415e0dfeda879690abcb03637b658bb\": container with ID starting with c66d067be4b524c1eceba229db9944d94415e0dfeda879690abcb03637b658bb not found: ID does not exist" Oct 30 14:43:35.249653 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:35.249622 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2"] Oct 30 14:43:35.257230 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:35.257207 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-c7c37b-openshift-default-6795b6cf59-zgfl2"] Oct 30 14:43:35.361327 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:35.361299 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-90b406-548d97d98d-tjsf4"] Oct 30 14:43:35.363411 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:43:35.363387 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ac28c86_bcac_404f_93e5_20f03557e069.slice/crio-430c1347b31e1590bd6d68a06d5a23a104c5d8de5050b583d54dd77bab245b86 WatchSource:0}: Error finding container 430c1347b31e1590bd6d68a06d5a23a104c5d8de5050b583d54dd77bab245b86: Status 404 returned error can't find the container with id 430c1347b31e1590bd6d68a06d5a23a104c5d8de5050b583d54dd77bab245b86 Oct 30 14:43:35.816718 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:35.816681 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:35.960354 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:35.960266 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:36.201823 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:36.201757 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-90b406-548d97d98d-tjsf4" event={"ID":"1ac28c86-bcac-404f-93e5-20f03557e069","Type":"ContainerStarted","Data":"6aa48cff6d0123a9a75ab6acca95c6bc5664e8833da3678de8713e3c25f8b74b"} Oct 30 14:43:36.201823 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:36.201820 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-90b406-548d97d98d-tjsf4" event={"ID":"1ac28c86-bcac-404f-93e5-20f03557e069","Type":"ContainerStarted","Data":"430c1347b31e1590bd6d68a06d5a23a104c5d8de5050b583d54dd77bab245b86"} Oct 30 14:43:36.202072 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:36.201995 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-90b406-548d97d98d-tjsf4" Oct 30 14:43:36.203447 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:36.203420 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90b406-98b94bb7-4wb24" event={"ID":"57dd3cfb-792c-49c3-90c6-c58c34f9b5c4","Type":"ContainerStarted","Data":"e02a140d414d4758240fc1675340cc50d573ce84a52c7ef50f90ddd6f2d0545b"} Oct 30 14:43:36.203574 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:36.203467 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90b406-98b94bb7-4wb24" Oct 30 14:43:36.249004 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:36.248915 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-90b406-548d97d98d-tjsf4" podStartSLOduration=2.248899657 podStartE2EDuration="2.248899657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:43:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:43:36.245003073 +0000 UTC m=+3285.989137116" watchObservedRunningTime="2026-10-30 14:43:36.248899657 +0000 UTC m=+3285.993033700" Oct 30 14:43:36.288620 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:36.288576 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90b406-98b94bb7-4wb24" podStartSLOduration=2.288564867 podStartE2EDuration="2.288564867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:43:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:43:36.288280843 +0000 UTC m=+3286.032414884" watchObservedRunningTime="2026-10-30 14:43:36.288564867 +0000 UTC m=+3286.032698908" Oct 30 14:43:36.404090 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:36.404058 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8c75528-f183-4379-841e-e413ac8d9c35" path="/var/lib/kubelet/pods/f8c75528-f183-4379-841e-e413ac8d9c35/volumes" Oct 30 14:43:37.764049 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:37.764014 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:37.912837 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:37.912803 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:38.041712 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:38.041261 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js"] Oct 30 14:43:38.041712 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:38.041557 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js" podUID="463ccf6c-67b5-45b6-970c-ee89fc36a12b" containerName="istio-proxy" containerID="cri-o://a86dba336c731ca5f3563df4f5bfa9420a6257540891a3daff4c54c4d1553a4e" gracePeriod=30 Oct 30 14:43:38.081006 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:38.080946 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:38.272503 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:38.272405 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:38.559601 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:38.559563 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-acefbb-8bwwk"] Oct 30 14:43:38.560232 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:38.560216 4982 cpu_manager.go:414] "containerMap: removing container" podUID="f8c75528-f183-4379-841e-e413ac8d9c35" containerName="istio-proxy" Oct 30 14:43:38.560284 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:38.560233 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8c75528-f183-4379-841e-e413ac8d9c35" containerName="istio-proxy" Oct 30 14:43:38.560359 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:38.560349 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8c75528-f183-4379-841e-e413ac8d9c35" containerName="istio-proxy" Oct 30 14:43:38.565194 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:38.565174 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-acefbb-8bwwk" Oct 30 14:43:38.589572 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:38.589544 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-acefbb-8bwwk"] Oct 30 14:43:38.597403 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:38.597381 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:43:38.597572 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:38.597409 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:43:38.647198 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:38.647173 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06340c7e-9397-4280-bd4f-6c540ca20f73-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-acefbb-8bwwk\" (UID: \"06340c7e-9397-4280-bd4f-6c540ca20f73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-acefbb-8bwwk" Oct 30 14:43:38.647324 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:38.647286 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sr9nw\" (UniqueName: \"kubernetes.io/projected/06340c7e-9397-4280-bd4f-6c540ca20f73-kube-api-access-sr9nw\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-acefbb-8bwwk\" (UID: \"06340c7e-9397-4280-bd4f-6c540ca20f73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-acefbb-8bwwk" Oct 30 14:43:38.647371 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:38.647356 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06340c7e-9397-4280-bd4f-6c540ca20f73-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-acefbb-8bwwk\" (UID: \"06340c7e-9397-4280-bd4f-6c540ca20f73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-acefbb-8bwwk" Oct 30 14:43:38.748536 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:38.748504 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06340c7e-9397-4280-bd4f-6c540ca20f73-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-acefbb-8bwwk\" (UID: \"06340c7e-9397-4280-bd4f-6c540ca20f73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-acefbb-8bwwk" Oct 30 14:43:38.748682 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:38.748597 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr9nw\" (UniqueName: \"kubernetes.io/projected/06340c7e-9397-4280-bd4f-6c540ca20f73-kube-api-access-sr9nw\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-acefbb-8bwwk\" (UID: \"06340c7e-9397-4280-bd4f-6c540ca20f73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-acefbb-8bwwk" Oct 30 14:43:38.748729 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:38.748680 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06340c7e-9397-4280-bd4f-6c540ca20f73-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-acefbb-8bwwk\" (UID: \"06340c7e-9397-4280-bd4f-6c540ca20f73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-acefbb-8bwwk" Oct 30 14:43:38.749161 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:38.749140 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06340c7e-9397-4280-bd4f-6c540ca20f73-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-acefbb-8bwwk\" (UID: \"06340c7e-9397-4280-bd4f-6c540ca20f73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-acefbb-8bwwk" Oct 30 14:43:38.751088 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:38.751064 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06340c7e-9397-4280-bd4f-6c540ca20f73-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-acefbb-8bwwk\" (UID: \"06340c7e-9397-4280-bd4f-6c540ca20f73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-acefbb-8bwwk" Oct 30 14:43:38.775359 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:38.775336 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sr9nw\" (UniqueName: \"kubernetes.io/projected/06340c7e-9397-4280-bd4f-6c540ca20f73-kube-api-access-sr9nw\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-acefbb-8bwwk\" (UID: \"06340c7e-9397-4280-bd4f-6c540ca20f73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-acefbb-8bwwk" Oct 30 14:43:38.879179 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:38.879125 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-acefbb-8bwwk" Oct 30 14:43:39.194126 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:43:39.194095 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06340c7e_9397_4280_bd4f_6c540ca20f73.slice/crio-9c7fa6859473428cb5055ff7e7f5c38e7abab98e2a6bbaf453653128bb5de3e9 WatchSource:0}: Error finding container 9c7fa6859473428cb5055ff7e7f5c38e7abab98e2a6bbaf453653128bb5de3e9: Status 404 returned error can't find the container with id 9c7fa6859473428cb5055ff7e7f5c38e7abab98e2a6bbaf453653128bb5de3e9 Oct 30 14:43:39.195004 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:39.194973 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-acefbb-8bwwk"] Oct 30 14:43:39.250536 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:39.250512 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-acefbb-8bwwk" event={"ID":"06340c7e-9397-4280-bd4f-6c540ca20f73","Type":"ContainerStarted","Data":"9c7fa6859473428cb5055ff7e7f5c38e7abab98e2a6bbaf453653128bb5de3e9"} Oct 30 14:43:40.051927 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:40.051889 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:40.073430 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:40.073390 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e50e63-df24z"] Oct 30 14:43:40.078259 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:40.078229 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e50e63-df24z" Oct 30 14:43:40.112384 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:40.112340 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e50e63-df24z"] Oct 30 14:43:40.169831 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:40.169801 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e4c87c4-5edb-4292-9201-8ae74f5e3e60-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-e50e63-df24z\" (UID: \"2e4c87c4-5edb-4292-9201-8ae74f5e3e60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e50e63-df24z" Oct 30 14:43:40.170007 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:40.169979 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdcdj\" (UniqueName: \"kubernetes.io/projected/2e4c87c4-5edb-4292-9201-8ae74f5e3e60-kube-api-access-sdcdj\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-e50e63-df24z\" (UID: \"2e4c87c4-5edb-4292-9201-8ae74f5e3e60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e50e63-df24z" Oct 30 14:43:40.170104 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:40.170088 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e4c87c4-5edb-4292-9201-8ae74f5e3e60-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-e50e63-df24z\" (UID: \"2e4c87c4-5edb-4292-9201-8ae74f5e3e60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e50e63-df24z" Oct 30 14:43:40.210796 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:40.210760 4982 patch_prober.go:29] interesting pod/e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 14:43:40.210912 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:40.210816 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js" podUID="463ccf6c-67b5-45b6-970c-ee89fc36a12b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 14:43:40.258913 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:40.258882 4982 generic.go:356] "Generic (PLEG): container finished" podID="06340c7e-9397-4280-bd4f-6c540ca20f73" containerID="b0a4f021c4c7a943ae2d1067b7fdb54688a2a26e00010134a39bbe883bcc33c8" exitCode=0 Oct 30 14:43:40.259020 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:40.258963 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-acefbb-8bwwk" event={"ID":"06340c7e-9397-4280-bd4f-6c540ca20f73","Type":"ContainerDied","Data":"b0a4f021c4c7a943ae2d1067b7fdb54688a2a26e00010134a39bbe883bcc33c8"} Oct 30 14:43:40.271065 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:40.271032 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e4c87c4-5edb-4292-9201-8ae74f5e3e60-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-e50e63-df24z\" (UID: \"2e4c87c4-5edb-4292-9201-8ae74f5e3e60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e50e63-df24z" Oct 30 14:43:40.271179 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:40.271141 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdcdj\" (UniqueName: \"kubernetes.io/projected/2e4c87c4-5edb-4292-9201-8ae74f5e3e60-kube-api-access-sdcdj\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-e50e63-df24z\" (UID: \"2e4c87c4-5edb-4292-9201-8ae74f5e3e60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e50e63-df24z" Oct 30 14:43:40.271222 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:40.271208 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e4c87c4-5edb-4292-9201-8ae74f5e3e60-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-e50e63-df24z\" (UID: \"2e4c87c4-5edb-4292-9201-8ae74f5e3e60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e50e63-df24z" Oct 30 14:43:40.271728 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:40.271712 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e4c87c4-5edb-4292-9201-8ae74f5e3e60-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-e50e63-df24z\" (UID: \"2e4c87c4-5edb-4292-9201-8ae74f5e3e60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e50e63-df24z" Oct 30 14:43:40.273782 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:40.273762 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e4c87c4-5edb-4292-9201-8ae74f5e3e60-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-e50e63-df24z\" (UID: \"2e4c87c4-5edb-4292-9201-8ae74f5e3e60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e50e63-df24z" Oct 30 14:43:40.297741 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:40.297717 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdcdj\" (UniqueName: \"kubernetes.io/projected/2e4c87c4-5edb-4292-9201-8ae74f5e3e60-kube-api-access-sdcdj\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-e50e63-df24z\" (UID: \"2e4c87c4-5edb-4292-9201-8ae74f5e3e60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e50e63-df24z" Oct 30 14:43:40.476711 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:40.476681 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e50e63-df24z" Oct 30 14:43:40.798368 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:43:40.798334 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e4c87c4_5edb_4292_9201_8ae74f5e3e60.slice/crio-fc067498506d5ea9253e9ca0baa873589c683721b6dd3b6ac0c9442e6fa6eb84 WatchSource:0}: Error finding container fc067498506d5ea9253e9ca0baa873589c683721b6dd3b6ac0c9442e6fa6eb84: Status 404 returned error can't find the container with id fc067498506d5ea9253e9ca0baa873589c683721b6dd3b6ac0c9442e6fa6eb84 Oct 30 14:43:40.798835 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:40.798807 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e50e63-df24z"] Oct 30 14:43:41.267263 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:41.267227 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e50e63-df24z" event={"ID":"2e4c87c4-5edb-4292-9201-8ae74f5e3e60","Type":"ContainerStarted","Data":"82359b46efd57ac7d58e57ca35221ea9e544965124dda220b3084916fe0a3c22"} Oct 30 14:43:41.267756 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:41.267308 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e50e63-df24z" event={"ID":"2e4c87c4-5edb-4292-9201-8ae74f5e3e60","Type":"ContainerStarted","Data":"fc067498506d5ea9253e9ca0baa873589c683721b6dd3b6ac0c9442e6fa6eb84"} Oct 30 14:43:41.314453 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:41.314394 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e50e63-df24z" podStartSLOduration=1.314379028 podStartE2EDuration="1.314379028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:43:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:43:41.312159235 +0000 UTC m=+3291.056293278" watchObservedRunningTime="2026-10-30 14:43:41.314379028 +0000 UTC m=+3291.058513129" Oct 30 14:43:41.475169 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:41.475143 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-acefbb-8bwwk" Oct 30 14:43:41.588152 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:41.588123 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06340c7e-9397-4280-bd4f-6c540ca20f73-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06340c7e-9397-4280-bd4f-6c540ca20f73-maas-api-openshift-ca\") pod \"06340c7e-9397-4280-bd4f-6c540ca20f73\" (UID: \"06340c7e-9397-4280-bd4f-6c540ca20f73\") " Oct 30 14:43:41.588293 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:41.588174 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06340c7e-9397-4280-bd4f-6c540ca20f73-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06340c7e-9397-4280-bd4f-6c540ca20f73-maas-api-xks-ca\") pod \"06340c7e-9397-4280-bd4f-6c540ca20f73\" (UID: \"06340c7e-9397-4280-bd4f-6c540ca20f73\") " Oct 30 14:43:41.588365 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:41.588317 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06340c7e-9397-4280-bd4f-6c540ca20f73-kube-api-access-sr9nw\" (UniqueName: \"kubernetes.io/projected/06340c7e-9397-4280-bd4f-6c540ca20f73-kube-api-access-sr9nw\") pod \"06340c7e-9397-4280-bd4f-6c540ca20f73\" (UID: \"06340c7e-9397-4280-bd4f-6c540ca20f73\") " Oct 30 14:43:41.588529 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:41.588506 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06340c7e-9397-4280-bd4f-6c540ca20f73-maas-api-openshift-ca" pod "06340c7e-9397-4280-bd4f-6c540ca20f73" (UID: "06340c7e-9397-4280-bd4f-6c540ca20f73"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:43:41.589098 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:41.589067 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06340c7e-9397-4280-bd4f-6c540ca20f73-maas-api-openshift-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:43:41.590392 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:41.590364 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06340c7e-9397-4280-bd4f-6c540ca20f73-kube-api-access-sr9nw" pod "06340c7e-9397-4280-bd4f-6c540ca20f73" (UID: "06340c7e-9397-4280-bd4f-6c540ca20f73"). InnerVolumeSpecName "kube-api-access-sr9nw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:43:41.590564 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:41.590539 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06340c7e-9397-4280-bd4f-6c540ca20f73-maas-api-xks-ca" pod "06340c7e-9397-4280-bd4f-6c540ca20f73" (UID: "06340c7e-9397-4280-bd4f-6c540ca20f73"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:43:41.691349 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:41.691322 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sr9nw\" (UniqueName: \"kubernetes.io/projected/06340c7e-9397-4280-bd4f-6c540ca20f73-kube-api-access-sr9nw\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:43:41.691455 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:41.691351 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06340c7e-9397-4280-bd4f-6c540ca20f73-maas-api-xks-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:43:42.065387 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:42.065355 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:42.276754 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:42.276721 4982 generic.go:356] "Generic (PLEG): container finished" podID="2e4c87c4-5edb-4292-9201-8ae74f5e3e60" containerID="82359b46efd57ac7d58e57ca35221ea9e544965124dda220b3084916fe0a3c22" exitCode=0 Oct 30 14:43:42.277107 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:42.276800 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e50e63-df24z" event={"ID":"2e4c87c4-5edb-4292-9201-8ae74f5e3e60","Type":"ContainerDied","Data":"82359b46efd57ac7d58e57ca35221ea9e544965124dda220b3084916fe0a3c22"} Oct 30 14:43:42.278401 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:42.278381 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-acefbb-8bwwk" event={"ID":"06340c7e-9397-4280-bd4f-6c540ca20f73","Type":"ContainerDied","Data":"9c7fa6859473428cb5055ff7e7f5c38e7abab98e2a6bbaf453653128bb5de3e9"} Oct 30 14:43:42.278523 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:42.278404 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-acefbb-8bwwk" Oct 30 14:43:42.278523 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:42.278415 4982 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c7fa6859473428cb5055ff7e7f5c38e7abab98e2a6bbaf453653128bb5de3e9" Oct 30 14:43:43.170671 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:43.170613 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4be240-84fd4f496c-7z9qv" Oct 30 14:43:43.170782 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:43.170706 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-4be240-58b8ddbdcb-n8p5d" Oct 30 14:43:43.298936 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:43.298099 4982 generic.go:356] "Generic (PLEG): container finished" podID="463ccf6c-67b5-45b6-970c-ee89fc36a12b" containerID="a86dba336c731ca5f3563df4f5bfa9420a6257540891a3daff4c54c4d1553a4e" exitCode=0 Oct 30 14:43:43.298936 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:43.298168 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js" event={"ID":"463ccf6c-67b5-45b6-970c-ee89fc36a12b","Type":"ContainerDied","Data":"a86dba336c731ca5f3563df4f5bfa9420a6257540891a3daff4c54c4d1553a4e"} Oct 30 14:43:43.298936 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:43.298236 4982 scope.go:122] "RemoveContainer" containerID="a99ebb3920b7a686f4de743c079efb06dc7326eb8abf9ca22b6c033fae601b79" Oct 30 14:43:43.393778 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:43.390287 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js" Oct 30 14:43:43.517520 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:43.517493 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/463ccf6c-67b5-45b6-970c-ee89fc36a12b-kube-api-access-6jfc5\" (UniqueName: \"kubernetes.io/projected/463ccf6c-67b5-45b6-970c-ee89fc36a12b-kube-api-access-6jfc5\") pod \"463ccf6c-67b5-45b6-970c-ee89fc36a12b\" (UID: \"463ccf6c-67b5-45b6-970c-ee89fc36a12b\") " Oct 30 14:43:43.517689 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:43.517603 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/463ccf6c-67b5-45b6-970c-ee89fc36a12b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/463ccf6c-67b5-45b6-970c-ee89fc36a12b-istio-podinfo\") pod \"463ccf6c-67b5-45b6-970c-ee89fc36a12b\" (UID: \"463ccf6c-67b5-45b6-970c-ee89fc36a12b\") " Oct 30 14:43:43.517760 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:43.517735 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/463ccf6c-67b5-45b6-970c-ee89fc36a12b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/463ccf6c-67b5-45b6-970c-ee89fc36a12b-istio-envoy\") pod \"463ccf6c-67b5-45b6-970c-ee89fc36a12b\" (UID: \"463ccf6c-67b5-45b6-970c-ee89fc36a12b\") " Oct 30 14:43:43.517816 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:43.517785 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/463ccf6c-67b5-45b6-970c-ee89fc36a12b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/463ccf6c-67b5-45b6-970c-ee89fc36a12b-istiod-ca-cert\") pod \"463ccf6c-67b5-45b6-970c-ee89fc36a12b\" (UID: \"463ccf6c-67b5-45b6-970c-ee89fc36a12b\") " Oct 30 14:43:43.517916 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:43.517899 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/463ccf6c-67b5-45b6-970c-ee89fc36a12b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/463ccf6c-67b5-45b6-970c-ee89fc36a12b-credential-socket\") pod \"463ccf6c-67b5-45b6-970c-ee89fc36a12b\" (UID: \"463ccf6c-67b5-45b6-970c-ee89fc36a12b\") " Oct 30 14:43:43.517975 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:43.517943 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/463ccf6c-67b5-45b6-970c-ee89fc36a12b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/463ccf6c-67b5-45b6-970c-ee89fc36a12b-workload-certs\") pod \"463ccf6c-67b5-45b6-970c-ee89fc36a12b\" (UID: \"463ccf6c-67b5-45b6-970c-ee89fc36a12b\") " Oct 30 14:43:43.518033 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:43.517975 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/463ccf6c-67b5-45b6-970c-ee89fc36a12b-istio-token\" (UniqueName: \"kubernetes.io/projected/463ccf6c-67b5-45b6-970c-ee89fc36a12b-istio-token\") pod \"463ccf6c-67b5-45b6-970c-ee89fc36a12b\" (UID: \"463ccf6c-67b5-45b6-970c-ee89fc36a12b\") " Oct 30 14:43:43.518087 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:43.518067 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/463ccf6c-67b5-45b6-970c-ee89fc36a12b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/463ccf6c-67b5-45b6-970c-ee89fc36a12b-istio-data\") pod \"463ccf6c-67b5-45b6-970c-ee89fc36a12b\" (UID: \"463ccf6c-67b5-45b6-970c-ee89fc36a12b\") " Oct 30 14:43:43.518144 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:43.518109 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/463ccf6c-67b5-45b6-970c-ee89fc36a12b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/463ccf6c-67b5-45b6-970c-ee89fc36a12b-workload-socket\") pod \"463ccf6c-67b5-45b6-970c-ee89fc36a12b\" (UID: \"463ccf6c-67b5-45b6-970c-ee89fc36a12b\") " Oct 30 14:43:43.518303 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:43.518271 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/463ccf6c-67b5-45b6-970c-ee89fc36a12b-credential-socket" pod "463ccf6c-67b5-45b6-970c-ee89fc36a12b" (UID: "463ccf6c-67b5-45b6-970c-ee89fc36a12b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:43:43.518380 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:43.518286 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/463ccf6c-67b5-45b6-970c-ee89fc36a12b-workload-certs" pod "463ccf6c-67b5-45b6-970c-ee89fc36a12b" (UID: "463ccf6c-67b5-45b6-970c-ee89fc36a12b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:43:43.518496 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:43.518474 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/463ccf6c-67b5-45b6-970c-ee89fc36a12b-istio-data" pod "463ccf6c-67b5-45b6-970c-ee89fc36a12b" (UID: "463ccf6c-67b5-45b6-970c-ee89fc36a12b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:43:43.518592 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:43.518568 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/463ccf6c-67b5-45b6-970c-ee89fc36a12b-workload-socket" pod "463ccf6c-67b5-45b6-970c-ee89fc36a12b" (UID: "463ccf6c-67b5-45b6-970c-ee89fc36a12b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:43:43.518694 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:43.518631 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/463ccf6c-67b5-45b6-970c-ee89fc36a12b-istiod-ca-cert" pod "463ccf6c-67b5-45b6-970c-ee89fc36a12b" (UID: "463ccf6c-67b5-45b6-970c-ee89fc36a12b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:43:43.518783 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:43.518764 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e50e63-df24z" Oct 30 14:43:43.518935 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:43.518915 4982 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/463ccf6c-67b5-45b6-970c-ee89fc36a12b-istiod-ca-cert\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:43:43.519016 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:43.518941 4982 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/463ccf6c-67b5-45b6-970c-ee89fc36a12b-credential-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:43:43.519016 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:43.518958 4982 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/463ccf6c-67b5-45b6-970c-ee89fc36a12b-workload-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:43:43.519016 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:43.518971 4982 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/463ccf6c-67b5-45b6-970c-ee89fc36a12b-istio-data\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:43:43.519016 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:43.518983 4982 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/463ccf6c-67b5-45b6-970c-ee89fc36a12b-workload-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:43:43.520070 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:43.520035 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/463ccf6c-67b5-45b6-970c-ee89fc36a12b-istio-envoy" pod "463ccf6c-67b5-45b6-970c-ee89fc36a12b" (UID: "463ccf6c-67b5-45b6-970c-ee89fc36a12b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:43:43.520893 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:43.520866 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/463ccf6c-67b5-45b6-970c-ee89fc36a12b-istio-token" pod "463ccf6c-67b5-45b6-970c-ee89fc36a12b" (UID: "463ccf6c-67b5-45b6-970c-ee89fc36a12b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:43:43.521094 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:43.521078 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/463ccf6c-67b5-45b6-970c-ee89fc36a12b-kube-api-access-6jfc5" pod "463ccf6c-67b5-45b6-970c-ee89fc36a12b" (UID: "463ccf6c-67b5-45b6-970c-ee89fc36a12b"). InnerVolumeSpecName "kube-api-access-6jfc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:43:43.521306 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:43.521271 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/463ccf6c-67b5-45b6-970c-ee89fc36a12b-istio-podinfo" pod "463ccf6c-67b5-45b6-970c-ee89fc36a12b" (UID: "463ccf6c-67b5-45b6-970c-ee89fc36a12b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:43:43.620256 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:43.620169 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e4c87c4-5edb-4292-9201-8ae74f5e3e60-kube-api-access-sdcdj\" (UniqueName: \"kubernetes.io/projected/2e4c87c4-5edb-4292-9201-8ae74f5e3e60-kube-api-access-sdcdj\") pod \"2e4c87c4-5edb-4292-9201-8ae74f5e3e60\" (UID: \"2e4c87c4-5edb-4292-9201-8ae74f5e3e60\") " Oct 30 14:43:43.620256 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:43.620221 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e4c87c4-5edb-4292-9201-8ae74f5e3e60-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e4c87c4-5edb-4292-9201-8ae74f5e3e60-maas-api-xks-ca\") pod \"2e4c87c4-5edb-4292-9201-8ae74f5e3e60\" (UID: \"2e4c87c4-5edb-4292-9201-8ae74f5e3e60\") " Oct 30 14:43:43.620491 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:43.620340 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e4c87c4-5edb-4292-9201-8ae74f5e3e60-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e4c87c4-5edb-4292-9201-8ae74f5e3e60-maas-api-openshift-ca\") pod \"2e4c87c4-5edb-4292-9201-8ae74f5e3e60\" (UID: \"2e4c87c4-5edb-4292-9201-8ae74f5e3e60\") " Oct 30 14:43:43.621036 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:43.620905 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jfc5\" (UniqueName: \"kubernetes.io/projected/463ccf6c-67b5-45b6-970c-ee89fc36a12b-kube-api-access-6jfc5\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:43:43.621036 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:43.620930 4982 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/463ccf6c-67b5-45b6-970c-ee89fc36a12b-istio-podinfo\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:43:43.621036 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:43.620945 4982 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/463ccf6c-67b5-45b6-970c-ee89fc36a12b-istio-envoy\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:43:43.621036 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:43.620959 4982 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/463ccf6c-67b5-45b6-970c-ee89fc36a12b-istio-token\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:43:43.621669 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:43.621599 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e4c87c4-5edb-4292-9201-8ae74f5e3e60-maas-api-openshift-ca" pod "2e4c87c4-5edb-4292-9201-8ae74f5e3e60" (UID: "2e4c87c4-5edb-4292-9201-8ae74f5e3e60"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:43:43.624186 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:43.624151 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e4c87c4-5edb-4292-9201-8ae74f5e3e60-kube-api-access-sdcdj" pod "2e4c87c4-5edb-4292-9201-8ae74f5e3e60" (UID: "2e4c87c4-5edb-4292-9201-8ae74f5e3e60"). InnerVolumeSpecName "kube-api-access-sdcdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:43:43.624431 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:43.624401 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e4c87c4-5edb-4292-9201-8ae74f5e3e60-maas-api-xks-ca" pod "2e4c87c4-5edb-4292-9201-8ae74f5e3e60" (UID: "2e4c87c4-5edb-4292-9201-8ae74f5e3e60"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:43:43.722723 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:43.722680 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdcdj\" (UniqueName: \"kubernetes.io/projected/2e4c87c4-5edb-4292-9201-8ae74f5e3e60-kube-api-access-sdcdj\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:43:43.722723 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:43.722719 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e4c87c4-5edb-4292-9201-8ae74f5e3e60-maas-api-xks-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:43:43.722948 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:43.722733 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e4c87c4-5edb-4292-9201-8ae74f5e3e60-maas-api-openshift-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:43:44.050008 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:44.049927 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:44.230548 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:44.230510 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx"] Oct 30 14:43:44.230833 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:44.230781 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx" podUID="13d8ee0f-f4c1-4630-85d7-5fdee28f3187" containerName="istio-proxy" containerID="cri-o://c368a4e6fd60dcada4f4f3248ba1737ceb64c48779562ea3b450cd6162b359c3" gracePeriod=30 Oct 30 14:43:44.308029 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:44.308004 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js" Oct 30 14:43:44.308452 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:44.308008 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js" event={"ID":"463ccf6c-67b5-45b6-970c-ee89fc36a12b","Type":"ContainerDied","Data":"1c701fdec2057027418749a81821d572a160a78556de43709cb1d3f0cdf38ae3"} Oct 30 14:43:44.308452 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:44.308141 4982 scope.go:122] "RemoveContainer" containerID="a86dba336c731ca5f3563df4f5bfa9420a6257540891a3daff4c54c4d1553a4e" Oct 30 14:43:44.310429 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:44.310396 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e50e63-df24z" event={"ID":"2e4c87c4-5edb-4292-9201-8ae74f5e3e60","Type":"ContainerDied","Data":"fc067498506d5ea9253e9ca0baa873589c683721b6dd3b6ac0c9442e6fa6eb84"} Oct 30 14:43:44.310600 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:44.310535 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e50e63-df24z" Oct 30 14:43:44.324694 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:44.323324 4982 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc067498506d5ea9253e9ca0baa873589c683721b6dd3b6ac0c9442e6fa6eb84" Oct 30 14:43:44.389578 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:44.389547 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js"] Oct 30 14:43:44.402923 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:44.402889 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-460b0b-openshift-default-596d6cdf7-x67js"] Oct 30 14:43:45.864168 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:45.864114 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:46.403132 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:46.403098 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="463ccf6c-67b5-45b6-970c-ee89fc36a12b" path="/var/lib/kubelet/pods/463ccf6c-67b5-45b6-970c-ee89fc36a12b/volumes" Oct 30 14:43:47.091479 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:47.091438 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:47.231870 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:47.231838 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-90b406-548d97d98d-tjsf4" Oct 30 14:43:47.241975 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:47.241954 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90b406-98b94bb7-4wb24" Oct 30 14:43:47.738910 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:47.738875 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:49.206278 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:49.206223 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:49.350101 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:49.350063 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:49.354827 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:49.354803 4982 generic.go:356] "Generic (PLEG): container finished" podID="13d8ee0f-f4c1-4630-85d7-5fdee28f3187" containerID="c368a4e6fd60dcada4f4f3248ba1737ceb64c48779562ea3b450cd6162b359c3" exitCode=0 Oct 30 14:43:49.354945 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:49.354885 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx" event={"ID":"13d8ee0f-f4c1-4630-85d7-5fdee28f3187","Type":"ContainerDied","Data":"c368a4e6fd60dcada4f4f3248ba1737ceb64c48779562ea3b450cd6162b359c3"} Oct 30 14:43:49.530789 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:49.530765 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx" Oct 30 14:43:49.644010 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:49.643978 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:49.695863 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:49.695836 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13d8ee0f-f4c1-4630-85d7-5fdee28f3187-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13d8ee0f-f4c1-4630-85d7-5fdee28f3187-workload-socket\") pod \"13d8ee0f-f4c1-4630-85d7-5fdee28f3187\" (UID: \"13d8ee0f-f4c1-4630-85d7-5fdee28f3187\") " Oct 30 14:43:49.696008 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:49.695874 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13d8ee0f-f4c1-4630-85d7-5fdee28f3187-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13d8ee0f-f4c1-4630-85d7-5fdee28f3187-istiod-ca-cert\") pod \"13d8ee0f-f4c1-4630-85d7-5fdee28f3187\" (UID: \"13d8ee0f-f4c1-4630-85d7-5fdee28f3187\") " Oct 30 14:43:49.696008 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:49.695908 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13d8ee0f-f4c1-4630-85d7-5fdee28f3187-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13d8ee0f-f4c1-4630-85d7-5fdee28f3187-workload-certs\") pod \"13d8ee0f-f4c1-4630-85d7-5fdee28f3187\" (UID: \"13d8ee0f-f4c1-4630-85d7-5fdee28f3187\") " Oct 30 14:43:49.696008 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:49.695939 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13d8ee0f-f4c1-4630-85d7-5fdee28f3187-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13d8ee0f-f4c1-4630-85d7-5fdee28f3187-istio-data\") pod \"13d8ee0f-f4c1-4630-85d7-5fdee28f3187\" (UID: \"13d8ee0f-f4c1-4630-85d7-5fdee28f3187\") " Oct 30 14:43:49.696008 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:49.695979 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13d8ee0f-f4c1-4630-85d7-5fdee28f3187-kube-api-access-r8q65\" (UniqueName: \"kubernetes.io/projected/13d8ee0f-f4c1-4630-85d7-5fdee28f3187-kube-api-access-r8q65\") pod \"13d8ee0f-f4c1-4630-85d7-5fdee28f3187\" (UID: \"13d8ee0f-f4c1-4630-85d7-5fdee28f3187\") " Oct 30 14:43:49.696250 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:49.696023 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/13d8ee0f-f4c1-4630-85d7-5fdee28f3187-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13d8ee0f-f4c1-4630-85d7-5fdee28f3187-istio-podinfo\") pod \"13d8ee0f-f4c1-4630-85d7-5fdee28f3187\" (UID: \"13d8ee0f-f4c1-4630-85d7-5fdee28f3187\") " Oct 30 14:43:49.696250 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:49.696071 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13d8ee0f-f4c1-4630-85d7-5fdee28f3187-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13d8ee0f-f4c1-4630-85d7-5fdee28f3187-credential-socket\") pod \"13d8ee0f-f4c1-4630-85d7-5fdee28f3187\" (UID: \"13d8ee0f-f4c1-4630-85d7-5fdee28f3187\") " Oct 30 14:43:49.696250 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:49.696125 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13d8ee0f-f4c1-4630-85d7-5fdee28f3187-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13d8ee0f-f4c1-4630-85d7-5fdee28f3187-istio-envoy\") pod \"13d8ee0f-f4c1-4630-85d7-5fdee28f3187\" (UID: \"13d8ee0f-f4c1-4630-85d7-5fdee28f3187\") " Oct 30 14:43:49.696250 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:49.696139 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13d8ee0f-f4c1-4630-85d7-5fdee28f3187-workload-socket" pod "13d8ee0f-f4c1-4630-85d7-5fdee28f3187" (UID: "13d8ee0f-f4c1-4630-85d7-5fdee28f3187"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:43:49.696473 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:49.696264 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13d8ee0f-f4c1-4630-85d7-5fdee28f3187-istio-token\" (UniqueName: \"kubernetes.io/projected/13d8ee0f-f4c1-4630-85d7-5fdee28f3187-istio-token\") pod \"13d8ee0f-f4c1-4630-85d7-5fdee28f3187\" (UID: \"13d8ee0f-f4c1-4630-85d7-5fdee28f3187\") " Oct 30 14:43:49.696473 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:49.696286 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13d8ee0f-f4c1-4630-85d7-5fdee28f3187-istiod-ca-cert" pod "13d8ee0f-f4c1-4630-85d7-5fdee28f3187" (UID: "13d8ee0f-f4c1-4630-85d7-5fdee28f3187"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:43:49.696596 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:49.696465 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13d8ee0f-f4c1-4630-85d7-5fdee28f3187-credential-socket" pod "13d8ee0f-f4c1-4630-85d7-5fdee28f3187" (UID: "13d8ee0f-f4c1-4630-85d7-5fdee28f3187"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:43:49.696796 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:49.696764 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13d8ee0f-f4c1-4630-85d7-5fdee28f3187-workload-certs" pod "13d8ee0f-f4c1-4630-85d7-5fdee28f3187" (UID: "13d8ee0f-f4c1-4630-85d7-5fdee28f3187"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:43:49.696796 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:49.696787 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13d8ee0f-f4c1-4630-85d7-5fdee28f3187-istio-data" pod "13d8ee0f-f4c1-4630-85d7-5fdee28f3187" (UID: "13d8ee0f-f4c1-4630-85d7-5fdee28f3187"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:43:49.697010 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:49.696873 4982 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13d8ee0f-f4c1-4630-85d7-5fdee28f3187-workload-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:43:49.697010 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:49.696906 4982 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13d8ee0f-f4c1-4630-85d7-5fdee28f3187-istiod-ca-cert\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:43:49.697010 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:49.696920 4982 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13d8ee0f-f4c1-4630-85d7-5fdee28f3187-credential-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:43:49.698411 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:49.698377 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/13d8ee0f-f4c1-4630-85d7-5fdee28f3187-istio-podinfo" pod "13d8ee0f-f4c1-4630-85d7-5fdee28f3187" (UID: "13d8ee0f-f4c1-4630-85d7-5fdee28f3187"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:43:49.698553 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:49.698533 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13d8ee0f-f4c1-4630-85d7-5fdee28f3187-istio-token" pod "13d8ee0f-f4c1-4630-85d7-5fdee28f3187" (UID: "13d8ee0f-f4c1-4630-85d7-5fdee28f3187"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:43:49.698616 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:49.698556 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13d8ee0f-f4c1-4630-85d7-5fdee28f3187-istio-envoy" pod "13d8ee0f-f4c1-4630-85d7-5fdee28f3187" (UID: "13d8ee0f-f4c1-4630-85d7-5fdee28f3187"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:43:49.698691 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:49.698667 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13d8ee0f-f4c1-4630-85d7-5fdee28f3187-kube-api-access-r8q65" pod "13d8ee0f-f4c1-4630-85d7-5fdee28f3187" (UID: "13d8ee0f-f4c1-4630-85d7-5fdee28f3187"). InnerVolumeSpecName "kube-api-access-r8q65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:43:49.801721 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:49.798784 4982 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13d8ee0f-f4c1-4630-85d7-5fdee28f3187-istio-data\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:43:49.801721 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:49.798830 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8q65\" (UniqueName: \"kubernetes.io/projected/13d8ee0f-f4c1-4630-85d7-5fdee28f3187-kube-api-access-r8q65\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:43:49.801721 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:49.798846 4982 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13d8ee0f-f4c1-4630-85d7-5fdee28f3187-istio-podinfo\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:43:49.801721 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:49.798860 4982 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13d8ee0f-f4c1-4630-85d7-5fdee28f3187-istio-envoy\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:43:49.801721 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:49.798873 4982 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13d8ee0f-f4c1-4630-85d7-5fdee28f3187-istio-token\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:43:49.801721 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:49.798885 4982 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13d8ee0f-f4c1-4630-85d7-5fdee28f3187-workload-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:43:50.363574 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:50.363540 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx" Oct 30 14:43:50.364046 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:50.363587 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx" event={"ID":"13d8ee0f-f4c1-4630-85d7-5fdee28f3187","Type":"ContainerDied","Data":"b39f104d40cb1cabb6ab5ec2fc9a4d982d37fe6983a99dc77b4e45a26310f5bd"} Oct 30 14:43:50.364046 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:50.363673 4982 scope.go:122] "RemoveContainer" containerID="c368a4e6fd60dcada4f4f3248ba1737ceb64c48779562ea3b450cd6162b359c3" Oct 30 14:43:50.423883 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:50.423854 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx"] Oct 30 14:43:50.436837 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:50.436811 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-gqvzx"] Oct 30 14:43:50.696263 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:50.696166 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:50.844390 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:50.844357 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:50.944758 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:50.944718 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ntlqq_dc967875-1546-40bb-8376-fde09416c204/console-operator/2.log" Oct 30 14:43:50.950280 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:50.950206 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhxjf_aca47fe8-7068-4743-b6ab-283ee0874f66/ovn-acl-logging/0.log" Oct 30 14:43:50.957755 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:50.957733 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ntlqq_dc967875-1546-40bb-8376-fde09416c204/console-operator/2.log" Oct 30 14:43:50.962769 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:50.962739 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhxjf_aca47fe8-7068-4743-b6ab-283ee0874f66/ovn-acl-logging/0.log" Oct 30 14:43:51.451422 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:51.451377 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:52.337657 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:52.337618 4982 scope.go:122] "RemoveContainer" containerID="90a2b3b3168f8a4af54a68946309dea29dc6dc7118c34e3d187c2abc7ef8c340" Oct 30 14:43:52.347830 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:52.347753 4982 scope.go:122] "RemoveContainer" containerID="c1d25184c03263c6bfd74313f78bfd5f540f44884faea71a0955dfeb9e01651e" Oct 30 14:43:52.356766 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:52.356748 4982 scope.go:122] "RemoveContainer" containerID="02ce3148e67b35adf05397ec7c2328594b68197f5ffd1da04a50980432d5e59a" Oct 30 14:43:52.365619 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:52.365595 4982 scope.go:122] "RemoveContainer" containerID="36b695902601745a61bea495b17f69f85e7a71b748a9e26ca07e48bb8ffdc134" Oct 30 14:43:52.403609 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:52.403550 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13d8ee0f-f4c1-4630-85d7-5fdee28f3187" path="/var/lib/kubelet/pods/13d8ee0f-f4c1-4630-85d7-5fdee28f3187/volumes" Oct 30 14:43:53.326330 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:53.326281 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:54.701282 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:54.701248 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:54.891149 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:54.891116 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:55.093966 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:55.093926 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2927fd-5cfc5f6ff9-p2hhp"] Oct 30 14:43:55.095087 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:55.095052 4982 cpu_manager.go:414] "containerMap: removing container" podUID="463ccf6c-67b5-45b6-970c-ee89fc36a12b" containerName="istio-proxy" Oct 30 14:43:55.095087 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:55.095084 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="463ccf6c-67b5-45b6-970c-ee89fc36a12b" containerName="istio-proxy" Oct 30 14:43:55.095304 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:55.095103 4982 cpu_manager.go:414] "containerMap: removing container" podUID="2e4c87c4-5edb-4292-9201-8ae74f5e3e60" containerName="revoke-keys" Oct 30 14:43:55.095304 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:55.095110 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e4c87c4-5edb-4292-9201-8ae74f5e3e60" containerName="revoke-keys" Oct 30 14:43:55.095304 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:55.095125 4982 cpu_manager.go:414] "containerMap: removing container" podUID="13d8ee0f-f4c1-4630-85d7-5fdee28f3187" containerName="istio-proxy" Oct 30 14:43:55.095304 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:55.095132 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13d8ee0f-f4c1-4630-85d7-5fdee28f3187" containerName="istio-proxy" Oct 30 14:43:55.095304 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:55.095159 4982 cpu_manager.go:414] "containerMap: removing container" podUID="06340c7e-9397-4280-bd4f-6c540ca20f73" containerName="revoke-keys" Oct 30 14:43:55.095304 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:55.095168 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06340c7e-9397-4280-bd4f-6c540ca20f73" containerName="revoke-keys" Oct 30 14:43:55.095603 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:55.095351 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e4c87c4-5edb-4292-9201-8ae74f5e3e60" containerName="revoke-keys" Oct 30 14:43:55.095603 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:55.095365 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="13d8ee0f-f4c1-4630-85d7-5fdee28f3187" containerName="istio-proxy" Oct 30 14:43:55.095603 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:55.095377 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="463ccf6c-67b5-45b6-970c-ee89fc36a12b" containerName="istio-proxy" Oct 30 14:43:55.095603 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:55.095385 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="06340c7e-9397-4280-bd4f-6c540ca20f73" containerName="revoke-keys" Oct 30 14:43:55.124658 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:55.121183 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2927fd-5cfc5f6ff9-p2hhp" Oct 30 14:43:55.124658 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:55.122411 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2927fd-5cfc5f6ff9-p2hhp"] Oct 30 14:43:55.160480 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:55.160166 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-2927fd\"" Oct 30 14:43:55.176458 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:55.176422 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-2927fd-dockercfg-d5c4w\"" Oct 30 14:43:55.237502 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:55.237470 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2927fd-58947b47f4-wh6th"] Oct 30 14:43:55.238184 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:55.238163 4982 cpu_manager.go:414] "containerMap: removing container" podUID="463ccf6c-67b5-45b6-970c-ee89fc36a12b" containerName="istio-proxy" Oct 30 14:43:55.238284 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:55.238184 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="463ccf6c-67b5-45b6-970c-ee89fc36a12b" containerName="istio-proxy" Oct 30 14:43:55.238419 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:55.238404 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="463ccf6c-67b5-45b6-970c-ee89fc36a12b" containerName="istio-proxy" Oct 30 14:43:55.241932 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:55.241911 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2927fd-58947b47f4-wh6th" Oct 30 14:43:55.266678 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:55.266632 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2927fd-58947b47f4-wh6th"] Oct 30 14:43:55.278432 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:55.278403 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9703a518-851b-4a8e-9e2c-586ea7fff550-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2927fd-58947b47f4-wh6th\" (UID: \"9703a518-851b-4a8e-9e2c-586ea7fff550\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2927fd-58947b47f4-wh6th" Oct 30 14:43:55.278535 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:55.278517 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vj28\" (UniqueName: \"kubernetes.io/projected/9703a518-851b-4a8e-9e2c-586ea7fff550-kube-api-access-2vj28\") pod \"payload-processing-e2e-ipp-a-2927fd-58947b47f4-wh6th\" (UID: \"9703a518-851b-4a8e-9e2c-586ea7fff550\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2927fd-58947b47f4-wh6th" Oct 30 14:43:55.278579 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:55.278550 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjcg5\" (UniqueName: \"kubernetes.io/projected/543506fd-c659-42d5-8338-8fdc25422ecb-kube-api-access-mjcg5\") pod \"payload-pre-processing-e2e-ipp-a-2927fd-5cfc5f6ff9-p2hhp\" (UID: \"543506fd-c659-42d5-8338-8fdc25422ecb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2927fd-5cfc5f6ff9-p2hhp" Oct 30 14:43:55.278710 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:55.278693 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/543506fd-c659-42d5-8338-8fdc25422ecb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2927fd-5cfc5f6ff9-p2hhp\" (UID: \"543506fd-c659-42d5-8338-8fdc25422ecb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2927fd-5cfc5f6ff9-p2hhp" Oct 30 14:43:55.379397 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:55.379327 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9703a518-851b-4a8e-9e2c-586ea7fff550-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2927fd-58947b47f4-wh6th\" (UID: \"9703a518-851b-4a8e-9e2c-586ea7fff550\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2927fd-58947b47f4-wh6th" Oct 30 14:43:55.379546 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:55.379437 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vj28\" (UniqueName: \"kubernetes.io/projected/9703a518-851b-4a8e-9e2c-586ea7fff550-kube-api-access-2vj28\") pod \"payload-processing-e2e-ipp-a-2927fd-58947b47f4-wh6th\" (UID: \"9703a518-851b-4a8e-9e2c-586ea7fff550\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2927fd-58947b47f4-wh6th" Oct 30 14:43:55.379546 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:55.379462 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjcg5\" (UniqueName: \"kubernetes.io/projected/543506fd-c659-42d5-8338-8fdc25422ecb-kube-api-access-mjcg5\") pod \"payload-pre-processing-e2e-ipp-a-2927fd-5cfc5f6ff9-p2hhp\" (UID: \"543506fd-c659-42d5-8338-8fdc25422ecb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2927fd-5cfc5f6ff9-p2hhp" Oct 30 14:43:55.379711 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:55.379552 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/543506fd-c659-42d5-8338-8fdc25422ecb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2927fd-5cfc5f6ff9-p2hhp\" (UID: \"543506fd-c659-42d5-8338-8fdc25422ecb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2927fd-5cfc5f6ff9-p2hhp" Oct 30 14:43:55.380239 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:55.380213 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9703a518-851b-4a8e-9e2c-586ea7fff550-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2927fd-58947b47f4-wh6th\" (UID: \"9703a518-851b-4a8e-9e2c-586ea7fff550\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2927fd-58947b47f4-wh6th" Oct 30 14:43:55.380424 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:55.380400 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/543506fd-c659-42d5-8338-8fdc25422ecb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2927fd-5cfc5f6ff9-p2hhp\" (UID: \"543506fd-c659-42d5-8338-8fdc25422ecb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2927fd-5cfc5f6ff9-p2hhp" Oct 30 14:43:55.407484 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:55.407449 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjcg5\" (UniqueName: \"kubernetes.io/projected/543506fd-c659-42d5-8338-8fdc25422ecb-kube-api-access-mjcg5\") pod \"payload-pre-processing-e2e-ipp-a-2927fd-5cfc5f6ff9-p2hhp\" (UID: \"543506fd-c659-42d5-8338-8fdc25422ecb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2927fd-5cfc5f6ff9-p2hhp" Oct 30 14:43:55.408802 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:55.408774 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vj28\" (UniqueName: \"kubernetes.io/projected/9703a518-851b-4a8e-9e2c-586ea7fff550-kube-api-access-2vj28\") pod \"payload-processing-e2e-ipp-a-2927fd-58947b47f4-wh6th\" (UID: \"9703a518-851b-4a8e-9e2c-586ea7fff550\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2927fd-58947b47f4-wh6th" Oct 30 14:43:55.497821 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:55.497791 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2927fd-5cfc5f6ff9-p2hhp" Oct 30 14:43:55.557147 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:55.557117 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2927fd-58947b47f4-wh6th" Oct 30 14:43:55.835096 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:55.835057 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2927fd-5cfc5f6ff9-p2hhp"] Oct 30 14:43:55.835535 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:43:55.835508 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod543506fd_c659_42d5_8338_8fdc25422ecb.slice/crio-99937d7286c0fc6c6e2b480727d966285cfb86cfba8a90827f38f897f39f43e7 WatchSource:0}: Error finding container 99937d7286c0fc6c6e2b480727d966285cfb86cfba8a90827f38f897f39f43e7: Status 404 returned error can't find the container with id 99937d7286c0fc6c6e2b480727d966285cfb86cfba8a90827f38f897f39f43e7 Oct 30 14:43:55.909891 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:55.909858 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2927fd-58947b47f4-wh6th"] Oct 30 14:43:55.910461 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:43:55.910431 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9703a518_851b_4a8e_9e2c_586ea7fff550.slice/crio-ef186bc5750618d490a73f5700ab880039b6c738d67b888bce67c3cae1c6323f WatchSource:0}: Error finding container ef186bc5750618d490a73f5700ab880039b6c738d67b888bce67c3cae1c6323f: Status 404 returned error can't find the container with id ef186bc5750618d490a73f5700ab880039b6c738d67b888bce67c3cae1c6323f Oct 30 14:43:55.991183 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:55.991155 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:56.039766 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:56.039731 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf3a515c-pgkfd"] Oct 30 14:43:56.042115 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:56.042088 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf3a515c-pgkfd"] Oct 30 14:43:56.406660 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:56.405243 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7bca45f-6601-4b89-8f43-6f98b878635f" path="/var/lib/kubelet/pods/e7bca45f-6601-4b89-8f43-6f98b878635f/volumes" Oct 30 14:43:56.435905 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:56.435864 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2927fd-5cfc5f6ff9-p2hhp" event={"ID":"543506fd-c659-42d5-8338-8fdc25422ecb","Type":"ContainerStarted","Data":"f1341bdf8345d08a3aed63ca42a6f3847be39f0caae14f641939b1ba05fe3570"} Oct 30 14:43:56.436061 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:56.436044 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2927fd-5cfc5f6ff9-p2hhp" Oct 30 14:43:56.436116 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:56.436076 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2927fd-5cfc5f6ff9-p2hhp" event={"ID":"543506fd-c659-42d5-8338-8fdc25422ecb","Type":"ContainerStarted","Data":"99937d7286c0fc6c6e2b480727d966285cfb86cfba8a90827f38f897f39f43e7"} Oct 30 14:43:56.437842 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:56.437814 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2927fd-58947b47f4-wh6th" event={"ID":"9703a518-851b-4a8e-9e2c-586ea7fff550","Type":"ContainerStarted","Data":"60f3daab65e9ee43cd9997931d8699107a54e5a08eee0d335eb2dfa872f82684"} Oct 30 14:43:56.437981 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:56.437857 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2927fd-58947b47f4-wh6th" event={"ID":"9703a518-851b-4a8e-9e2c-586ea7fff550","Type":"ContainerStarted","Data":"ef186bc5750618d490a73f5700ab880039b6c738d67b888bce67c3cae1c6323f"} Oct 30 14:43:56.437981 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:56.437882 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2927fd-58947b47f4-wh6th" Oct 30 14:43:56.483660 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:56.482388 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2927fd-5cfc5f6ff9-p2hhp" podStartSLOduration=1.482371557 podStartE2EDuration="1.482371557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:43:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:43:56.480293571 +0000 UTC m=+3306.224427612" watchObservedRunningTime="2026-10-30 14:43:56.482371557 +0000 UTC m=+3306.226505599" Oct 30 14:43:56.529612 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:56.529562 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-2927fd-58947b47f4-wh6th" podStartSLOduration=1.5295467409999999 podStartE2EDuration="1.529546741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:43:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:43:56.524351156 +0000 UTC m=+3306.268485198" watchObservedRunningTime="2026-10-30 14:43:56.529546741 +0000 UTC m=+3306.273680784" Oct 30 14:43:57.865840 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:43:57.865799 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:43:58.013298 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:43:58.013256 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:44:00.293162 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:00.293121 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:44:01.062704 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:01.062669 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83df63cb-cr85q"] Oct 30 14:44:01.075629 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:01.075596 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83df63cb-cr85q"] Oct 30 14:44:02.401190 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:02.401151 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e49340e-9383-4d23-a3b6-4128f67f086d" path="/var/lib/kubelet/pods/6e49340e-9383-4d23-a3b6-4128f67f086d/volumes" Oct 30 14:44:02.850998 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:02.850966 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:44:04.740767 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:04.740673 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:44:06.050463 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:06.050427 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce8ddfab-wnzjv"] Oct 30 14:44:06.063334 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:06.063309 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce8ddfab-wnzjv"] Oct 30 14:44:06.403469 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:06.403435 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0625c6bf-319a-4be3-84fb-d9fb16127f96" path="/var/lib/kubelet/pods/0625c6bf-319a-4be3-84fb-d9fb16127f96/volumes" Oct 30 14:44:07.146865 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:07.146830 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:44:07.448614 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:07.448532 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2927fd-58947b47f4-wh6th" Oct 30 14:44:07.448908 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:07.448883 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2927fd-5cfc5f6ff9-p2hhp" Oct 30 14:44:10.253456 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:10.253418 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:44:11.021299 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:11.021265 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:44:11.043995 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:11.043961 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d10039ce-nkvds"] Oct 30 14:44:11.055818 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:11.055787 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d10039ce-nkvds"] Oct 30 14:44:12.400584 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:12.400546 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5ba56a4-e3c3-446b-a537-ecb24d8236d1" path="/var/lib/kubelet/pods/e5ba56a4-e3c3-446b-a537-ecb24d8236d1/volumes" Oct 30 14:44:12.988333 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:12.988285 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:44:13.468034 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:13.468001 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:44:13.632844 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:13.632756 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:44:13.811135 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:13.811100 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:44:15.518278 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:15.518239 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:44:17.534994 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:17.534941 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:44:17.696664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:17.692258 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:44:18.057146 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:18.057111 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4bb39cb-87vp8"] Oct 30 14:44:18.071787 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:18.071754 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4bb39cb-87vp8"] Oct 30 14:44:18.400130 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:18.400102 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b2ae3e3-d20c-49a4-a8cc-4685ba38f6cd" path="/var/lib/kubelet/pods/2b2ae3e3-d20c-49a4-a8cc-4685ba38f6cd/volumes" Oct 30 14:44:19.232169 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:19.232136 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:44:19.289502 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:19.289459 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bcc710-7478845cd-x7lwp"] Oct 30 14:44:19.296399 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:19.296374 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-bcc710-7478845cd-x7lwp" Oct 30 14:44:19.319160 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:19.319131 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bcc710-7478845cd-x7lwp"] Oct 30 14:44:19.325357 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:19.325308 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-bcc710\"" Oct 30 14:44:19.350692 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:19.350658 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bcc710-76656769cf-dmk7s"] Oct 30 14:44:19.356398 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:19.356367 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcc710-76656769cf-dmk7s" Oct 30 14:44:19.395068 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:19.378128 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-bcc710-dockercfg-zmsd8\"" Oct 30 14:44:19.395068 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:19.389931 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bcc710-76656769cf-dmk7s"] Oct 30 14:44:19.395755 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:19.395724 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:44:19.547756 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:19.521564 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-227v9\" (UniqueName: \"kubernetes.io/projected/9139ba02-42d7-4ab6-895e-4bcf265f0449-kube-api-access-227v9\") pod \"payload-pre-processing-e2e-ipp-b-bcc710-76656769cf-dmk7s\" (UID: \"9139ba02-42d7-4ab6-895e-4bcf265f0449\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcc710-76656769cf-dmk7s" Oct 30 14:44:19.547756 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:19.521886 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9139ba02-42d7-4ab6-895e-4bcf265f0449-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-bcc710-76656769cf-dmk7s\" (UID: \"9139ba02-42d7-4ab6-895e-4bcf265f0449\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcc710-76656769cf-dmk7s" Oct 30 14:44:19.547756 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:19.521987 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4j5x\" (UniqueName: \"kubernetes.io/projected/a422b119-172d-4748-bfad-7cae7a82021c-kube-api-access-b4j5x\") pod \"payload-processing-e2e-ipp-b-bcc710-7478845cd-x7lwp\" (UID: \"a422b119-172d-4748-bfad-7cae7a82021c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bcc710-7478845cd-x7lwp" Oct 30 14:44:19.547756 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:19.522065 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a422b119-172d-4748-bfad-7cae7a82021c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-bcc710-7478845cd-x7lwp\" (UID: \"a422b119-172d-4748-bfad-7cae7a82021c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bcc710-7478845cd-x7lwp" Oct 30 14:44:19.623925 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:19.623879 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4j5x\" (UniqueName: \"kubernetes.io/projected/a422b119-172d-4748-bfad-7cae7a82021c-kube-api-access-b4j5x\") pod \"payload-processing-e2e-ipp-b-bcc710-7478845cd-x7lwp\" (UID: \"a422b119-172d-4748-bfad-7cae7a82021c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bcc710-7478845cd-x7lwp" Oct 30 14:44:19.624124 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:19.623986 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a422b119-172d-4748-bfad-7cae7a82021c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-bcc710-7478845cd-x7lwp\" (UID: \"a422b119-172d-4748-bfad-7cae7a82021c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bcc710-7478845cd-x7lwp" Oct 30 14:44:19.624124 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:19.624055 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-227v9\" (UniqueName: \"kubernetes.io/projected/9139ba02-42d7-4ab6-895e-4bcf265f0449-kube-api-access-227v9\") pod \"payload-pre-processing-e2e-ipp-b-bcc710-76656769cf-dmk7s\" (UID: \"9139ba02-42d7-4ab6-895e-4bcf265f0449\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcc710-76656769cf-dmk7s" Oct 30 14:44:19.624305 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:19.624278 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9139ba02-42d7-4ab6-895e-4bcf265f0449-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-bcc710-76656769cf-dmk7s\" (UID: \"9139ba02-42d7-4ab6-895e-4bcf265f0449\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcc710-76656769cf-dmk7s" Oct 30 14:44:19.624971 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:19.624940 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a422b119-172d-4748-bfad-7cae7a82021c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-bcc710-7478845cd-x7lwp\" (UID: \"a422b119-172d-4748-bfad-7cae7a82021c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bcc710-7478845cd-x7lwp" Oct 30 14:44:19.625217 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:19.625192 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9139ba02-42d7-4ab6-895e-4bcf265f0449-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-bcc710-76656769cf-dmk7s\" (UID: \"9139ba02-42d7-4ab6-895e-4bcf265f0449\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcc710-76656769cf-dmk7s" Oct 30 14:44:19.650014 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:19.649985 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4j5x\" (UniqueName: \"kubernetes.io/projected/a422b119-172d-4748-bfad-7cae7a82021c-kube-api-access-b4j5x\") pod \"payload-processing-e2e-ipp-b-bcc710-7478845cd-x7lwp\" (UID: \"a422b119-172d-4748-bfad-7cae7a82021c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bcc710-7478845cd-x7lwp" Oct 30 14:44:19.650180 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:19.650020 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-227v9\" (UniqueName: \"kubernetes.io/projected/9139ba02-42d7-4ab6-895e-4bcf265f0449-kube-api-access-227v9\") pod \"payload-pre-processing-e2e-ipp-b-bcc710-76656769cf-dmk7s\" (UID: \"9139ba02-42d7-4ab6-895e-4bcf265f0449\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcc710-76656769cf-dmk7s" Oct 30 14:44:19.697958 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:19.697916 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcc710-76656769cf-dmk7s" Oct 30 14:44:19.912578 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:19.912547 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-bcc710-7478845cd-x7lwp" Oct 30 14:44:19.968609 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:19.968578 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:44:20.022224 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:20.021804 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bcc710-76656769cf-dmk7s"] Oct 30 14:44:20.026681 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:44:20.025626 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9139ba02_42d7_4ab6_895e_4bcf265f0449.slice/crio-e2fd2fc4d20159b2fd3ac1cb9c4be89fde9d8851db82ae31ddc1860872f167a3 WatchSource:0}: Error finding container e2fd2fc4d20159b2fd3ac1cb9c4be89fde9d8851db82ae31ddc1860872f167a3: Status 404 returned error can't find the container with id e2fd2fc4d20159b2fd3ac1cb9c4be89fde9d8851db82ae31ddc1860872f167a3 Oct 30 14:44:20.219333 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:20.219254 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:44:20.245804 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:20.245780 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bcc710-7478845cd-x7lwp"] Oct 30 14:44:20.248747 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:44:20.248714 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda422b119_172d_4748_bfad_7cae7a82021c.slice/crio-4051f393c06851c8277ee66d96a808662088f7e629920a6045a937e21cd30c5d WatchSource:0}: Error finding container 4051f393c06851c8277ee66d96a808662088f7e629920a6045a937e21cd30c5d: Status 404 returned error can't find the container with id 4051f393c06851c8277ee66d96a808662088f7e629920a6045a937e21cd30c5d Oct 30 14:44:20.670092 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:20.670052 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-bcc710-7478845cd-x7lwp" event={"ID":"a422b119-172d-4748-bfad-7cae7a82021c","Type":"ContainerStarted","Data":"c98fbff5cb97829a22fced2550d54a050dc0daeebef6103028ed6a8a353eb513"} Oct 30 14:44:20.670288 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:20.670115 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-bcc710-7478845cd-x7lwp" event={"ID":"a422b119-172d-4748-bfad-7cae7a82021c","Type":"ContainerStarted","Data":"4051f393c06851c8277ee66d96a808662088f7e629920a6045a937e21cd30c5d"} Oct 30 14:44:20.670288 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:20.670142 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-bcc710-7478845cd-x7lwp" Oct 30 14:44:20.671953 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:20.671923 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcc710-76656769cf-dmk7s" event={"ID":"9139ba02-42d7-4ab6-895e-4bcf265f0449","Type":"ContainerStarted","Data":"869eed9b05c780139f0d307d217b8539acc9388092a1b1a26ad565e00712bc59"} Oct 30 14:44:20.672082 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:20.671967 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcc710-76656769cf-dmk7s" event={"ID":"9139ba02-42d7-4ab6-895e-4bcf265f0449","Type":"ContainerStarted","Data":"e2fd2fc4d20159b2fd3ac1cb9c4be89fde9d8851db82ae31ddc1860872f167a3"} Oct 30 14:44:20.672082 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:20.672033 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcc710-76656769cf-dmk7s" Oct 30 14:44:20.709763 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:20.709731 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:44:20.719210 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:20.719162 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-bcc710-7478845cd-x7lwp" podStartSLOduration=1.719145885 podStartE2EDuration="1.719145885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:44:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:44:20.714112869 +0000 UTC m=+3330.458246911" watchObservedRunningTime="2026-10-30 14:44:20.719145885 +0000 UTC m=+3330.463279927" Oct 30 14:44:20.761101 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:20.761045 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcc710-76656769cf-dmk7s" podStartSLOduration=1.7610269870000002 podStartE2EDuration="1.761026987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:44:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:44:20.75857097 +0000 UTC m=+3330.502705017" watchObservedRunningTime="2026-10-30 14:44:20.761026987 +0000 UTC m=+3330.505161029" Oct 30 14:44:21.405015 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:21.404979 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:44:21.567512 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:21.567459 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt"] Oct 30 14:44:21.576951 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:21.576916 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt" Oct 30 14:44:21.609141 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:21.609104 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:44:21.620350 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:21.620320 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt"] Oct 30 14:44:21.671135 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:21.671054 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/393471d3-e435-4bf0-835d-8e4c33ae17a7-workload-certs\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt\" (UID: \"393471d3-e435-4bf0-835d-8e4c33ae17a7\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt" Oct 30 14:44:21.671135 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:21.671104 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/393471d3-e435-4bf0-835d-8e4c33ae17a7-credential-socket\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt\" (UID: \"393471d3-e435-4bf0-835d-8e4c33ae17a7\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt" Oct 30 14:44:21.671312 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:21.671248 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/393471d3-e435-4bf0-835d-8e4c33ae17a7-istiod-ca-cert\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt\" (UID: \"393471d3-e435-4bf0-835d-8e4c33ae17a7\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt" Oct 30 14:44:21.671312 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:21.671302 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/393471d3-e435-4bf0-835d-8e4c33ae17a7-istio-token\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt\" (UID: \"393471d3-e435-4bf0-835d-8e4c33ae17a7\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt" Oct 30 14:44:21.671445 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:21.671429 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/393471d3-e435-4bf0-835d-8e4c33ae17a7-workload-socket\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt\" (UID: \"393471d3-e435-4bf0-835d-8e4c33ae17a7\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt" Oct 30 14:44:21.671514 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:21.671490 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/393471d3-e435-4bf0-835d-8e4c33ae17a7-istio-data\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt\" (UID: \"393471d3-e435-4bf0-835d-8e4c33ae17a7\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt" Oct 30 14:44:21.671572 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:21.671527 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/393471d3-e435-4bf0-835d-8e4c33ae17a7-istio-podinfo\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt\" (UID: \"393471d3-e435-4bf0-835d-8e4c33ae17a7\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt" Oct 30 14:44:21.671629 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:21.671572 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/393471d3-e435-4bf0-835d-8e4c33ae17a7-istio-envoy\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt\" (UID: \"393471d3-e435-4bf0-835d-8e4c33ae17a7\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt" Oct 30 14:44:21.671629 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:21.671602 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pswf\" (UniqueName: \"kubernetes.io/projected/393471d3-e435-4bf0-835d-8e4c33ae17a7-kube-api-access-6pswf\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt\" (UID: \"393471d3-e435-4bf0-835d-8e4c33ae17a7\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt" Oct 30 14:44:21.776684 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:21.776603 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/393471d3-e435-4bf0-835d-8e4c33ae17a7-workload-certs\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt\" (UID: \"393471d3-e435-4bf0-835d-8e4c33ae17a7\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt" Oct 30 14:44:21.785971 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:21.776685 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/393471d3-e435-4bf0-835d-8e4c33ae17a7-credential-socket\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt\" (UID: \"393471d3-e435-4bf0-835d-8e4c33ae17a7\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt" Oct 30 14:44:21.785971 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:21.776794 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/393471d3-e435-4bf0-835d-8e4c33ae17a7-istiod-ca-cert\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt\" (UID: \"393471d3-e435-4bf0-835d-8e4c33ae17a7\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt" Oct 30 14:44:21.785971 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:21.776866 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/393471d3-e435-4bf0-835d-8e4c33ae17a7-istio-token\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt\" (UID: \"393471d3-e435-4bf0-835d-8e4c33ae17a7\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt" Oct 30 14:44:21.785971 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:21.776932 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/393471d3-e435-4bf0-835d-8e4c33ae17a7-workload-socket\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt\" (UID: \"393471d3-e435-4bf0-835d-8e4c33ae17a7\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt" Oct 30 14:44:21.785971 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:21.777005 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/393471d3-e435-4bf0-835d-8e4c33ae17a7-istio-data\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt\" (UID: \"393471d3-e435-4bf0-835d-8e4c33ae17a7\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt" Oct 30 14:44:21.785971 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:21.777040 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/393471d3-e435-4bf0-835d-8e4c33ae17a7-istio-podinfo\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt\" (UID: \"393471d3-e435-4bf0-835d-8e4c33ae17a7\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt" Oct 30 14:44:21.785971 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:21.777077 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/393471d3-e435-4bf0-835d-8e4c33ae17a7-istio-envoy\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt\" (UID: \"393471d3-e435-4bf0-835d-8e4c33ae17a7\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt" Oct 30 14:44:21.785971 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:21.777120 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pswf\" (UniqueName: \"kubernetes.io/projected/393471d3-e435-4bf0-835d-8e4c33ae17a7-kube-api-access-6pswf\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt\" (UID: \"393471d3-e435-4bf0-835d-8e4c33ae17a7\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt" Oct 30 14:44:21.793581 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:21.792505 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/393471d3-e435-4bf0-835d-8e4c33ae17a7-istiod-ca-cert\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt\" (UID: \"393471d3-e435-4bf0-835d-8e4c33ae17a7\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt" Oct 30 14:44:21.793581 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:21.792858 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/393471d3-e435-4bf0-835d-8e4c33ae17a7-credential-socket\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt\" (UID: \"393471d3-e435-4bf0-835d-8e4c33ae17a7\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt" Oct 30 14:44:21.795155 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:21.795125 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/393471d3-e435-4bf0-835d-8e4c33ae17a7-istio-data\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt\" (UID: \"393471d3-e435-4bf0-835d-8e4c33ae17a7\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt" Oct 30 14:44:21.799691 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:21.799470 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/393471d3-e435-4bf0-835d-8e4c33ae17a7-workload-certs\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt\" (UID: \"393471d3-e435-4bf0-835d-8e4c33ae17a7\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt" Oct 30 14:44:21.800335 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:21.800310 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/393471d3-e435-4bf0-835d-8e4c33ae17a7-workload-socket\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt\" (UID: \"393471d3-e435-4bf0-835d-8e4c33ae17a7\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt" Oct 30 14:44:21.847660 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:21.837268 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pswf\" (UniqueName: \"kubernetes.io/projected/393471d3-e435-4bf0-835d-8e4c33ae17a7-kube-api-access-6pswf\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt\" (UID: \"393471d3-e435-4bf0-835d-8e4c33ae17a7\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt" Oct 30 14:44:21.877322 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:21.862971 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/393471d3-e435-4bf0-835d-8e4c33ae17a7-istio-envoy\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt\" (UID: \"393471d3-e435-4bf0-835d-8e4c33ae17a7\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt" Oct 30 14:44:21.877322 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:21.863480 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/393471d3-e435-4bf0-835d-8e4c33ae17a7-istio-podinfo\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt\" (UID: \"393471d3-e435-4bf0-835d-8e4c33ae17a7\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt" Oct 30 14:44:21.877322 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:21.863955 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/393471d3-e435-4bf0-835d-8e4c33ae17a7-istio-token\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt\" (UID: \"393471d3-e435-4bf0-835d-8e4c33ae17a7\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt" Oct 30 14:44:21.935754 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:21.928039 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt" Oct 30 14:44:22.034357 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:22.034315 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:44:22.131438 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:22.131405 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3b95dd-6969b8789f-cwgl2"] Oct 30 14:44:22.135997 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:22.135978 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b95dd-6969b8789f-cwgl2" Oct 30 14:44:22.167373 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:22.167324 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-3b95dd-dockercfg-kbrdd\"" Oct 30 14:44:22.170706 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:22.170683 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-3b95dd\"" Oct 30 14:44:22.173498 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:22.173472 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3b95dd-6969b8789f-cwgl2"] Oct 30 14:44:22.276999 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:22.276969 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt"] Oct 30 14:44:22.277693 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:44:22.277666 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod393471d3_e435_4bf0_835d_8e4c33ae17a7.slice/crio-a7246c961ec1f7f9f9f5308277a43fa925a5ae2a010bdcbd27936009cae35b43 WatchSource:0}: Error finding container a7246c961ec1f7f9f9f5308277a43fa925a5ae2a010bdcbd27936009cae35b43: Status 404 returned error can't find the container with id a7246c961ec1f7f9f9f5308277a43fa925a5ae2a010bdcbd27936009cae35b43 Oct 30 14:44:22.280526 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:22.280493 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:44:22.280619 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:22.280555 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:44:22.280619 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:22.280581 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:44:22.290168 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:22.290145 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jzml\" (UniqueName: \"kubernetes.io/projected/ef2a7ae4-27bb-4854-9736-b9f765bcb83c-kube-api-access-7jzml\") pod \"payload-pre-processing-e2e-ipp-b-3b95dd-6969b8789f-cwgl2\" (UID: \"ef2a7ae4-27bb-4854-9736-b9f765bcb83c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b95dd-6969b8789f-cwgl2" Oct 30 14:44:22.290381 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:22.290356 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef2a7ae4-27bb-4854-9736-b9f765bcb83c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3b95dd-6969b8789f-cwgl2\" (UID: \"ef2a7ae4-27bb-4854-9736-b9f765bcb83c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b95dd-6969b8789f-cwgl2" Oct 30 14:44:22.392245 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:22.392206 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jzml\" (UniqueName: \"kubernetes.io/projected/ef2a7ae4-27bb-4854-9736-b9f765bcb83c-kube-api-access-7jzml\") pod \"payload-pre-processing-e2e-ipp-b-3b95dd-6969b8789f-cwgl2\" (UID: \"ef2a7ae4-27bb-4854-9736-b9f765bcb83c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b95dd-6969b8789f-cwgl2" Oct 30 14:44:22.392472 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:22.392450 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef2a7ae4-27bb-4854-9736-b9f765bcb83c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3b95dd-6969b8789f-cwgl2\" (UID: \"ef2a7ae4-27bb-4854-9736-b9f765bcb83c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b95dd-6969b8789f-cwgl2" Oct 30 14:44:22.393309 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:22.393284 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef2a7ae4-27bb-4854-9736-b9f765bcb83c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3b95dd-6969b8789f-cwgl2\" (UID: \"ef2a7ae4-27bb-4854-9736-b9f765bcb83c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b95dd-6969b8789f-cwgl2" Oct 30 14:44:22.422459 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:22.422434 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jzml\" (UniqueName: \"kubernetes.io/projected/ef2a7ae4-27bb-4854-9736-b9f765bcb83c-kube-api-access-7jzml\") pod \"payload-pre-processing-e2e-ipp-b-3b95dd-6969b8789f-cwgl2\" (UID: \"ef2a7ae4-27bb-4854-9736-b9f765bcb83c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b95dd-6969b8789f-cwgl2" Oct 30 14:44:22.450114 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:22.450092 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b95dd-6969b8789f-cwgl2" Oct 30 14:44:22.675335 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:22.675303 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:44:22.699338 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:22.699303 4982 generic.go:356] "Generic (PLEG): container finished" podID="b5d8a300-e5ec-462a-95b3-d6f1e220dc42" containerID="b8ce2f3cc2284ca2be475a1f1e9b3f16132f1c5a2534b8f9dc61223fc01176ad" exitCode=0 Oct 30 14:44:22.699864 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:22.699839 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp" event={"ID":"b5d8a300-e5ec-462a-95b3-d6f1e220dc42","Type":"ContainerDied","Data":"b8ce2f3cc2284ca2be475a1f1e9b3f16132f1c5a2534b8f9dc61223fc01176ad"} Oct 30 14:44:22.701506 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:22.701484 4982 scope.go:122] "RemoveContainer" containerID="b8ce2f3cc2284ca2be475a1f1e9b3f16132f1c5a2534b8f9dc61223fc01176ad" Oct 30 14:44:22.705533 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:22.704251 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:44:22.705533 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:22.704353 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:44:22.705533 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:22.704398 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:44:22.706048 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:22.706012 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt" event={"ID":"393471d3-e435-4bf0-835d-8e4c33ae17a7","Type":"ContainerStarted","Data":"ba8ce946fc23f4ae69b314fa1433006e0f0dad111e03d0225e26bee922319aaa"} Oct 30 14:44:22.706141 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:22.706069 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt" event={"ID":"393471d3-e435-4bf0-835d-8e4c33ae17a7","Type":"ContainerStarted","Data":"a7246c961ec1f7f9f9f5308277a43fa925a5ae2a010bdcbd27936009cae35b43"} Oct 30 14:44:22.781824 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:22.780660 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3b95dd-6969b8789f-cwgl2"] Oct 30 14:44:22.818021 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:22.817963 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt" podStartSLOduration=1.817930696 podStartE2EDuration="1.817930696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:44:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:44:22.813824363 +0000 UTC m=+3332.557958405" watchObservedRunningTime="2026-10-30 14:44:22.817930696 +0000 UTC m=+3332.562064738" Oct 30 14:44:22.929300 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:22.929267 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt" Oct 30 14:44:23.736099 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:23.736056 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp" event={"ID":"b5d8a300-e5ec-462a-95b3-d6f1e220dc42","Type":"ContainerStarted","Data":"7da8182ed5d2a327a9f2305f9bf341b2691c4e38e16ea0b63bc4d79805f7e55a"} Oct 30 14:44:23.738827 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:23.738792 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b95dd-6969b8789f-cwgl2" event={"ID":"ef2a7ae4-27bb-4854-9736-b9f765bcb83c","Type":"ContainerStarted","Data":"7fcc56797e6fb75ac4944dfa04509deedc4a5cb7837ba6b8bf0f26a8c88c4807"} Oct 30 14:44:23.738983 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:23.738844 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b95dd-6969b8789f-cwgl2" event={"ID":"ef2a7ae4-27bb-4854-9736-b9f765bcb83c","Type":"ContainerStarted","Data":"672ddc28eec223e9f517a6f88ffec52e9660762e5f2983f7effd78dc0bb98199"} Oct 30 14:44:23.738983 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:23.738965 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b95dd-6969b8789f-cwgl2" Oct 30 14:44:23.868115 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:23.868049 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b95dd-6969b8789f-cwgl2" podStartSLOduration=1.868034825 podStartE2EDuration="1.868034825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:44:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:44:23.865918523 +0000 UTC m=+3333.610052563" watchObservedRunningTime="2026-10-30 14:44:23.868034825 +0000 UTC m=+3333.612168865" Oct 30 14:44:23.933718 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:23.933671 4982 patch_prober.go:29] interesting pod/e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body= Oct 30 14:44:23.933920 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:23.933737 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt" podUID="393471d3-e435-4bf0-835d-8e4c33ae17a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" Oct 30 14:44:23.963197 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:23.963157 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp" Oct 30 14:44:23.964559 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:23.964525 4982 patch_prober.go:29] interesting pod/e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 30 14:44:23.964888 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:23.964829 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp" podUID="b5d8a300-e5ec-462a-95b3-d6f1e220dc42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 30 14:44:24.139658 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:24.138766 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp" Oct 30 14:44:24.869665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:24.868841 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:44:24.942665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:24.941995 4982 patch_prober.go:29] interesting pod/e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body= Oct 30 14:44:24.942665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:24.942055 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt" podUID="393471d3-e435-4bf0-835d-8e4c33ae17a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" Oct 30 14:44:24.965067 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:24.964999 4982 patch_prober.go:29] interesting pod/e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 30 14:44:24.965265 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:24.965091 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp" podUID="b5d8a300-e5ec-462a-95b3-d6f1e220dc42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 30 14:44:25.931664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:25.930976 4982 patch_prober.go:29] interesting pod/e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body= Oct 30 14:44:25.931664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:25.931034 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt" podUID="393471d3-e435-4bf0-835d-8e4c33ae17a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" Oct 30 14:44:25.962745 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:25.962703 4982 patch_prober.go:29] interesting pod/e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 30 14:44:25.962921 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:25.962758 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp" podUID="b5d8a300-e5ec-462a-95b3-d6f1e220dc42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 30 14:44:26.750407 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:26.750351 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:44:26.934082 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:26.934042 4982 patch_prober.go:29] interesting pod/e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body= Oct 30 14:44:26.934683 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:26.934627 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt" podUID="393471d3-e435-4bf0-835d-8e4c33ae17a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" Oct 30 14:44:26.962230 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:26.962165 4982 patch_prober.go:29] interesting pod/e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 30 14:44:26.962230 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:26.962221 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp" podUID="b5d8a300-e5ec-462a-95b3-d6f1e220dc42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 30 14:44:27.930663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:27.929978 4982 patch_prober.go:29] interesting pod/e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body= Oct 30 14:44:27.930663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:27.930041 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt" podUID="393471d3-e435-4bf0-835d-8e4c33ae17a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" Oct 30 14:44:27.963115 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:27.963085 4982 patch_prober.go:29] interesting pod/e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 30 14:44:27.963499 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:27.963217 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp" podUID="b5d8a300-e5ec-462a-95b3-d6f1e220dc42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 30 14:44:28.052598 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:28.052557 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-030ea30d-cvdkq"] Oct 30 14:44:28.064875 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:28.064844 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-030ea30d-cvdkq"] Oct 30 14:44:28.067185 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:44:28.067158 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:44:28.450208 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:28.450165 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6ae17d8-7e51-411a-bd43-75331301f48a" path="/var/lib/kubelet/pods/c6ae17d8-7e51-411a-bd43-75331301f48a/volumes" Oct 30 14:44:28.468600 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:28.468567 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4be240-78755bcc95-r6pkn"] Oct 30 14:44:28.473320 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:28.473294 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4be240-78755bcc95-r6pkn" Oct 30 14:44:28.526816 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:28.526784 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-4be240-serving-cert\"" Oct 30 14:44:28.556390 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:28.556357 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4be240-78755bcc95-r6pkn"] Oct 30 14:44:28.573884 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:28.573854 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8130ecc-2ff2-416b-99fb-d0f7ae285432-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4be240-78755bcc95-r6pkn\" (UID: \"a8130ecc-2ff2-416b-99fb-d0f7ae285432\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4be240-78755bcc95-r6pkn" Oct 30 14:44:28.574125 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:28.574110 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8130ecc-2ff2-416b-99fb-d0f7ae285432-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-4be240-78755bcc95-r6pkn\" (UID: \"a8130ecc-2ff2-416b-99fb-d0f7ae285432\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4be240-78755bcc95-r6pkn" Oct 30 14:44:28.574339 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:28.574315 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8130ecc-2ff2-416b-99fb-d0f7ae285432-metrics-certs\") pod \"maas-api-e2e-ipp-b-4be240-78755bcc95-r6pkn\" (UID: \"a8130ecc-2ff2-416b-99fb-d0f7ae285432\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4be240-78755bcc95-r6pkn" Oct 30 14:44:28.574394 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:28.574351 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g52k\" (UniqueName: \"kubernetes.io/projected/a8130ecc-2ff2-416b-99fb-d0f7ae285432-kube-api-access-9g52k\") pod \"maas-api-e2e-ipp-b-4be240-78755bcc95-r6pkn\" (UID: \"a8130ecc-2ff2-416b-99fb-d0f7ae285432\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4be240-78755bcc95-r6pkn" Oct 30 14:44:28.649365 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:28.649305 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:44:28.675913 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:28.675880 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8130ecc-2ff2-416b-99fb-d0f7ae285432-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4be240-78755bcc95-r6pkn\" (UID: \"a8130ecc-2ff2-416b-99fb-d0f7ae285432\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4be240-78755bcc95-r6pkn" Oct 30 14:44:28.676098 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:28.675971 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8130ecc-2ff2-416b-99fb-d0f7ae285432-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-4be240-78755bcc95-r6pkn\" (UID: \"a8130ecc-2ff2-416b-99fb-d0f7ae285432\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4be240-78755bcc95-r6pkn" Oct 30 14:44:28.676098 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:28.676063 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8130ecc-2ff2-416b-99fb-d0f7ae285432-metrics-certs\") pod \"maas-api-e2e-ipp-b-4be240-78755bcc95-r6pkn\" (UID: \"a8130ecc-2ff2-416b-99fb-d0f7ae285432\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4be240-78755bcc95-r6pkn" Oct 30 14:44:28.676098 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:28.676085 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g52k\" (UniqueName: \"kubernetes.io/projected/a8130ecc-2ff2-416b-99fb-d0f7ae285432-kube-api-access-9g52k\") pod \"maas-api-e2e-ipp-b-4be240-78755bcc95-r6pkn\" (UID: \"a8130ecc-2ff2-416b-99fb-d0f7ae285432\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4be240-78755bcc95-r6pkn" Oct 30 14:44:28.676481 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:44:28.676455 4982 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-4be240-serving-cert: secret "maas-api-e2e-ipp-b-4be240-serving-cert" not found Oct 30 14:44:28.677417 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:44:28.676702 4982 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8130ecc-2ff2-416b-99fb-d0f7ae285432-maas-api-tls podName:a8130ecc-2ff2-416b-99fb-d0f7ae285432 nodeName:}" failed. No retries permitted until 2026-10-30 14:44:29.176678688 +0000 UTC m=+3338.920812726 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a8130ecc-2ff2-416b-99fb-d0f7ae285432-maas-api-tls") pod "maas-api-e2e-ipp-b-4be240-78755bcc95-r6pkn" (UID: "a8130ecc-2ff2-416b-99fb-d0f7ae285432") : secret "maas-api-e2e-ipp-b-4be240-serving-cert" not found Oct 30 14:44:28.677576 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:28.677472 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8130ecc-2ff2-416b-99fb-d0f7ae285432-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-4be240-78755bcc95-r6pkn\" (UID: \"a8130ecc-2ff2-416b-99fb-d0f7ae285432\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4be240-78755bcc95-r6pkn" Oct 30 14:44:28.748427 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:28.748342 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g52k\" (UniqueName: \"kubernetes.io/projected/a8130ecc-2ff2-416b-99fb-d0f7ae285432-kube-api-access-9g52k\") pod \"maas-api-e2e-ipp-b-4be240-78755bcc95-r6pkn\" (UID: \"a8130ecc-2ff2-416b-99fb-d0f7ae285432\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4be240-78755bcc95-r6pkn" Oct 30 14:44:28.758855 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:28.749695 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8130ecc-2ff2-416b-99fb-d0f7ae285432-metrics-certs\") pod \"maas-api-e2e-ipp-b-4be240-78755bcc95-r6pkn\" (UID: \"a8130ecc-2ff2-416b-99fb-d0f7ae285432\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4be240-78755bcc95-r6pkn" Oct 30 14:44:28.817666 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:28.817232 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:44:28.933462 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:28.932956 4982 patch_prober.go:29] interesting pod/e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body= Oct 30 14:44:28.933462 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:28.933014 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt" podUID="393471d3-e435-4bf0-835d-8e4c33ae17a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" Oct 30 14:44:28.962212 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:28.962172 4982 patch_prober.go:29] interesting pod/e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 30 14:44:28.962425 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:28.962233 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp" podUID="b5d8a300-e5ec-462a-95b3-d6f1e220dc42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 30 14:44:29.106686 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:29.106109 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:44:29.213445 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:29.213401 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8130ecc-2ff2-416b-99fb-d0f7ae285432-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4be240-78755bcc95-r6pkn\" (UID: \"a8130ecc-2ff2-416b-99fb-d0f7ae285432\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4be240-78755bcc95-r6pkn" Oct 30 14:44:29.216334 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:29.216287 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8130ecc-2ff2-416b-99fb-d0f7ae285432-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4be240-78755bcc95-r6pkn\" (UID: \"a8130ecc-2ff2-416b-99fb-d0f7ae285432\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4be240-78755bcc95-r6pkn" Oct 30 14:44:29.456680 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:29.456582 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4be240-78755bcc95-r6pkn" Oct 30 14:44:29.797167 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:29.797139 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4be240-78755bcc95-r6pkn"] Oct 30 14:44:29.799000 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:44:29.798965 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8130ecc_2ff2_416b_99fb_d0f7ae285432.slice/crio-856145b507e5d401b6647a246254df8d4ca9f457247c62c55dd58cbd1cb252f1 WatchSource:0}: Error finding container 856145b507e5d401b6647a246254df8d4ca9f457247c62c55dd58cbd1cb252f1: Status 404 returned error can't find the container with id 856145b507e5d401b6647a246254df8d4ca9f457247c62c55dd58cbd1cb252f1 Oct 30 14:44:29.800576 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:29.800551 4982 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 14:44:29.897210 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:29.897167 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4be240-78755bcc95-r6pkn" event={"ID":"a8130ecc-2ff2-416b-99fb-d0f7ae285432","Type":"ContainerStarted","Data":"856145b507e5d401b6647a246254df8d4ca9f457247c62c55dd58cbd1cb252f1"} Oct 30 14:44:29.935435 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:29.935397 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt" Oct 30 14:44:29.935801 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:29.935779 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt" Oct 30 14:44:29.936396 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:29.936379 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt" Oct 30 14:44:29.967479 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:29.967455 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp" Oct 30 14:44:29.968810 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:29.968786 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp" Oct 30 14:44:30.678008 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:30.677973 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:44:30.907015 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:30.906974 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4be240-78755bcc95-r6pkn" event={"ID":"a8130ecc-2ff2-416b-99fb-d0f7ae285432","Type":"ContainerStarted","Data":"9bbb36009a89977191aa34637c4836f65d29c8271eda4acc47e3d2f6da8ccfe7"} Oct 30 14:44:30.907553 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:30.907527 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4be240-78755bcc95-r6pkn" Oct 30 14:44:30.953830 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:30.953743 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4be240-78755bcc95-r6pkn" podStartSLOduration=2.519123188 podStartE2EDuration="2.953730036s" totalImagesPullingTime="434.606848ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:44:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:44:30.951783911 +0000 UTC m=+3340.695917951" watchObservedRunningTime="2026-10-30 14:44:30.953730036 +0000 UTC m=+3340.697864130" Oct 30 14:44:31.683893 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:31.683861 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcc710-76656769cf-dmk7s" Oct 30 14:44:31.684359 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:31.683959 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-bcc710-7478845cd-x7lwp" Oct 30 14:44:32.359788 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:32.359756 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-90b406-56b44b6b55-9nd98"] Oct 30 14:44:32.364260 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:32.364242 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90b406-56b44b6b55-9nd98" Oct 30 14:44:32.384294 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:32.384268 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-90b406-serving-cert\"" Oct 30 14:44:32.388483 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:32.388458 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-90b406-56b44b6b55-9nd98"] Oct 30 14:44:32.484872 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:32.484840 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:44:32.494213 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:32.494176 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5298h\" (UniqueName: \"kubernetes.io/projected/4d6e8423-5356-4a73-9c9b-43c29eedb02b-kube-api-access-5298h\") pod \"maas-api-e2e-ipp-b-90b406-56b44b6b55-9nd98\" (UID: \"4d6e8423-5356-4a73-9c9b-43c29eedb02b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90b406-56b44b6b55-9nd98" Oct 30 14:44:32.494378 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:32.494280 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d6e8423-5356-4a73-9c9b-43c29eedb02b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-90b406-56b44b6b55-9nd98\" (UID: \"4d6e8423-5356-4a73-9c9b-43c29eedb02b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90b406-56b44b6b55-9nd98" Oct 30 14:44:32.494378 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:32.494313 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d6e8423-5356-4a73-9c9b-43c29eedb02b-metrics-certs\") pod \"maas-api-e2e-ipp-b-90b406-56b44b6b55-9nd98\" (UID: \"4d6e8423-5356-4a73-9c9b-43c29eedb02b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90b406-56b44b6b55-9nd98" Oct 30 14:44:32.494499 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:32.494396 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d6e8423-5356-4a73-9c9b-43c29eedb02b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-90b406-56b44b6b55-9nd98\" (UID: \"4d6e8423-5356-4a73-9c9b-43c29eedb02b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90b406-56b44b6b55-9nd98" Oct 30 14:44:32.599745 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:32.599667 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5298h\" (UniqueName: \"kubernetes.io/projected/4d6e8423-5356-4a73-9c9b-43c29eedb02b-kube-api-access-5298h\") pod \"maas-api-e2e-ipp-b-90b406-56b44b6b55-9nd98\" (UID: \"4d6e8423-5356-4a73-9c9b-43c29eedb02b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90b406-56b44b6b55-9nd98" Oct 30 14:44:32.599961 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:32.599756 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d6e8423-5356-4a73-9c9b-43c29eedb02b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-90b406-56b44b6b55-9nd98\" (UID: \"4d6e8423-5356-4a73-9c9b-43c29eedb02b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90b406-56b44b6b55-9nd98" Oct 30 14:44:32.599961 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:32.599798 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d6e8423-5356-4a73-9c9b-43c29eedb02b-metrics-certs\") pod \"maas-api-e2e-ipp-b-90b406-56b44b6b55-9nd98\" (UID: \"4d6e8423-5356-4a73-9c9b-43c29eedb02b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90b406-56b44b6b55-9nd98" Oct 30 14:44:32.599961 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:32.599871 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d6e8423-5356-4a73-9c9b-43c29eedb02b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-90b406-56b44b6b55-9nd98\" (UID: \"4d6e8423-5356-4a73-9c9b-43c29eedb02b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90b406-56b44b6b55-9nd98" Oct 30 14:44:32.600971 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:32.600944 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d6e8423-5356-4a73-9c9b-43c29eedb02b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-90b406-56b44b6b55-9nd98\" (UID: \"4d6e8423-5356-4a73-9c9b-43c29eedb02b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90b406-56b44b6b55-9nd98" Oct 30 14:44:32.604049 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:32.604017 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d6e8423-5356-4a73-9c9b-43c29eedb02b-metrics-certs\") pod \"maas-api-e2e-ipp-b-90b406-56b44b6b55-9nd98\" (UID: \"4d6e8423-5356-4a73-9c9b-43c29eedb02b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90b406-56b44b6b55-9nd98" Oct 30 14:44:32.604317 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:32.604283 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d6e8423-5356-4a73-9c9b-43c29eedb02b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-90b406-56b44b6b55-9nd98\" (UID: \"4d6e8423-5356-4a73-9c9b-43c29eedb02b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90b406-56b44b6b55-9nd98" Oct 30 14:44:32.627216 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:32.627146 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5298h\" (UniqueName: \"kubernetes.io/projected/4d6e8423-5356-4a73-9c9b-43c29eedb02b-kube-api-access-5298h\") pod \"maas-api-e2e-ipp-b-90b406-56b44b6b55-9nd98\" (UID: \"4d6e8423-5356-4a73-9c9b-43c29eedb02b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90b406-56b44b6b55-9nd98" Oct 30 14:44:32.645695 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:32.645665 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:44:32.678463 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:32.678434 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90b406-56b44b6b55-9nd98" Oct 30 14:44:32.993578 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:32.993549 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-90b406-56b44b6b55-9nd98"] Oct 30 14:44:32.996704 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:44:32.996673 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d6e8423_5356_4a73_9c9b_43c29eedb02b.slice/crio-50d4d09d95830e17e014e056d50c54edc4050913f1afabab02852b941604b8a7 WatchSource:0}: Error finding container 50d4d09d95830e17e014e056d50c54edc4050913f1afabab02852b941604b8a7: Status 404 returned error can't find the container with id 50d4d09d95830e17e014e056d50c54edc4050913f1afabab02852b941604b8a7 Oct 30 14:44:33.933209 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:33.933166 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90b406-56b44b6b55-9nd98" event={"ID":"4d6e8423-5356-4a73-9c9b-43c29eedb02b","Type":"ContainerStarted","Data":"f784b6390f0eca03503631854eeb0eb38f2de7eb5f066775b3ff258fb87d2bd4"} Oct 30 14:44:33.933384 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:33.933236 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90b406-56b44b6b55-9nd98" Oct 30 14:44:33.933384 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:33.933259 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90b406-56b44b6b55-9nd98" event={"ID":"4d6e8423-5356-4a73-9c9b-43c29eedb02b","Type":"ContainerStarted","Data":"50d4d09d95830e17e014e056d50c54edc4050913f1afabab02852b941604b8a7"} Oct 30 14:44:33.980660 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:33.980594 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90b406-56b44b6b55-9nd98" podStartSLOduration=1.644110741 podStartE2EDuration="1.980577746s" totalImagesPullingTime="336.467005ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:44:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:44:33.979361076 +0000 UTC m=+3343.723495116" watchObservedRunningTime="2026-10-30 14:44:33.980577746 +0000 UTC m=+3343.724711786" Oct 30 14:44:34.471252 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:34.471219 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:44:34.760259 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:34.760175 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b95dd-6969b8789f-cwgl2" Oct 30 14:44:36.423451 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:36.423416 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:44:36.922184 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:36.922159 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4be240-78755bcc95-r6pkn" Oct 30 14:44:38.265366 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:38.265327 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:44:39.754662 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:39.753737 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-f0e8b9/ipp-route-f0e8b9-kserve-7f4f866ccc-pz7w2"] Oct 30 14:44:39.759548 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:39.759524 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-f0e8b9/ipp-route-f0e8b9-kserve-7f4f866ccc-pz7w2" Oct 30 14:44:39.788886 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:39.788515 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-f0e8b9/ipp-route-f0e8b9-kserve-7f4f866ccc-pz7w2"] Oct 30 14:44:39.788886 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:39.788799 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-f0e8b9\"/\"openshift-service-ca.crt\"" Oct 30 14:44:39.789128 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:39.788896 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-f0e8b9\"/\"kube-root-ca.crt\"" Oct 30 14:44:39.789128 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:39.788939 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-f0e8b9\"/\"ipp-route-f0e8b9-kserve-self-signed-certs\"" Oct 30 14:44:39.790142 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:39.789612 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-f0e8b9\"/\"default-dockercfg-v5vwr\"" Oct 30 14:44:39.807404 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:39.807365 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8szkf\" (UniqueName: \"kubernetes.io/projected/02251d15-f79c-41a6-bd91-6ec5969e1f63-kube-api-access-8szkf\") pod \"ipp-route-f0e8b9-kserve-7f4f866ccc-pz7w2\" (UID: \"02251d15-f79c-41a6-bd91-6ec5969e1f63\") " pod="ai-tenant-e2e-ipp-a-f0e8b9/ipp-route-f0e8b9-kserve-7f4f866ccc-pz7w2" Oct 30 14:44:39.808785 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:39.808221 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02251d15-f79c-41a6-bd91-6ec5969e1f63-tmp-dir\") pod \"ipp-route-f0e8b9-kserve-7f4f866ccc-pz7w2\" (UID: \"02251d15-f79c-41a6-bd91-6ec5969e1f63\") " pod="ai-tenant-e2e-ipp-a-f0e8b9/ipp-route-f0e8b9-kserve-7f4f866ccc-pz7w2" Oct 30 14:44:39.808785 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:39.808272 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02251d15-f79c-41a6-bd91-6ec5969e1f63-model-cache\") pod \"ipp-route-f0e8b9-kserve-7f4f866ccc-pz7w2\" (UID: \"02251d15-f79c-41a6-bd91-6ec5969e1f63\") " pod="ai-tenant-e2e-ipp-a-f0e8b9/ipp-route-f0e8b9-kserve-7f4f866ccc-pz7w2" Oct 30 14:44:39.808785 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:39.808355 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02251d15-f79c-41a6-bd91-6ec5969e1f63-dshm\") pod \"ipp-route-f0e8b9-kserve-7f4f866ccc-pz7w2\" (UID: \"02251d15-f79c-41a6-bd91-6ec5969e1f63\") " pod="ai-tenant-e2e-ipp-a-f0e8b9/ipp-route-f0e8b9-kserve-7f4f866ccc-pz7w2" Oct 30 14:44:39.808785 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:39.808656 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02251d15-f79c-41a6-bd91-6ec5969e1f63-tls-certs\") pod \"ipp-route-f0e8b9-kserve-7f4f866ccc-pz7w2\" (UID: \"02251d15-f79c-41a6-bd91-6ec5969e1f63\") " pod="ai-tenant-e2e-ipp-a-f0e8b9/ipp-route-f0e8b9-kserve-7f4f866ccc-pz7w2" Oct 30 14:44:39.808785 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:39.808719 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02251d15-f79c-41a6-bd91-6ec5969e1f63-home\") pod \"ipp-route-f0e8b9-kserve-7f4f866ccc-pz7w2\" (UID: \"02251d15-f79c-41a6-bd91-6ec5969e1f63\") " pod="ai-tenant-e2e-ipp-a-f0e8b9/ipp-route-f0e8b9-kserve-7f4f866ccc-pz7w2" Oct 30 14:44:39.826057 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:39.826029 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:44:39.910670 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:39.910616 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02251d15-f79c-41a6-bd91-6ec5969e1f63-tls-certs\") pod \"ipp-route-f0e8b9-kserve-7f4f866ccc-pz7w2\" (UID: \"02251d15-f79c-41a6-bd91-6ec5969e1f63\") " pod="ai-tenant-e2e-ipp-a-f0e8b9/ipp-route-f0e8b9-kserve-7f4f866ccc-pz7w2" Oct 30 14:44:39.910853 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:39.910708 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02251d15-f79c-41a6-bd91-6ec5969e1f63-home\") pod \"ipp-route-f0e8b9-kserve-7f4f866ccc-pz7w2\" (UID: \"02251d15-f79c-41a6-bd91-6ec5969e1f63\") " pod="ai-tenant-e2e-ipp-a-f0e8b9/ipp-route-f0e8b9-kserve-7f4f866ccc-pz7w2" Oct 30 14:44:39.910915 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:39.910856 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8szkf\" (UniqueName: \"kubernetes.io/projected/02251d15-f79c-41a6-bd91-6ec5969e1f63-kube-api-access-8szkf\") pod \"ipp-route-f0e8b9-kserve-7f4f866ccc-pz7w2\" (UID: \"02251d15-f79c-41a6-bd91-6ec5969e1f63\") " pod="ai-tenant-e2e-ipp-a-f0e8b9/ipp-route-f0e8b9-kserve-7f4f866ccc-pz7w2" Oct 30 14:44:39.910915 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:39.910901 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02251d15-f79c-41a6-bd91-6ec5969e1f63-tmp-dir\") pod \"ipp-route-f0e8b9-kserve-7f4f866ccc-pz7w2\" (UID: \"02251d15-f79c-41a6-bd91-6ec5969e1f63\") " pod="ai-tenant-e2e-ipp-a-f0e8b9/ipp-route-f0e8b9-kserve-7f4f866ccc-pz7w2" Oct 30 14:44:39.911030 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:39.910937 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02251d15-f79c-41a6-bd91-6ec5969e1f63-model-cache\") pod \"ipp-route-f0e8b9-kserve-7f4f866ccc-pz7w2\" (UID: \"02251d15-f79c-41a6-bd91-6ec5969e1f63\") " pod="ai-tenant-e2e-ipp-a-f0e8b9/ipp-route-f0e8b9-kserve-7f4f866ccc-pz7w2" Oct 30 14:44:39.911030 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:39.911002 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02251d15-f79c-41a6-bd91-6ec5969e1f63-dshm\") pod \"ipp-route-f0e8b9-kserve-7f4f866ccc-pz7w2\" (UID: \"02251d15-f79c-41a6-bd91-6ec5969e1f63\") " pod="ai-tenant-e2e-ipp-a-f0e8b9/ipp-route-f0e8b9-kserve-7f4f866ccc-pz7w2" Oct 30 14:44:39.911467 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:39.911439 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02251d15-f79c-41a6-bd91-6ec5969e1f63-home\") pod \"ipp-route-f0e8b9-kserve-7f4f866ccc-pz7w2\" (UID: \"02251d15-f79c-41a6-bd91-6ec5969e1f63\") " pod="ai-tenant-e2e-ipp-a-f0e8b9/ipp-route-f0e8b9-kserve-7f4f866ccc-pz7w2" Oct 30 14:44:39.911752 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:39.911729 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02251d15-f79c-41a6-bd91-6ec5969e1f63-tmp-dir\") pod \"ipp-route-f0e8b9-kserve-7f4f866ccc-pz7w2\" (UID: \"02251d15-f79c-41a6-bd91-6ec5969e1f63\") " pod="ai-tenant-e2e-ipp-a-f0e8b9/ipp-route-f0e8b9-kserve-7f4f866ccc-pz7w2" Oct 30 14:44:39.911905 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:39.911794 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02251d15-f79c-41a6-bd91-6ec5969e1f63-model-cache\") pod \"ipp-route-f0e8b9-kserve-7f4f866ccc-pz7w2\" (UID: \"02251d15-f79c-41a6-bd91-6ec5969e1f63\") " pod="ai-tenant-e2e-ipp-a-f0e8b9/ipp-route-f0e8b9-kserve-7f4f866ccc-pz7w2" Oct 30 14:44:39.913842 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:39.913812 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02251d15-f79c-41a6-bd91-6ec5969e1f63-dshm\") pod \"ipp-route-f0e8b9-kserve-7f4f866ccc-pz7w2\" (UID: \"02251d15-f79c-41a6-bd91-6ec5969e1f63\") " pod="ai-tenant-e2e-ipp-a-f0e8b9/ipp-route-f0e8b9-kserve-7f4f866ccc-pz7w2" Oct 30 14:44:39.914261 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:39.914239 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02251d15-f79c-41a6-bd91-6ec5969e1f63-tls-certs\") pod \"ipp-route-f0e8b9-kserve-7f4f866ccc-pz7w2\" (UID: \"02251d15-f79c-41a6-bd91-6ec5969e1f63\") " pod="ai-tenant-e2e-ipp-a-f0e8b9/ipp-route-f0e8b9-kserve-7f4f866ccc-pz7w2" Oct 30 14:44:39.937333 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:39.937306 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8szkf\" (UniqueName: \"kubernetes.io/projected/02251d15-f79c-41a6-bd91-6ec5969e1f63-kube-api-access-8szkf\") pod \"ipp-route-f0e8b9-kserve-7f4f866ccc-pz7w2\" (UID: \"02251d15-f79c-41a6-bd91-6ec5969e1f63\") " pod="ai-tenant-e2e-ipp-a-f0e8b9/ipp-route-f0e8b9-kserve-7f4f866ccc-pz7w2" Oct 30 14:44:39.947912 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:39.947891 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90b406-56b44b6b55-9nd98" Oct 30 14:44:40.210655 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:40.210611 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-f0e8b9/ipp-route-f0e8b9-kserve-7f4f866ccc-pz7w2" Oct 30 14:44:40.210970 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:40.210806 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:44:40.622250 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:44:40.622210 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod02251d15_f79c_41a6_bd91_6ec5969e1f63.slice/crio-43559eb3b91b25bbb372c292d0169796798607c38c6a77eeeed0541c9dc60529 WatchSource:0}: Error finding container 43559eb3b91b25bbb372c292d0169796798607c38c6a77eeeed0541c9dc60529: Status 404 returned error can't find the container with id 43559eb3b91b25bbb372c292d0169796798607c38c6a77eeeed0541c9dc60529 Oct 30 14:44:40.622600 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:40.622569 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-f0e8b9/ipp-route-f0e8b9-kserve-7f4f866ccc-pz7w2"] Oct 30 14:44:40.991114 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:40.991027 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-f0e8b9/ipp-route-f0e8b9-kserve-7f4f866ccc-pz7w2" event={"ID":"02251d15-f79c-41a6-bd91-6ec5969e1f63","Type":"ContainerStarted","Data":"c9868e75fcb2c0122465e1018e29d33fea5f6ce793c786e9bebda6752d728fd3"} Oct 30 14:44:40.991114 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:40.991087 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-f0e8b9/ipp-route-f0e8b9-kserve-7f4f866ccc-pz7w2" event={"ID":"02251d15-f79c-41a6-bd91-6ec5969e1f63","Type":"ContainerStarted","Data":"43559eb3b91b25bbb372c292d0169796798607c38c6a77eeeed0541c9dc60529"} Oct 30 14:44:41.037558 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:41.037509 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-f0e8b9/ipp-route-f0e8b9-kserve-7f4f866ccc-pz7w2" podStartSLOduration=2.037495304 podStartE2EDuration="2.037495304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:44:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:44:41.035211441 +0000 UTC m=+3350.779345482" watchObservedRunningTime="2026-10-30 14:44:41.037495304 +0000 UTC m=+3350.781629344" Oct 30 14:44:41.211814 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:41.211774 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-f0e8b9/ipp-route-f0e8b9-kserve-7f4f866ccc-pz7w2" Oct 30 14:44:42.137022 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:42.136991 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:44:43.922824 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:43.922791 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:44:45.516515 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:45.516485 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:44:46.259535 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:46.259502 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:44:46.389361 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:46.389323 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e50e63-df24z"] Oct 30 14:44:46.400903 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:46.400871 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e50e63-df24z"] Oct 30 14:44:46.421395 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:46.419043 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:44:46.600987 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:46.600943 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:44:46.896983 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:46.896902 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:44:47.208229 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:47.208142 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:44:48.224603 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:48.224568 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:44:48.457304 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:48.457265 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e4c87c4-5edb-4292-9201-8ae74f5e3e60" path="/var/lib/kubelet/pods/2e4c87c4-5edb-4292-9201-8ae74f5e3e60/volumes" Oct 30 14:44:48.843768 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:48.843727 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d05e21-5767b6546c-hmlcp"] Oct 30 14:44:48.882235 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:48.882203 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:44:48.980150 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:48.980116 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-acefbb-8bwwk"] Oct 30 14:44:48.985421 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:48.985394 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-acefbb-8bwwk"] Oct 30 14:44:49.050519 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:49.050483 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:44:49.206710 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:49.199953 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:44:50.211564 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:50.211530 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-f0e8b9/ipp-route-f0e8b9-kserve-7f4f866ccc-pz7w2" Oct 30 14:44:50.224440 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:50.224415 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-f0e8b9/ipp-route-f0e8b9-kserve-7f4f866ccc-pz7w2" Oct 30 14:44:50.403847 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:50.403817 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06340c7e-9397-4280-bd4f-6c540ca20f73" path="/var/lib/kubelet/pods/06340c7e-9397-4280-bd4f-6c540ca20f73/volumes" Oct 30 14:44:50.532524 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:50.532446 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:44:50.970668 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:50.970600 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:44:51.100394 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:51.100361 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-f0e8b9/ipp-route-f0e8b9-kserve-7f4f866ccc-pz7w2" Oct 30 14:44:52.957160 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:52.957095 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:44:53.844145 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:53.844091 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d05e21-5767b6546c-hmlcp" podUID="1a79c636-bf50-4835-9896-916723340900" containerName="payload-pre-processing" containerID="cri-o://c23c1d0a6b2b6d45c5eb528d449245a7ce5a5802168c9724fd4df9eeedb4170e" gracePeriod=25 Oct 30 14:44:54.117660 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:54.117563 4982 generic.go:356] "Generic (PLEG): container finished" podID="1a79c636-bf50-4835-9896-916723340900" containerID="c23c1d0a6b2b6d45c5eb528d449245a7ce5a5802168c9724fd4df9eeedb4170e" exitCode=0 Oct 30 14:44:54.118089 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:54.117751 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d05e21-5767b6546c-hmlcp" event={"ID":"1a79c636-bf50-4835-9896-916723340900","Type":"ContainerDied","Data":"c23c1d0a6b2b6d45c5eb528d449245a7ce5a5802168c9724fd4df9eeedb4170e"} Oct 30 14:44:54.159406 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:54.159373 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d05e21-5767b6546c-hmlcp" Oct 30 14:44:54.173587 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:54.173564 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a79c636-bf50-4835-9896-916723340900-kube-api-access-bs28q\" (UniqueName: \"kubernetes.io/projected/1a79c636-bf50-4835-9896-916723340900-kube-api-access-bs28q\") pod \"1a79c636-bf50-4835-9896-916723340900\" (UID: \"1a79c636-bf50-4835-9896-916723340900\") " Oct 30 14:44:54.173769 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:54.173731 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a79c636-bf50-4835-9896-916723340900-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a79c636-bf50-4835-9896-916723340900-plugins-config-volume\") pod \"1a79c636-bf50-4835-9896-916723340900\" (UID: \"1a79c636-bf50-4835-9896-916723340900\") " Oct 30 14:44:54.174132 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:54.174106 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a79c636-bf50-4835-9896-916723340900-plugins-config-volume" pod "1a79c636-bf50-4835-9896-916723340900" (UID: "1a79c636-bf50-4835-9896-916723340900"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:44:54.174313 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:54.174297 4982 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a79c636-bf50-4835-9896-916723340900-plugins-config-volume\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:44:54.175761 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:54.175737 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a79c636-bf50-4835-9896-916723340900-kube-api-access-bs28q" pod "1a79c636-bf50-4835-9896-916723340900" (UID: "1a79c636-bf50-4835-9896-916723340900"). InnerVolumeSpecName "kube-api-access-bs28q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:44:54.275991 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:54.275963 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bs28q\" (UniqueName: \"kubernetes.io/projected/1a79c636-bf50-4835-9896-916723340900-kube-api-access-bs28q\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:44:54.494886 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:54.494847 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:44:54.647444 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:54.647398 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:44:54.810235 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:54.810195 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6"] Oct 30 14:44:54.810881 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:54.810831 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6" podUID="f6921ba6-c364-4314-8c37-b7a32e2b3be6" containerName="istio-proxy" containerID="cri-o://b0cee75778a4ca2eada123d92a17cf36a2091f7329210fa6d3e582dca109c0c0" gracePeriod=30 Oct 30 14:44:54.843187 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:54.843159 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:44:55.127888 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:55.127811 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d05e21-5767b6546c-hmlcp" Oct 30 14:44:55.127888 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:55.127837 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d05e21-5767b6546c-hmlcp" event={"ID":"1a79c636-bf50-4835-9896-916723340900","Type":"ContainerDied","Data":"d02bd3f2d447d664546ed4aef35822cb719890b22731676743da5b9f0bcaa989"} Oct 30 14:44:55.128425 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:55.127905 4982 scope.go:122] "RemoveContainer" containerID="c23c1d0a6b2b6d45c5eb528d449245a7ce5a5802168c9724fd4df9eeedb4170e" Oct 30 14:44:55.191697 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:55.191627 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d05e21-5767b6546c-hmlcp"] Oct 30 14:44:55.198388 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:55.198361 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d05e21-5767b6546c-hmlcp"] Oct 30 14:44:55.354579 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:55.354550 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:44:55.496518 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:55.496384 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:44:56.404702 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:56.404654 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a79c636-bf50-4835-9896-916723340900" path="/var/lib/kubelet/pods/1a79c636-bf50-4835-9896-916723340900/volumes" Oct 30 14:44:57.477354 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:57.477316 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:44:58.014552 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:44:58.014513 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:44:58.427224 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:58.427182 4982 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 14:44:58.427422 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:58.427245 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6" podUID="f6921ba6-c364-4314-8c37-b7a32e2b3be6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 14:44:58.599807 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:58.599765 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:44:58.743738 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:58.743615 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:44:58.897790 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:58.897751 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754"] Oct 30 14:44:58.898596 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:58.898556 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754" podUID="bd0cec81-8246-425e-8b19-8a3f2937dabc" containerName="istio-proxy" containerID="cri-o://242b6205c94f85c84c6820c6e46e7a4f482fbc439aed3e6d47f6a3873a0daf66" gracePeriod=30 Oct 30 14:44:58.954494 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:44:58.954464 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:45:00.118447 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.118413 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6" Oct 30 14:45:00.171195 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.171166 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6e91df-29889525-cw745"] Oct 30 14:45:00.173407 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.173362 4982 cpu_manager.go:414] "containerMap: removing container" podUID="1a79c636-bf50-4835-9896-916723340900" containerName="payload-pre-processing" Oct 30 14:45:00.173609 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.173587 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a79c636-bf50-4835-9896-916723340900" containerName="payload-pre-processing" Oct 30 14:45:00.173834 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.173814 4982 cpu_manager.go:414] "containerMap: removing container" podUID="f6921ba6-c364-4314-8c37-b7a32e2b3be6" containerName="istio-proxy" Oct 30 14:45:00.173834 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.173835 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6921ba6-c364-4314-8c37-b7a32e2b3be6" containerName="istio-proxy" Oct 30 14:45:00.174158 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.174125 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6921ba6-c364-4314-8c37-b7a32e2b3be6" containerName="istio-proxy" Oct 30 14:45:00.174251 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.174178 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a79c636-bf50-4835-9896-916723340900" containerName="payload-pre-processing" Oct 30 14:45:00.178289 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.178259 4982 generic.go:356] "Generic (PLEG): container finished" podID="f6921ba6-c364-4314-8c37-b7a32e2b3be6" containerID="b0cee75778a4ca2eada123d92a17cf36a2091f7329210fa6d3e582dca109c0c0" exitCode=0 Oct 30 14:45:00.178533 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.178519 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6" Oct 30 14:45:00.189870 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.189842 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-1d6347-29889525-dqf8x"] Oct 30 14:45:00.190668 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.190080 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6e91df-29889525-cw745" Oct 30 14:45:00.199385 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.199359 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6" event={"ID":"f6921ba6-c364-4314-8c37-b7a32e2b3be6","Type":"ContainerDied","Data":"b0cee75778a4ca2eada123d92a17cf36a2091f7329210fa6d3e582dca109c0c0"} Oct 30 14:45:00.199526 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.199509 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6" event={"ID":"f6921ba6-c364-4314-8c37-b7a32e2b3be6","Type":"ContainerDied","Data":"3a7a04fe49703bd0105f561e9746efd7368f439dda72bd0d1bad99a2ea490535"} Oct 30 14:45:00.201246 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.201224 4982 scope.go:122] "RemoveContainer" containerID="b0cee75778a4ca2eada123d92a17cf36a2091f7329210fa6d3e582dca109c0c0" Oct 30 14:45:00.201559 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.201530 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-1d6347-29889525-dqf8x" Oct 30 14:45:00.221481 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.220043 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6921ba6-c364-4314-8c37-b7a32e2b3be6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6921ba6-c364-4314-8c37-b7a32e2b3be6-workload-socket\") pod \"f6921ba6-c364-4314-8c37-b7a32e2b3be6\" (UID: \"f6921ba6-c364-4314-8c37-b7a32e2b3be6\") " Oct 30 14:45:00.221481 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.220128 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6921ba6-c364-4314-8c37-b7a32e2b3be6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6921ba6-c364-4314-8c37-b7a32e2b3be6-workload-certs\") pod \"f6921ba6-c364-4314-8c37-b7a32e2b3be6\" (UID: \"f6921ba6-c364-4314-8c37-b7a32e2b3be6\") " Oct 30 14:45:00.221481 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.220185 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6921ba6-c364-4314-8c37-b7a32e2b3be6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6921ba6-c364-4314-8c37-b7a32e2b3be6-istiod-ca-cert\") pod \"f6921ba6-c364-4314-8c37-b7a32e2b3be6\" (UID: \"f6921ba6-c364-4314-8c37-b7a32e2b3be6\") " Oct 30 14:45:00.221481 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.220235 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6921ba6-c364-4314-8c37-b7a32e2b3be6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6921ba6-c364-4314-8c37-b7a32e2b3be6-istio-data\") pod \"f6921ba6-c364-4314-8c37-b7a32e2b3be6\" (UID: \"f6921ba6-c364-4314-8c37-b7a32e2b3be6\") " Oct 30 14:45:00.221481 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.220324 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6921ba6-c364-4314-8c37-b7a32e2b3be6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6921ba6-c364-4314-8c37-b7a32e2b3be6-istio-envoy\") pod \"f6921ba6-c364-4314-8c37-b7a32e2b3be6\" (UID: \"f6921ba6-c364-4314-8c37-b7a32e2b3be6\") " Oct 30 14:45:00.221481 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.220370 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f6921ba6-c364-4314-8c37-b7a32e2b3be6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6921ba6-c364-4314-8c37-b7a32e2b3be6-istio-podinfo\") pod \"f6921ba6-c364-4314-8c37-b7a32e2b3be6\" (UID: \"f6921ba6-c364-4314-8c37-b7a32e2b3be6\") " Oct 30 14:45:00.221481 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.220416 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6921ba6-c364-4314-8c37-b7a32e2b3be6-kube-api-access-cgmdb\" (UniqueName: \"kubernetes.io/projected/f6921ba6-c364-4314-8c37-b7a32e2b3be6-kube-api-access-cgmdb\") pod \"f6921ba6-c364-4314-8c37-b7a32e2b3be6\" (UID: \"f6921ba6-c364-4314-8c37-b7a32e2b3be6\") " Oct 30 14:45:00.221481 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.220583 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6921ba6-c364-4314-8c37-b7a32e2b3be6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6921ba6-c364-4314-8c37-b7a32e2b3be6-credential-socket\") pod \"f6921ba6-c364-4314-8c37-b7a32e2b3be6\" (UID: \"f6921ba6-c364-4314-8c37-b7a32e2b3be6\") " Oct 30 14:45:00.221481 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.220612 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6921ba6-c364-4314-8c37-b7a32e2b3be6-istio-token\" (UniqueName: \"kubernetes.io/projected/f6921ba6-c364-4314-8c37-b7a32e2b3be6-istio-token\") pod \"f6921ba6-c364-4314-8c37-b7a32e2b3be6\" (UID: \"f6921ba6-c364-4314-8c37-b7a32e2b3be6\") " Oct 30 14:45:00.225364 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.225341 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6921ba6-c364-4314-8c37-b7a32e2b3be6-istio-data" pod "f6921ba6-c364-4314-8c37-b7a32e2b3be6" (UID: "f6921ba6-c364-4314-8c37-b7a32e2b3be6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:45:00.226043 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.226021 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6921ba6-c364-4314-8c37-b7a32e2b3be6-workload-socket" pod "f6921ba6-c364-4314-8c37-b7a32e2b3be6" (UID: "f6921ba6-c364-4314-8c37-b7a32e2b3be6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:45:00.226547 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.226529 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6921ba6-c364-4314-8c37-b7a32e2b3be6-workload-certs" pod "f6921ba6-c364-4314-8c37-b7a32e2b3be6" (UID: "f6921ba6-c364-4314-8c37-b7a32e2b3be6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:45:00.227251 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.227226 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6921ba6-c364-4314-8c37-b7a32e2b3be6-istiod-ca-cert" pod "f6921ba6-c364-4314-8c37-b7a32e2b3be6" (UID: "f6921ba6-c364-4314-8c37-b7a32e2b3be6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:45:00.230071 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.229795 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6921ba6-c364-4314-8c37-b7a32e2b3be6-istio-token" pod "f6921ba6-c364-4314-8c37-b7a32e2b3be6" (UID: "f6921ba6-c364-4314-8c37-b7a32e2b3be6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:45:00.231121 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.231084 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6921ba6-c364-4314-8c37-b7a32e2b3be6-credential-socket" pod "f6921ba6-c364-4314-8c37-b7a32e2b3be6" (UID: "f6921ba6-c364-4314-8c37-b7a32e2b3be6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:45:00.238858 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.238832 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6921ba6-c364-4314-8c37-b7a32e2b3be6-kube-api-access-cgmdb" pod "f6921ba6-c364-4314-8c37-b7a32e2b3be6" (UID: "f6921ba6-c364-4314-8c37-b7a32e2b3be6"). InnerVolumeSpecName "kube-api-access-cgmdb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:45:00.238980 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.238950 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6921ba6-c364-4314-8c37-b7a32e2b3be6-istio-envoy" pod "f6921ba6-c364-4314-8c37-b7a32e2b3be6" (UID: "f6921ba6-c364-4314-8c37-b7a32e2b3be6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:45:00.239689 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.239665 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889525-hz26s"] Oct 30 14:45:00.241061 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.241036 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f6921ba6-c364-4314-8c37-b7a32e2b3be6-istio-podinfo" pod "f6921ba6-c364-4314-8c37-b7a32e2b3be6" (UID: "f6921ba6-c364-4314-8c37-b7a32e2b3be6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:45:00.259864 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.259834 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6e91df-29889525-cw745"] Oct 30 14:45:00.260012 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.259996 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889525-hz26s" Oct 30 14:45:00.272125 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.272065 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-1d6347-29889525-dqf8x"] Oct 30 14:45:00.278120 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.278085 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889525-pjgvt"] Oct 30 14:45:00.282989 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.282715 4982 scope.go:122] "RemoveContainer" containerID="b0cee75778a4ca2eada123d92a17cf36a2091f7329210fa6d3e582dca109c0c0" Oct 30 14:45:00.286547 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:45:00.286516 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0cee75778a4ca2eada123d92a17cf36a2091f7329210fa6d3e582dca109c0c0\": container with ID starting with b0cee75778a4ca2eada123d92a17cf36a2091f7329210fa6d3e582dca109c0c0 not found: ID does not exist" containerID="b0cee75778a4ca2eada123d92a17cf36a2091f7329210fa6d3e582dca109c0c0" Oct 30 14:45:00.286680 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.286561 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0cee75778a4ca2eada123d92a17cf36a2091f7329210fa6d3e582dca109c0c0"} err="failed to get container status \"b0cee75778a4ca2eada123d92a17cf36a2091f7329210fa6d3e582dca109c0c0\": rpc error: code = NotFound desc = could not find container \"b0cee75778a4ca2eada123d92a17cf36a2091f7329210fa6d3e582dca109c0c0\": container with ID starting with b0cee75778a4ca2eada123d92a17cf36a2091f7329210fa6d3e582dca109c0c0 not found: ID does not exist" Oct 30 14:45:00.287462 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.287416 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889525-hz26s"] Oct 30 14:45:00.287564 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.287465 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889525-hvqhb"] Oct 30 14:45:00.291855 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.291830 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889525-pjgvt" Oct 30 14:45:00.314690 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.305229 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:45:00.314690 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.312964 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889525-pjzwp"] Oct 30 14:45:00.331682 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.317769 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889525-pjgvt"] Oct 30 14:45:00.331682 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.317899 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889525-hvqhb" Oct 30 14:45:00.331682 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.318768 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889525-pjzwp" Oct 30 14:45:00.331682 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.324511 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcs2t\" (UniqueName: \"kubernetes.io/projected/1a89d4da-0eef-472e-bf20-613e750dd598-kube-api-access-hcs2t\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-1d6347-29889525-dqf8x\" (UID: \"1a89d4da-0eef-472e-bf20-613e750dd598\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-1d6347-29889525-dqf8x" Oct 30 14:45:00.331682 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.324617 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a89d4da-0eef-472e-bf20-613e750dd598-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-1d6347-29889525-dqf8x\" (UID: \"1a89d4da-0eef-472e-bf20-613e750dd598\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-1d6347-29889525-dqf8x" Oct 30 14:45:00.331682 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.324687 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtnph\" (UniqueName: \"kubernetes.io/projected/f95c53ac-ceef-47a4-abe7-aed7fba44238-kube-api-access-gtnph\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-6e91df-29889525-cw745\" (UID: \"f95c53ac-ceef-47a4-abe7-aed7fba44238\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6e91df-29889525-cw745" Oct 30 14:45:00.331682 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.324753 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f95c53ac-ceef-47a4-abe7-aed7fba44238-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-6e91df-29889525-cw745\" (UID: \"f95c53ac-ceef-47a4-abe7-aed7fba44238\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6e91df-29889525-cw745" Oct 30 14:45:00.331682 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.324992 4982 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6921ba6-c364-4314-8c37-b7a32e2b3be6-credential-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:00.331682 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.325008 4982 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6921ba6-c364-4314-8c37-b7a32e2b3be6-istio-token\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:00.331682 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.325022 4982 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6921ba6-c364-4314-8c37-b7a32e2b3be6-workload-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:00.331682 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.325034 4982 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6921ba6-c364-4314-8c37-b7a32e2b3be6-workload-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:00.331682 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.325047 4982 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6921ba6-c364-4314-8c37-b7a32e2b3be6-istiod-ca-cert\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:00.331682 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.325061 4982 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6921ba6-c364-4314-8c37-b7a32e2b3be6-istio-data\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:00.331682 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.325072 4982 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6921ba6-c364-4314-8c37-b7a32e2b3be6-istio-envoy\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:00.331682 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.325083 4982 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6921ba6-c364-4314-8c37-b7a32e2b3be6-istio-podinfo\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:00.331682 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.325096 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgmdb\" (UniqueName: \"kubernetes.io/projected/f6921ba6-c364-4314-8c37-b7a32e2b3be6-kube-api-access-cgmdb\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:00.334522 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.334499 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889525-hvqhb"] Oct 30 14:45:00.340047 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.340020 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889525-pjzwp"] Oct 30 14:45:00.344464 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.344415 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-33c209-29889525-fj5nd"] Oct 30 14:45:00.351929 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.351913 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-33c209-29889525-fj5nd" Oct 30 14:45:00.354308 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.353749 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889525-bfv9w"] Oct 30 14:45:00.358851 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.358830 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-33c209-29889525-fj5nd"] Oct 30 14:45:00.358984 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.358902 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889525-bfv9w" Oct 30 14:45:00.364663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.364630 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2927fd-29889525-srgtf"] Oct 30 14:45:00.383898 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.383876 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889525-bfv9w"] Oct 30 14:45:00.384081 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.384067 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn"] Oct 30 14:45:00.384192 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.384181 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8"] Oct 30 14:45:00.384635 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.384613 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2927fd-29889525-srgtf" Oct 30 14:45:00.390234 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.390107 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3b95dd-29889525-mkkbq"] Oct 30 14:45:00.401800 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.401775 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3b95dd-29889525-mkkbq" Oct 30 14:45:00.424061 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.423441 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2927fd-29889525-srgtf"] Oct 30 14:45:00.424061 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.423544 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3b95dd-29889525-mkkbq"] Oct 30 14:45:00.426372 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.426345 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e292b6c-6d65-41cd-a634-0a33f9cd86a4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889525-hz26s\" (UID: \"4e292b6c-6d65-41cd-a634-0a33f9cd86a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889525-hz26s" Oct 30 14:45:00.426585 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.426563 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pkt4\" (UniqueName: \"kubernetes.io/projected/4e292b6c-6d65-41cd-a634-0a33f9cd86a4-kube-api-access-7pkt4\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889525-hz26s\" (UID: \"4e292b6c-6d65-41cd-a634-0a33f9cd86a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889525-hz26s" Oct 30 14:45:00.426838 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.426818 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df8648aa-12d5-4d90-88ad-e7a1a8333142-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-ca229c-29889525-hvqhb\" (UID: \"df8648aa-12d5-4d90-88ad-e7a1a8333142\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889525-hvqhb" Oct 30 14:45:00.426969 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.426952 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ca2abf9-60f0-489a-99fa-70a5f4d44def-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889525-pjgvt\" (UID: \"1ca2abf9-60f0-489a-99fa-70a5f4d44def\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889525-pjgvt" Oct 30 14:45:00.427086 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.427072 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcs2t\" (UniqueName: \"kubernetes.io/projected/1a89d4da-0eef-472e-bf20-613e750dd598-kube-api-access-hcs2t\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-1d6347-29889525-dqf8x\" (UID: \"1a89d4da-0eef-472e-bf20-613e750dd598\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-1d6347-29889525-dqf8x" Oct 30 14:45:00.427183 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.427169 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/381a4942-5434-499a-9d21-6e07dce9926b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-46b13b-29889525-pjzwp\" (UID: \"381a4942-5434-499a-9d21-6e07dce9926b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889525-pjzwp" Oct 30 14:45:00.427357 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.427342 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a89d4da-0eef-472e-bf20-613e750dd598-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-1d6347-29889525-dqf8x\" (UID: \"1a89d4da-0eef-472e-bf20-613e750dd598\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-1d6347-29889525-dqf8x" Oct 30 14:45:00.427481 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.427468 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtnph\" (UniqueName: \"kubernetes.io/projected/f95c53ac-ceef-47a4-abe7-aed7fba44238-kube-api-access-gtnph\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-6e91df-29889525-cw745\" (UID: \"f95c53ac-ceef-47a4-abe7-aed7fba44238\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6e91df-29889525-cw745" Oct 30 14:45:00.427612 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.427597 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-544ll\" (UniqueName: \"kubernetes.io/projected/1ca2abf9-60f0-489a-99fa-70a5f4d44def-kube-api-access-544ll\") pod \"maas-api-key-cleanup-29889525-pjgvt\" (UID: \"1ca2abf9-60f0-489a-99fa-70a5f4d44def\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889525-pjgvt" Oct 30 14:45:00.427747 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.427732 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f95c53ac-ceef-47a4-abe7-aed7fba44238-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-6e91df-29889525-cw745\" (UID: \"f95c53ac-ceef-47a4-abe7-aed7fba44238\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6e91df-29889525-cw745" Oct 30 14:45:00.427838 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.427825 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgnd5\" (UniqueName: \"kubernetes.io/projected/381a4942-5434-499a-9d21-6e07dce9926b-kube-api-access-qgnd5\") pod \"maas-api-key-cleanup-e2e-worker-w4-46b13b-29889525-pjzwp\" (UID: \"381a4942-5434-499a-9d21-6e07dce9926b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889525-pjzwp" Oct 30 14:45:00.427961 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.427947 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dqbc\" (UniqueName: \"kubernetes.io/projected/df8648aa-12d5-4d90-88ad-e7a1a8333142-kube-api-access-9dqbc\") pod \"maas-api-key-cleanup-e2e-worker-w1-ca229c-29889525-hvqhb\" (UID: \"df8648aa-12d5-4d90-88ad-e7a1a8333142\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889525-hvqhb" Oct 30 14:45:00.429204 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.429183 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a89d4da-0eef-472e-bf20-613e750dd598-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-1d6347-29889525-dqf8x\" (UID: \"1a89d4da-0eef-472e-bf20-613e750dd598\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-1d6347-29889525-dqf8x" Oct 30 14:45:00.429525 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.429505 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f95c53ac-ceef-47a4-abe7-aed7fba44238-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-6e91df-29889525-cw745\" (UID: \"f95c53ac-ceef-47a4-abe7-aed7fba44238\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6e91df-29889525-cw745" Oct 30 14:45:00.450974 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.450950 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:45:00.459304 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.458806 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcs2t\" (UniqueName: \"kubernetes.io/projected/1a89d4da-0eef-472e-bf20-613e750dd598-kube-api-access-hcs2t\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-1d6347-29889525-dqf8x\" (UID: \"1a89d4da-0eef-472e-bf20-613e750dd598\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-1d6347-29889525-dqf8x" Oct 30 14:45:00.461619 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.461591 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtnph\" (UniqueName: \"kubernetes.io/projected/f95c53ac-ceef-47a4-abe7-aed7fba44238-kube-api-access-gtnph\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-6e91df-29889525-cw745\" (UID: \"f95c53ac-ceef-47a4-abe7-aed7fba44238\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6e91df-29889525-cw745" Oct 30 14:45:00.529039 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.528998 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d524b057-6219-43cf-a11b-5048f36a8dfd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-33c209-29889525-fj5nd\" (UID: \"d524b057-6219-43cf-a11b-5048f36a8dfd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-33c209-29889525-fj5nd" Oct 30 14:45:00.529252 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.529079 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6dd7aa1-d123-4c72-889d-8a6371b7b483-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-00926c-29889525-bfv9w\" (UID: \"c6dd7aa1-d123-4c72-889d-8a6371b7b483\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889525-bfv9w" Oct 30 14:45:00.529252 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.529114 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-544ll\" (UniqueName: \"kubernetes.io/projected/1ca2abf9-60f0-489a-99fa-70a5f4d44def-kube-api-access-544ll\") pod \"maas-api-key-cleanup-29889525-pjgvt\" (UID: \"1ca2abf9-60f0-489a-99fa-70a5f4d44def\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889525-pjgvt" Oct 30 14:45:00.529390 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.529261 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgnd5\" (UniqueName: \"kubernetes.io/projected/381a4942-5434-499a-9d21-6e07dce9926b-kube-api-access-qgnd5\") pod \"maas-api-key-cleanup-e2e-worker-w4-46b13b-29889525-pjzwp\" (UID: \"381a4942-5434-499a-9d21-6e07dce9926b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889525-pjzwp" Oct 30 14:45:00.529390 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.529320 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/620719c8-c4d6-4f56-a843-24137f57fec0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-2927fd-29889525-srgtf\" (UID: \"620719c8-c4d6-4f56-a843-24137f57fec0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2927fd-29889525-srgtf" Oct 30 14:45:00.529390 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.529349 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dqbc\" (UniqueName: \"kubernetes.io/projected/df8648aa-12d5-4d90-88ad-e7a1a8333142-kube-api-access-9dqbc\") pod \"maas-api-key-cleanup-e2e-worker-w1-ca229c-29889525-hvqhb\" (UID: \"df8648aa-12d5-4d90-88ad-e7a1a8333142\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889525-hvqhb" Oct 30 14:45:00.529575 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.529392 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zmgz\" (UniqueName: \"kubernetes.io/projected/62a23269-9415-4a1f-b459-c6bb4a06cbae-kube-api-access-5zmgz\") pod \"maas-api-key-cleanup-e2e-ipp-b-3b95dd-29889525-mkkbq\" (UID: \"62a23269-9415-4a1f-b459-c6bb4a06cbae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3b95dd-29889525-mkkbq" Oct 30 14:45:00.529655 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.529608 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d8v5\" (UniqueName: \"kubernetes.io/projected/d524b057-6219-43cf-a11b-5048f36a8dfd-kube-api-access-4d8v5\") pod \"maas-api-key-cleanup-e2e-ipp-a-33c209-29889525-fj5nd\" (UID: \"d524b057-6219-43cf-a11b-5048f36a8dfd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-33c209-29889525-fj5nd" Oct 30 14:45:00.529710 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.529659 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z7vp\" (UniqueName: \"kubernetes.io/projected/c6dd7aa1-d123-4c72-889d-8a6371b7b483-kube-api-access-9z7vp\") pod \"maas-api-key-cleanup-e2e-worker-w3-00926c-29889525-bfv9w\" (UID: \"c6dd7aa1-d123-4c72-889d-8a6371b7b483\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889525-bfv9w" Oct 30 14:45:00.529747 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.529722 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e292b6c-6d65-41cd-a634-0a33f9cd86a4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889525-hz26s\" (UID: \"4e292b6c-6d65-41cd-a634-0a33f9cd86a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889525-hz26s" Oct 30 14:45:00.529781 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.529772 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62a23269-9415-4a1f-b459-c6bb4a06cbae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-3b95dd-29889525-mkkbq\" (UID: \"62a23269-9415-4a1f-b459-c6bb4a06cbae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3b95dd-29889525-mkkbq" Oct 30 14:45:00.529815 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.529795 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pkt4\" (UniqueName: \"kubernetes.io/projected/4e292b6c-6d65-41cd-a634-0a33f9cd86a4-kube-api-access-7pkt4\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889525-hz26s\" (UID: \"4e292b6c-6d65-41cd-a634-0a33f9cd86a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889525-hz26s" Oct 30 14:45:00.529920 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.529899 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df8648aa-12d5-4d90-88ad-e7a1a8333142-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-ca229c-29889525-hvqhb\" (UID: \"df8648aa-12d5-4d90-88ad-e7a1a8333142\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889525-hvqhb" Oct 30 14:45:00.529963 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.529936 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ca2abf9-60f0-489a-99fa-70a5f4d44def-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889525-pjgvt\" (UID: \"1ca2abf9-60f0-489a-99fa-70a5f4d44def\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889525-pjgvt" Oct 30 14:45:00.529998 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.529976 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6e91df-29889525-cw745" Oct 30 14:45:00.529998 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.529975 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4fwb\" (UniqueName: \"kubernetes.io/projected/620719c8-c4d6-4f56-a843-24137f57fec0-kube-api-access-q4fwb\") pod \"maas-api-key-cleanup-e2e-ipp-a-2927fd-29889525-srgtf\" (UID: \"620719c8-c4d6-4f56-a843-24137f57fec0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2927fd-29889525-srgtf" Oct 30 14:45:00.530142 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.530032 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/381a4942-5434-499a-9d21-6e07dce9926b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-46b13b-29889525-pjzwp\" (UID: \"381a4942-5434-499a-9d21-6e07dce9926b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889525-pjzwp" Oct 30 14:45:00.530614 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.530584 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e292b6c-6d65-41cd-a634-0a33f9cd86a4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889525-hz26s\" (UID: \"4e292b6c-6d65-41cd-a634-0a33f9cd86a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889525-hz26s" Oct 30 14:45:00.530785 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.530760 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ca2abf9-60f0-489a-99fa-70a5f4d44def-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889525-pjgvt\" (UID: \"1ca2abf9-60f0-489a-99fa-70a5f4d44def\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889525-pjgvt" Oct 30 14:45:00.530870 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.530850 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/381a4942-5434-499a-9d21-6e07dce9926b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-46b13b-29889525-pjzwp\" (UID: \"381a4942-5434-499a-9d21-6e07dce9926b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889525-pjzwp" Oct 30 14:45:00.531217 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.531190 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df8648aa-12d5-4d90-88ad-e7a1a8333142-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-ca229c-29889525-hvqhb\" (UID: \"df8648aa-12d5-4d90-88ad-e7a1a8333142\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889525-hvqhb" Oct 30 14:45:00.556138 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.556116 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-544ll\" (UniqueName: \"kubernetes.io/projected/1ca2abf9-60f0-489a-99fa-70a5f4d44def-kube-api-access-544ll\") pod \"maas-api-key-cleanup-29889525-pjgvt\" (UID: \"1ca2abf9-60f0-489a-99fa-70a5f4d44def\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889525-pjgvt" Oct 30 14:45:00.557057 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.557033 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dqbc\" (UniqueName: \"kubernetes.io/projected/df8648aa-12d5-4d90-88ad-e7a1a8333142-kube-api-access-9dqbc\") pod \"maas-api-key-cleanup-e2e-worker-w1-ca229c-29889525-hvqhb\" (UID: \"df8648aa-12d5-4d90-88ad-e7a1a8333142\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889525-hvqhb" Oct 30 14:45:00.557139 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.557037 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgnd5\" (UniqueName: \"kubernetes.io/projected/381a4942-5434-499a-9d21-6e07dce9926b-kube-api-access-qgnd5\") pod \"maas-api-key-cleanup-e2e-worker-w4-46b13b-29889525-pjzwp\" (UID: \"381a4942-5434-499a-9d21-6e07dce9926b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889525-pjzwp" Oct 30 14:45:00.557139 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.557074 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pkt4\" (UniqueName: \"kubernetes.io/projected/4e292b6c-6d65-41cd-a634-0a33f9cd86a4-kube-api-access-7pkt4\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889525-hz26s\" (UID: \"4e292b6c-6d65-41cd-a634-0a33f9cd86a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889525-hz26s" Oct 30 14:45:00.609588 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.608426 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-1d6347-29889525-dqf8x" Oct 30 14:45:00.612371 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.612278 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:45:00.627832 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.627628 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889525-hz26s" Oct 30 14:45:00.631200 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.631168 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4fwb\" (UniqueName: \"kubernetes.io/projected/620719c8-c4d6-4f56-a843-24137f57fec0-kube-api-access-q4fwb\") pod \"maas-api-key-cleanup-e2e-ipp-a-2927fd-29889525-srgtf\" (UID: \"620719c8-c4d6-4f56-a843-24137f57fec0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2927fd-29889525-srgtf" Oct 30 14:45:00.631294 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.631283 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d524b057-6219-43cf-a11b-5048f36a8dfd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-33c209-29889525-fj5nd\" (UID: \"d524b057-6219-43cf-a11b-5048f36a8dfd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-33c209-29889525-fj5nd" Oct 30 14:45:00.631398 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.631374 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6dd7aa1-d123-4c72-889d-8a6371b7b483-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-00926c-29889525-bfv9w\" (UID: \"c6dd7aa1-d123-4c72-889d-8a6371b7b483\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889525-bfv9w" Oct 30 14:45:00.631482 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.631464 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/620719c8-c4d6-4f56-a843-24137f57fec0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-2927fd-29889525-srgtf\" (UID: \"620719c8-c4d6-4f56-a843-24137f57fec0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2927fd-29889525-srgtf" Oct 30 14:45:00.631662 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.631596 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zmgz\" (UniqueName: \"kubernetes.io/projected/62a23269-9415-4a1f-b459-c6bb4a06cbae-kube-api-access-5zmgz\") pod \"maas-api-key-cleanup-e2e-ipp-b-3b95dd-29889525-mkkbq\" (UID: \"62a23269-9415-4a1f-b459-c6bb4a06cbae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3b95dd-29889525-mkkbq" Oct 30 14:45:00.632055 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.631978 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d8v5\" (UniqueName: \"kubernetes.io/projected/d524b057-6219-43cf-a11b-5048f36a8dfd-kube-api-access-4d8v5\") pod \"maas-api-key-cleanup-e2e-ipp-a-33c209-29889525-fj5nd\" (UID: \"d524b057-6219-43cf-a11b-5048f36a8dfd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-33c209-29889525-fj5nd" Oct 30 14:45:00.632294 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.632028 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z7vp\" (UniqueName: \"kubernetes.io/projected/c6dd7aa1-d123-4c72-889d-8a6371b7b483-kube-api-access-9z7vp\") pod \"maas-api-key-cleanup-e2e-worker-w3-00926c-29889525-bfv9w\" (UID: \"c6dd7aa1-d123-4c72-889d-8a6371b7b483\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889525-bfv9w" Oct 30 14:45:00.633472 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.633261 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6dd7aa1-d123-4c72-889d-8a6371b7b483-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-00926c-29889525-bfv9w\" (UID: \"c6dd7aa1-d123-4c72-889d-8a6371b7b483\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889525-bfv9w" Oct 30 14:45:00.635220 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.633721 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62a23269-9415-4a1f-b459-c6bb4a06cbae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-3b95dd-29889525-mkkbq\" (UID: \"62a23269-9415-4a1f-b459-c6bb4a06cbae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3b95dd-29889525-mkkbq" Oct 30 14:45:00.635220 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.634259 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/620719c8-c4d6-4f56-a843-24137f57fec0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-2927fd-29889525-srgtf\" (UID: \"620719c8-c4d6-4f56-a843-24137f57fec0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2927fd-29889525-srgtf" Oct 30 14:45:00.635220 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.634714 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62a23269-9415-4a1f-b459-c6bb4a06cbae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-3b95dd-29889525-mkkbq\" (UID: \"62a23269-9415-4a1f-b459-c6bb4a06cbae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3b95dd-29889525-mkkbq" Oct 30 14:45:00.635220 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.634919 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d524b057-6219-43cf-a11b-5048f36a8dfd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-33c209-29889525-fj5nd\" (UID: \"d524b057-6219-43cf-a11b-5048f36a8dfd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-33c209-29889525-fj5nd" Oct 30 14:45:00.635220 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.635024 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889525-pjgvt" Oct 30 14:45:00.655535 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.655506 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889525-hvqhb" Oct 30 14:45:00.663775 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.661753 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z7vp\" (UniqueName: \"kubernetes.io/projected/c6dd7aa1-d123-4c72-889d-8a6371b7b483-kube-api-access-9z7vp\") pod \"maas-api-key-cleanup-e2e-worker-w3-00926c-29889525-bfv9w\" (UID: \"c6dd7aa1-d123-4c72-889d-8a6371b7b483\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889525-bfv9w" Oct 30 14:45:00.669913 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.669888 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zmgz\" (UniqueName: \"kubernetes.io/projected/62a23269-9415-4a1f-b459-c6bb4a06cbae-kube-api-access-5zmgz\") pod \"maas-api-key-cleanup-e2e-ipp-b-3b95dd-29889525-mkkbq\" (UID: \"62a23269-9415-4a1f-b459-c6bb4a06cbae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3b95dd-29889525-mkkbq" Oct 30 14:45:00.673133 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.673085 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4fwb\" (UniqueName: \"kubernetes.io/projected/620719c8-c4d6-4f56-a843-24137f57fec0-kube-api-access-q4fwb\") pod \"maas-api-key-cleanup-e2e-ipp-a-2927fd-29889525-srgtf\" (UID: \"620719c8-c4d6-4f56-a843-24137f57fec0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2927fd-29889525-srgtf" Oct 30 14:45:00.676883 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.676015 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d8v5\" (UniqueName: \"kubernetes.io/projected/d524b057-6219-43cf-a11b-5048f36a8dfd-kube-api-access-4d8v5\") pod \"maas-api-key-cleanup-e2e-ipp-a-33c209-29889525-fj5nd\" (UID: \"d524b057-6219-43cf-a11b-5048f36a8dfd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-33c209-29889525-fj5nd" Oct 30 14:45:00.678372 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.678351 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889525-pjzwp" Oct 30 14:45:00.714314 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.713941 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-33c209-29889525-fj5nd" Oct 30 14:45:00.736719 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.736479 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889525-bfv9w" Oct 30 14:45:00.745374 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.745194 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2927fd-29889525-srgtf" Oct 30 14:45:00.765311 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.764244 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3b95dd-29889525-mkkbq" Oct 30 14:45:00.799263 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.797810 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6"] Oct 30 14:45:00.842036 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.842001 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e50e63-openshift-default-f89f68fdc-5zwp6"] Oct 30 14:45:00.899381 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:00.897158 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6e91df-29889525-cw745"] Oct 30 14:45:01.190159 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:01.190115 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-1d6347-29889525-dqf8x"] Oct 30 14:45:01.270710 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:01.269933 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889525-hz26s"] Oct 30 14:45:01.336931 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:01.334679 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889525-pjgvt"] Oct 30 14:45:01.351623 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:45:01.351581 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ca2abf9_60f0_489a_99fa_70a5f4d44def.slice/crio-155cc567fff2ee58a2801da7051c853dea7bc81c01b3f1d038a64a452dc989fe WatchSource:0}: Error finding container 155cc567fff2ee58a2801da7051c853dea7bc81c01b3f1d038a64a452dc989fe: Status 404 returned error can't find the container with id 155cc567fff2ee58a2801da7051c853dea7bc81c01b3f1d038a64a452dc989fe Oct 30 14:45:01.356403 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:01.356322 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-1d6347-29889525-dqf8x" event={"ID":"1a89d4da-0eef-472e-bf20-613e750dd598","Type":"ContainerStarted","Data":"205f09f335e3eebd461c5d161f33b3609f4b10cb6fed3e83a1e08deee161043c"} Oct 30 14:45:01.358805 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:01.358740 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6e91df-29889525-cw745" event={"ID":"f95c53ac-ceef-47a4-abe7-aed7fba44238","Type":"ContainerStarted","Data":"5f884511e91846e16eba8cb09cbfcb33b798b61df5b1c6388384205a5901f406"} Oct 30 14:45:01.369453 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:01.369434 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889525-pjzwp"] Oct 30 14:45:01.387259 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:01.387239 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889525-hvqhb"] Oct 30 14:45:01.398105 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:01.398073 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889525-bfv9w"] Oct 30 14:45:01.398218 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:01.398193 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-33c209-29889525-fj5nd"] Oct 30 14:45:01.444358 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:01.444336 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2927fd-29889525-srgtf"] Oct 30 14:45:01.449496 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:01.449473 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3b95dd-29889525-mkkbq"] Oct 30 14:45:02.369370 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:02.369340 4982 generic.go:356] "Generic (PLEG): container finished" podID="f95c53ac-ceef-47a4-abe7-aed7fba44238" containerID="b5f4d12f40ec77e720e86075a5cf22ca768c510c7d347213abb0c9414942822b" exitCode=60 Oct 30 14:45:02.369767 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:02.369412 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6e91df-29889525-cw745" event={"ID":"f95c53ac-ceef-47a4-abe7-aed7fba44238","Type":"ContainerDied","Data":"b5f4d12f40ec77e720e86075a5cf22ca768c510c7d347213abb0c9414942822b"} Oct 30 14:45:02.369767 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:02.369681 4982 scope.go:122] "RemoveContainer" containerID="b5f4d12f40ec77e720e86075a5cf22ca768c510c7d347213abb0c9414942822b" Oct 30 14:45:02.371036 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:02.371006 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3b95dd-29889525-mkkbq" event={"ID":"62a23269-9415-4a1f-b459-c6bb4a06cbae","Type":"ContainerStarted","Data":"5926fac640544ad205bd99f106244a0e7d4e5d96a8c4463abfc380b1b238498f"} Oct 30 14:45:02.372935 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:02.372896 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889525-bfv9w" event={"ID":"c6dd7aa1-d123-4c72-889d-8a6371b7b483","Type":"ContainerStarted","Data":"755213e9eb86d7bf34026af7c1d812e890a6057d105ffba449ad8ff1215b1d87"} Oct 30 14:45:02.374721 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:02.374692 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889525-hvqhb" event={"ID":"df8648aa-12d5-4d90-88ad-e7a1a8333142","Type":"ContainerStarted","Data":"19edc87057e6a3459ba5f36ac05dd50df051d488b1a558246bc4c98e45d4e731"} Oct 30 14:45:02.374858 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:02.374753 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889525-hvqhb" event={"ID":"df8648aa-12d5-4d90-88ad-e7a1a8333142","Type":"ContainerStarted","Data":"fcce525882d7299a22482a99c1f77860af60a58562c32271df28bcd25a66f487"} Oct 30 14:45:02.376361 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:02.376340 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889525-pjzwp" event={"ID":"381a4942-5434-499a-9d21-6e07dce9926b","Type":"ContainerStarted","Data":"af7855bd58c576f1cd14c604506706a26209517bc16e028cb8b7b9ce50d75b3b"} Oct 30 14:45:02.376501 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:02.376482 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889525-pjzwp" event={"ID":"381a4942-5434-499a-9d21-6e07dce9926b","Type":"ContainerStarted","Data":"47c057534f9672c8d08c4452661c7ba5a1152f7cfe13f0cd98f69a8ae6c5a4cd"} Oct 30 14:45:02.377710 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:02.377684 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-33c209-29889525-fj5nd" event={"ID":"d524b057-6219-43cf-a11b-5048f36a8dfd","Type":"ContainerStarted","Data":"11a42ddea3207c9a8f4051305b59c109a5e1d8c8b40f59c072a8f01657be6ea7"} Oct 30 14:45:02.379374 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:02.379353 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889525-hz26s" event={"ID":"4e292b6c-6d65-41cd-a634-0a33f9cd86a4","Type":"ContainerStarted","Data":"48657d825eb647545d8860dbaedf21ba62d6c50009b1ebc011d53e6099530aa5"} Oct 30 14:45:02.379467 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:02.379390 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889525-hz26s" event={"ID":"4e292b6c-6d65-41cd-a634-0a33f9cd86a4","Type":"ContainerStarted","Data":"40f96a6d6d4aeadc631454a6fe3e22c8bdf7bfffba7c30bf01538b66eab92686"} Oct 30 14:45:02.436085 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:02.436059 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6921ba6-c364-4314-8c37-b7a32e2b3be6" path="/var/lib/kubelet/pods/f6921ba6-c364-4314-8c37-b7a32e2b3be6/volumes" Oct 30 14:45:02.436446 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:02.436426 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2927fd-29889525-srgtf" event={"ID":"620719c8-c4d6-4f56-a843-24137f57fec0","Type":"ContainerStarted","Data":"0f1b5dc625d4ce4791dd4f5a4c0d26a1907a9c8d9c60cc5c5133464a8669ff92"} Oct 30 14:45:02.436503 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:02.436466 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889525-pjgvt" event={"ID":"1ca2abf9-60f0-489a-99fa-70a5f4d44def","Type":"ContainerStarted","Data":"aff52f41431b8af0d73480d95eeef346e7d9f252550630cbb44b2d175b4507f2"} Oct 30 14:45:02.436539 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:02.436506 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889525-pjgvt" event={"ID":"1ca2abf9-60f0-489a-99fa-70a5f4d44def","Type":"ContainerStarted","Data":"155cc567fff2ee58a2801da7051c853dea7bc81c01b3f1d038a64a452dc989fe"} Oct 30 14:45:02.436539 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:02.436525 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-1d6347-29889525-dqf8x" event={"ID":"1a89d4da-0eef-472e-bf20-613e750dd598","Type":"ContainerStarted","Data":"d86bd9dfb0de65a75313a9ac5a87787ae1e26349bf56bd5fde90a4b0278575a4"} Oct 30 14:45:02.472261 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:02.472205 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889525-pjzwp" podStartSLOduration=2.472188767 podStartE2EDuration="2.472188767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:45:02.465716029 +0000 UTC m=+3372.209850069" watchObservedRunningTime="2026-10-30 14:45:02.472188767 +0000 UTC m=+3372.216322811" Oct 30 14:45:02.521209 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:02.521145 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889525-hz26s" podStartSLOduration=2.5211272129999998 podStartE2EDuration="2.521127213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:45:02.516357906 +0000 UTC m=+3372.260491949" watchObservedRunningTime="2026-10-30 14:45:02.521127213 +0000 UTC m=+3372.265261256" Oct 30 14:45:02.562003 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:02.561955 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889525-hvqhb" podStartSLOduration=2.561939003 podStartE2EDuration="2.561939003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:45:02.558960847 +0000 UTC m=+3372.303094888" watchObservedRunningTime="2026-10-30 14:45:02.561939003 +0000 UTC m=+3372.306073045" Oct 30 14:45:02.621663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:02.617744 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-1d6347-29889525-dqf8x" podStartSLOduration=2.617722932 podStartE2EDuration="2.617722932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:45:02.616376208 +0000 UTC m=+3372.360510252" watchObservedRunningTime="2026-10-30 14:45:02.617722932 +0000 UTC m=+3372.361856973" Oct 30 14:45:02.659902 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:02.659852 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889525-pjgvt" podStartSLOduration=2.659840036 podStartE2EDuration="2.659840036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:45:02.659550383 +0000 UTC m=+3372.403684425" watchObservedRunningTime="2026-10-30 14:45:02.659840036 +0000 UTC m=+3372.403974077" Oct 30 14:45:03.446373 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:03.446338 4982 generic.go:356] "Generic (PLEG): container finished" podID="1ca2abf9-60f0-489a-99fa-70a5f4d44def" containerID="aff52f41431b8af0d73480d95eeef346e7d9f252550630cbb44b2d175b4507f2" exitCode=60 Oct 30 14:45:03.446909 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:03.446393 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889525-pjgvt" event={"ID":"1ca2abf9-60f0-489a-99fa-70a5f4d44def","Type":"ContainerDied","Data":"aff52f41431b8af0d73480d95eeef346e7d9f252550630cbb44b2d175b4507f2"} Oct 30 14:45:03.446909 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:03.446839 4982 scope.go:122] "RemoveContainer" containerID="aff52f41431b8af0d73480d95eeef346e7d9f252550630cbb44b2d175b4507f2" Oct 30 14:45:03.448533 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:03.448409 4982 generic.go:356] "Generic (PLEG): container finished" podID="1a89d4da-0eef-472e-bf20-613e750dd598" containerID="d86bd9dfb0de65a75313a9ac5a87787ae1e26349bf56bd5fde90a4b0278575a4" exitCode=60 Oct 30 14:45:03.448533 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:03.448443 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-1d6347-29889525-dqf8x" event={"ID":"1a89d4da-0eef-472e-bf20-613e750dd598","Type":"ContainerDied","Data":"d86bd9dfb0de65a75313a9ac5a87787ae1e26349bf56bd5fde90a4b0278575a4"} Oct 30 14:45:03.448863 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:03.448844 4982 scope.go:122] "RemoveContainer" containerID="d86bd9dfb0de65a75313a9ac5a87787ae1e26349bf56bd5fde90a4b0278575a4" Oct 30 14:45:03.454281 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:03.453721 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6e91df-29889525-cw745" event={"ID":"f95c53ac-ceef-47a4-abe7-aed7fba44238","Type":"ContainerStarted","Data":"6e02d67cd3c2aebc20ee841d4e7ca399272d86f5a56073da51cc679edd62bbd9"} Oct 30 14:45:03.455962 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:03.455924 4982 generic.go:356] "Generic (PLEG): container finished" podID="62a23269-9415-4a1f-b459-c6bb4a06cbae" containerID="760afe08b8c68ff0d1a54122dd4c686a60b180c105e513d03009c5598683bc82" exitCode=60 Oct 30 14:45:03.456065 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:03.456021 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3b95dd-29889525-mkkbq" event={"ID":"62a23269-9415-4a1f-b459-c6bb4a06cbae","Type":"ContainerDied","Data":"760afe08b8c68ff0d1a54122dd4c686a60b180c105e513d03009c5598683bc82"} Oct 30 14:45:03.456412 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:03.456213 4982 scope.go:122] "RemoveContainer" containerID="760afe08b8c68ff0d1a54122dd4c686a60b180c105e513d03009c5598683bc82" Oct 30 14:45:03.458072 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:03.458038 4982 generic.go:356] "Generic (PLEG): container finished" podID="c6dd7aa1-d123-4c72-889d-8a6371b7b483" containerID="b7510d38f44fe8aa8302b4cbb3ebc6110e445efaee93a01bf991e5ca4d4eb2b7" exitCode=60 Oct 30 14:45:03.458171 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:03.458114 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889525-bfv9w" event={"ID":"c6dd7aa1-d123-4c72-889d-8a6371b7b483","Type":"ContainerDied","Data":"b7510d38f44fe8aa8302b4cbb3ebc6110e445efaee93a01bf991e5ca4d4eb2b7"} Oct 30 14:45:03.458371 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:03.458351 4982 scope.go:122] "RemoveContainer" containerID="b7510d38f44fe8aa8302b4cbb3ebc6110e445efaee93a01bf991e5ca4d4eb2b7" Oct 30 14:45:03.461708 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:03.460145 4982 generic.go:356] "Generic (PLEG): container finished" podID="df8648aa-12d5-4d90-88ad-e7a1a8333142" containerID="19edc87057e6a3459ba5f36ac05dd50df051d488b1a558246bc4c98e45d4e731" exitCode=60 Oct 30 14:45:03.461708 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:03.460207 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889525-hvqhb" event={"ID":"df8648aa-12d5-4d90-88ad-e7a1a8333142","Type":"ContainerDied","Data":"19edc87057e6a3459ba5f36ac05dd50df051d488b1a558246bc4c98e45d4e731"} Oct 30 14:45:03.461708 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:03.460467 4982 scope.go:122] "RemoveContainer" containerID="19edc87057e6a3459ba5f36ac05dd50df051d488b1a558246bc4c98e45d4e731" Oct 30 14:45:03.464556 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:03.464534 4982 generic.go:356] "Generic (PLEG): container finished" podID="381a4942-5434-499a-9d21-6e07dce9926b" containerID="af7855bd58c576f1cd14c604506706a26209517bc16e028cb8b7b9ce50d75b3b" exitCode=60 Oct 30 14:45:03.464688 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:03.464625 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889525-pjzwp" event={"ID":"381a4942-5434-499a-9d21-6e07dce9926b","Type":"ContainerDied","Data":"af7855bd58c576f1cd14c604506706a26209517bc16e028cb8b7b9ce50d75b3b"} Oct 30 14:45:03.464973 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:03.464955 4982 scope.go:122] "RemoveContainer" containerID="af7855bd58c576f1cd14c604506706a26209517bc16e028cb8b7b9ce50d75b3b" Oct 30 14:45:03.467032 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:03.467011 4982 generic.go:356] "Generic (PLEG): container finished" podID="d524b057-6219-43cf-a11b-5048f36a8dfd" containerID="f5ebd6e0b20697ef9e6bf8171899c3a5d149034a8d8817c049a88159d6962b79" exitCode=60 Oct 30 14:45:03.467150 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:03.467098 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-33c209-29889525-fj5nd" event={"ID":"d524b057-6219-43cf-a11b-5048f36a8dfd","Type":"ContainerDied","Data":"f5ebd6e0b20697ef9e6bf8171899c3a5d149034a8d8817c049a88159d6962b79"} Oct 30 14:45:03.467401 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:03.467383 4982 scope.go:122] "RemoveContainer" containerID="f5ebd6e0b20697ef9e6bf8171899c3a5d149034a8d8817c049a88159d6962b79" Oct 30 14:45:03.470184 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:03.470162 4982 generic.go:356] "Generic (PLEG): container finished" podID="4e292b6c-6d65-41cd-a634-0a33f9cd86a4" containerID="48657d825eb647545d8860dbaedf21ba62d6c50009b1ebc011d53e6099530aa5" exitCode=60 Oct 30 14:45:03.470710 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:03.470544 4982 scope.go:122] "RemoveContainer" containerID="48657d825eb647545d8860dbaedf21ba62d6c50009b1ebc011d53e6099530aa5" Oct 30 14:45:03.470882 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:03.470861 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889525-hz26s" event={"ID":"4e292b6c-6d65-41cd-a634-0a33f9cd86a4","Type":"ContainerDied","Data":"48657d825eb647545d8860dbaedf21ba62d6c50009b1ebc011d53e6099530aa5"} Oct 30 14:45:03.488632 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:03.488610 4982 generic.go:356] "Generic (PLEG): container finished" podID="620719c8-c4d6-4f56-a843-24137f57fec0" containerID="16e60e85499ccb1a92746ef6604350e9072cc31d246817af4e436b604bd9a334" exitCode=60 Oct 30 14:45:03.488823 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:03.488799 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2927fd-29889525-srgtf" event={"ID":"620719c8-c4d6-4f56-a843-24137f57fec0","Type":"ContainerDied","Data":"16e60e85499ccb1a92746ef6604350e9072cc31d246817af4e436b604bd9a334"} Oct 30 14:45:03.489334 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:03.489318 4982 scope.go:122] "RemoveContainer" containerID="16e60e85499ccb1a92746ef6604350e9072cc31d246817af4e436b604bd9a334" Oct 30 14:45:03.721247 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:03.721190 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6e91df-29889525-cw745" podStartSLOduration=3.721172065 podStartE2EDuration="3.721172065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:45:03.719946662 +0000 UTC m=+3373.464080705" watchObservedRunningTime="2026-10-30 14:45:03.721172065 +0000 UTC m=+3373.465306110" Oct 30 14:45:03.759237 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:03.759168 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:45:04.472821 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:04.472564 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754" Oct 30 14:45:04.557783 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:04.557736 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd0cec81-8246-425e-8b19-8a3f2937dabc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd0cec81-8246-425e-8b19-8a3f2937dabc-credential-socket\") pod \"bd0cec81-8246-425e-8b19-8a3f2937dabc\" (UID: \"bd0cec81-8246-425e-8b19-8a3f2937dabc\") " Oct 30 14:45:04.557956 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:04.557812 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd0cec81-8246-425e-8b19-8a3f2937dabc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd0cec81-8246-425e-8b19-8a3f2937dabc-workload-certs\") pod \"bd0cec81-8246-425e-8b19-8a3f2937dabc\" (UID: \"bd0cec81-8246-425e-8b19-8a3f2937dabc\") " Oct 30 14:45:04.558113 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:04.557954 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd0cec81-8246-425e-8b19-8a3f2937dabc-istio-token\" (UniqueName: \"kubernetes.io/projected/bd0cec81-8246-425e-8b19-8a3f2937dabc-istio-token\") pod \"bd0cec81-8246-425e-8b19-8a3f2937dabc\" (UID: \"bd0cec81-8246-425e-8b19-8a3f2937dabc\") " Oct 30 14:45:04.558113 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:04.558084 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bd0cec81-8246-425e-8b19-8a3f2937dabc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd0cec81-8246-425e-8b19-8a3f2937dabc-istio-podinfo\") pod \"bd0cec81-8246-425e-8b19-8a3f2937dabc\" (UID: \"bd0cec81-8246-425e-8b19-8a3f2937dabc\") " Oct 30 14:45:04.558235 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:04.558185 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd0cec81-8246-425e-8b19-8a3f2937dabc-kube-api-access-cb679\" (UniqueName: \"kubernetes.io/projected/bd0cec81-8246-425e-8b19-8a3f2937dabc-kube-api-access-cb679\") pod \"bd0cec81-8246-425e-8b19-8a3f2937dabc\" (UID: \"bd0cec81-8246-425e-8b19-8a3f2937dabc\") " Oct 30 14:45:04.558296 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:04.558230 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd0cec81-8246-425e-8b19-8a3f2937dabc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd0cec81-8246-425e-8b19-8a3f2937dabc-istiod-ca-cert\") pod \"bd0cec81-8246-425e-8b19-8a3f2937dabc\" (UID: \"bd0cec81-8246-425e-8b19-8a3f2937dabc\") " Oct 30 14:45:04.558296 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:04.558279 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd0cec81-8246-425e-8b19-8a3f2937dabc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd0cec81-8246-425e-8b19-8a3f2937dabc-workload-socket\") pod \"bd0cec81-8246-425e-8b19-8a3f2937dabc\" (UID: \"bd0cec81-8246-425e-8b19-8a3f2937dabc\") " Oct 30 14:45:04.558402 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:04.558314 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd0cec81-8246-425e-8b19-8a3f2937dabc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd0cec81-8246-425e-8b19-8a3f2937dabc-istio-data\") pod \"bd0cec81-8246-425e-8b19-8a3f2937dabc\" (UID: \"bd0cec81-8246-425e-8b19-8a3f2937dabc\") " Oct 30 14:45:04.558668 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:04.558462 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd0cec81-8246-425e-8b19-8a3f2937dabc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd0cec81-8246-425e-8b19-8a3f2937dabc-istio-envoy\") pod \"bd0cec81-8246-425e-8b19-8a3f2937dabc\" (UID: \"bd0cec81-8246-425e-8b19-8a3f2937dabc\") " Oct 30 14:45:04.563732 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:04.563511 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd0cec81-8246-425e-8b19-8a3f2937dabc-credential-socket" pod "bd0cec81-8246-425e-8b19-8a3f2937dabc" (UID: "bd0cec81-8246-425e-8b19-8a3f2937dabc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:45:04.564129 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:04.563892 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd0cec81-8246-425e-8b19-8a3f2937dabc-workload-certs" pod "bd0cec81-8246-425e-8b19-8a3f2937dabc" (UID: "bd0cec81-8246-425e-8b19-8a3f2937dabc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:45:04.564747 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:04.564370 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd0cec81-8246-425e-8b19-8a3f2937dabc-istiod-ca-cert" pod "bd0cec81-8246-425e-8b19-8a3f2937dabc" (UID: "bd0cec81-8246-425e-8b19-8a3f2937dabc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:45:04.564747 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:04.564449 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd0cec81-8246-425e-8b19-8a3f2937dabc-istio-podinfo" pod "bd0cec81-8246-425e-8b19-8a3f2937dabc" (UID: "bd0cec81-8246-425e-8b19-8a3f2937dabc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:45:04.565121 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:04.564953 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd0cec81-8246-425e-8b19-8a3f2937dabc-workload-socket" pod "bd0cec81-8246-425e-8b19-8a3f2937dabc" (UID: "bd0cec81-8246-425e-8b19-8a3f2937dabc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:45:04.566706 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:04.565231 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd0cec81-8246-425e-8b19-8a3f2937dabc-istio-data" pod "bd0cec81-8246-425e-8b19-8a3f2937dabc" (UID: "bd0cec81-8246-425e-8b19-8a3f2937dabc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:45:04.621662 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:04.598784 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-33c209-29889525-fj5nd" event={"ID":"d524b057-6219-43cf-a11b-5048f36a8dfd","Type":"ContainerStarted","Data":"52e69aae9b6a227d176f2c3c37177c43b1a3130ff1aaf4e937e9d1f82e0279d5"} Oct 30 14:45:04.643111 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:04.642053 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd0cec81-8246-425e-8b19-8a3f2937dabc-kube-api-access-cb679" pod "bd0cec81-8246-425e-8b19-8a3f2937dabc" (UID: "bd0cec81-8246-425e-8b19-8a3f2937dabc"). InnerVolumeSpecName "kube-api-access-cb679". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:45:04.643111 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:04.642356 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889525-hz26s" event={"ID":"4e292b6c-6d65-41cd-a634-0a33f9cd86a4","Type":"ContainerStarted","Data":"5d4691d5b0d1f5d91dbb63d34bf1b195caccbd92a44c96df408728c7b19625aa"} Oct 30 14:45:04.643896 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:04.643357 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd0cec81-8246-425e-8b19-8a3f2937dabc-istio-envoy" pod "bd0cec81-8246-425e-8b19-8a3f2937dabc" (UID: "bd0cec81-8246-425e-8b19-8a3f2937dabc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:45:04.650658 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:04.650505 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2927fd-29889525-srgtf" event={"ID":"620719c8-c4d6-4f56-a843-24137f57fec0","Type":"ContainerStarted","Data":"89a2c29a4dd95a6ba99405575ce22f9478035787f1e7817a9fa33d55450b2595"} Oct 30 14:45:04.654377 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:04.654352 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889525-pjgvt" event={"ID":"1ca2abf9-60f0-489a-99fa-70a5f4d44def","Type":"ContainerStarted","Data":"6b33bfb4e0e2f2c2531897378fd2d0039317b537974038224dc04830428392d4"} Oct 30 14:45:04.657670 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:04.657630 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-1d6347-29889525-dqf8x" event={"ID":"1a89d4da-0eef-472e-bf20-613e750dd598","Type":"ContainerStarted","Data":"a69539550df24c2a57356c92ef0f8d52c29860416cd65c3ac0426bc91867d912"} Oct 30 14:45:04.660449 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:04.660420 4982 generic.go:356] "Generic (PLEG): container finished" podID="bd0cec81-8246-425e-8b19-8a3f2937dabc" containerID="242b6205c94f85c84c6820c6e46e7a4f482fbc439aed3e6d47f6a3873a0daf66" exitCode=0 Oct 30 14:45:04.660540 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:04.660515 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754" event={"ID":"bd0cec81-8246-425e-8b19-8a3f2937dabc","Type":"ContainerDied","Data":"242b6205c94f85c84c6820c6e46e7a4f482fbc439aed3e6d47f6a3873a0daf66"} Oct 30 14:45:04.660611 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:04.660552 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754" event={"ID":"bd0cec81-8246-425e-8b19-8a3f2937dabc","Type":"ContainerDied","Data":"f21207fd6ac85850b7e5ea846c2426e124c2ab734973b370b5fa4dd7f7354f6c"} Oct 30 14:45:04.660611 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:04.660582 4982 scope.go:122] "RemoveContainer" containerID="242b6205c94f85c84c6820c6e46e7a4f482fbc439aed3e6d47f6a3873a0daf66" Oct 30 14:45:04.661699 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:04.661677 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb679\" (UniqueName: \"kubernetes.io/projected/bd0cec81-8246-425e-8b19-8a3f2937dabc-kube-api-access-cb679\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:04.661831 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:04.661816 4982 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd0cec81-8246-425e-8b19-8a3f2937dabc-istiod-ca-cert\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:04.661977 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:04.661960 4982 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd0cec81-8246-425e-8b19-8a3f2937dabc-workload-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:04.662092 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:04.662078 4982 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd0cec81-8246-425e-8b19-8a3f2937dabc-istio-data\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:04.662186 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:04.662177 4982 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd0cec81-8246-425e-8b19-8a3f2937dabc-istio-envoy\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:04.662719 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:04.662701 4982 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd0cec81-8246-425e-8b19-8a3f2937dabc-credential-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:04.662836 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:04.662823 4982 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd0cec81-8246-425e-8b19-8a3f2937dabc-workload-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:04.662925 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:04.662915 4982 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd0cec81-8246-425e-8b19-8a3f2937dabc-istio-podinfo\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:04.662993 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:04.662837 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754" Oct 30 14:45:04.663848 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:04.663823 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd0cec81-8246-425e-8b19-8a3f2937dabc-istio-token" pod "bd0cec81-8246-425e-8b19-8a3f2937dabc" (UID: "bd0cec81-8246-425e-8b19-8a3f2937dabc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:45:04.666839 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:04.666817 4982 generic.go:356] "Generic (PLEG): container finished" podID="f95c53ac-ceef-47a4-abe7-aed7fba44238" containerID="6e02d67cd3c2aebc20ee841d4e7ca399272d86f5a56073da51cc679edd62bbd9" exitCode=60 Oct 30 14:45:04.667017 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:04.667000 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6e91df-29889525-cw745" event={"ID":"f95c53ac-ceef-47a4-abe7-aed7fba44238","Type":"ContainerDied","Data":"6e02d67cd3c2aebc20ee841d4e7ca399272d86f5a56073da51cc679edd62bbd9"} Oct 30 14:45:04.667488 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:04.667469 4982 scope.go:122] "RemoveContainer" containerID="6e02d67cd3c2aebc20ee841d4e7ca399272d86f5a56073da51cc679edd62bbd9" Oct 30 14:45:04.668062 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:45:04.668037 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-6e91df-29889525-cw745_odh-ai-gateway-infra(f95c53ac-ceef-47a4-abe7-aed7fba44238)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6e91df-29889525-cw745" podUID="f95c53ac-ceef-47a4-abe7-aed7fba44238" Oct 30 14:45:04.669107 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:04.669062 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-33c209-29889525-fj5nd" podStartSLOduration=4.669048009 podStartE2EDuration="4.669048009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:45:04.66420797 +0000 UTC m=+3374.408342012" watchObservedRunningTime="2026-10-30 14:45:04.669048009 +0000 UTC m=+3374.413182051" Oct 30 14:45:04.670581 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:04.670542 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3b95dd-29889525-mkkbq" event={"ID":"62a23269-9415-4a1f-b459-c6bb4a06cbae","Type":"ContainerStarted","Data":"ac14e6a45cf059eeee9968ac88e86f0eb6e0e8cf2c387a5ec7f48ff349eae62c"} Oct 30 14:45:04.676997 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:04.675096 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889525-bfv9w" event={"ID":"c6dd7aa1-d123-4c72-889d-8a6371b7b483","Type":"ContainerStarted","Data":"038fe0c5c954dc5dcfba0045d19584a2e5f69b6134f4762c4e2c869ee0d79d63"} Oct 30 14:45:04.699072 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:04.692606 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889525-hvqhb" event={"ID":"df8648aa-12d5-4d90-88ad-e7a1a8333142","Type":"ContainerStarted","Data":"3d79ca8a6c25e6831a8e4b69149069db43f80e630dd3420f467c784483fd2914"} Oct 30 14:45:04.715619 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:04.715123 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2927fd-29889525-srgtf" podStartSLOduration=4.71510702 podStartE2EDuration="4.71510702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:45:04.714429779 +0000 UTC m=+3374.458563842" watchObservedRunningTime="2026-10-30 14:45:04.71510702 +0000 UTC m=+3374.459241072" Oct 30 14:45:04.745939 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:04.745873 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889525-pjzwp" event={"ID":"381a4942-5434-499a-9d21-6e07dce9926b","Type":"ContainerStarted","Data":"282f3cba92a3573839f3da495c0ee08934a931ed2c88fb4972ebd094e232a904"} Oct 30 14:45:04.752427 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:04.752401 4982 scope.go:122] "RemoveContainer" containerID="242b6205c94f85c84c6820c6e46e7a4f482fbc439aed3e6d47f6a3873a0daf66" Oct 30 14:45:04.752797 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:45:04.752765 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"242b6205c94f85c84c6820c6e46e7a4f482fbc439aed3e6d47f6a3873a0daf66\": container with ID starting with 242b6205c94f85c84c6820c6e46e7a4f482fbc439aed3e6d47f6a3873a0daf66 not found: ID does not exist" containerID="242b6205c94f85c84c6820c6e46e7a4f482fbc439aed3e6d47f6a3873a0daf66" Oct 30 14:45:04.752914 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:04.752811 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"242b6205c94f85c84c6820c6e46e7a4f482fbc439aed3e6d47f6a3873a0daf66"} err="failed to get container status \"242b6205c94f85c84c6820c6e46e7a4f482fbc439aed3e6d47f6a3873a0daf66\": rpc error: code = NotFound desc = could not find container \"242b6205c94f85c84c6820c6e46e7a4f482fbc439aed3e6d47f6a3873a0daf66\": container with ID starting with 242b6205c94f85c84c6820c6e46e7a4f482fbc439aed3e6d47f6a3873a0daf66 not found: ID does not exist" Oct 30 14:45:04.752914 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:04.752834 4982 scope.go:122] "RemoveContainer" containerID="b5f4d12f40ec77e720e86075a5cf22ca768c510c7d347213abb0c9414942822b" Oct 30 14:45:04.764224 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:04.764201 4982 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd0cec81-8246-425e-8b19-8a3f2937dabc-istio-token\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:05.052932 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:05.052883 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889525-bfv9w" podStartSLOduration=5.052868006 podStartE2EDuration="5.052868006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:45:05.052108365 +0000 UTC m=+3374.796242408" watchObservedRunningTime="2026-10-30 14:45:05.052868006 +0000 UTC m=+3374.797002046" Oct 30 14:45:05.145553 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:05.145501 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3b95dd-29889525-mkkbq" podStartSLOduration=5.145468608 podStartE2EDuration="5.145468608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:45:05.100833638 +0000 UTC m=+3374.844967681" watchObservedRunningTime="2026-10-30 14:45:05.145468608 +0000 UTC m=+3374.889602649" Oct 30 14:45:05.250839 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:05.250798 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754"] Oct 30 14:45:05.256769 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:05.256742 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-acefbb-openshift-default-7c99445d75-cf754"] Oct 30 14:45:05.756700 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:05.756662 4982 generic.go:356] "Generic (PLEG): container finished" podID="1ca2abf9-60f0-489a-99fa-70a5f4d44def" containerID="6b33bfb4e0e2f2c2531897378fd2d0039317b537974038224dc04830428392d4" exitCode=60 Oct 30 14:45:05.757174 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:05.756740 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889525-pjgvt" event={"ID":"1ca2abf9-60f0-489a-99fa-70a5f4d44def","Type":"ContainerDied","Data":"6b33bfb4e0e2f2c2531897378fd2d0039317b537974038224dc04830428392d4"} Oct 30 14:45:05.757174 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:05.756828 4982 scope.go:122] "RemoveContainer" containerID="aff52f41431b8af0d73480d95eeef346e7d9f252550630cbb44b2d175b4507f2" Oct 30 14:45:05.757174 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:05.757045 4982 scope.go:122] "RemoveContainer" containerID="6b33bfb4e0e2f2c2531897378fd2d0039317b537974038224dc04830428392d4" Oct 30 14:45:05.757405 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:45:05.757386 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889525-pjgvt_odh-ai-gateway-infra(1ca2abf9-60f0-489a-99fa-70a5f4d44def)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889525-pjgvt" podUID="1ca2abf9-60f0-489a-99fa-70a5f4d44def" Oct 30 14:45:05.759604 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:05.759582 4982 generic.go:356] "Generic (PLEG): container finished" podID="1a89d4da-0eef-472e-bf20-613e750dd598" containerID="a69539550df24c2a57356c92ef0f8d52c29860416cd65c3ac0426bc91867d912" exitCode=60 Oct 30 14:45:05.759737 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:05.759683 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-1d6347-29889525-dqf8x" event={"ID":"1a89d4da-0eef-472e-bf20-613e750dd598","Type":"ContainerDied","Data":"a69539550df24c2a57356c92ef0f8d52c29860416cd65c3ac0426bc91867d912"} Oct 30 14:45:05.759949 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:05.759933 4982 scope.go:122] "RemoveContainer" containerID="a69539550df24c2a57356c92ef0f8d52c29860416cd65c3ac0426bc91867d912" Oct 30 14:45:05.760274 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:45:05.760252 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-1d6347-29889525-dqf8x_odh-ai-gateway-infra(1a89d4da-0eef-472e-bf20-613e750dd598)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-1d6347-29889525-dqf8x" podUID="1a89d4da-0eef-472e-bf20-613e750dd598" Oct 30 14:45:05.762986 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:05.762967 4982 scope.go:122] "RemoveContainer" containerID="6e02d67cd3c2aebc20ee841d4e7ca399272d86f5a56073da51cc679edd62bbd9" Oct 30 14:45:05.763308 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:45:05.763288 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-6e91df-29889525-cw745_odh-ai-gateway-infra(f95c53ac-ceef-47a4-abe7-aed7fba44238)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6e91df-29889525-cw745" podUID="f95c53ac-ceef-47a4-abe7-aed7fba44238" Oct 30 14:45:05.764761 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:05.764739 4982 generic.go:356] "Generic (PLEG): container finished" podID="62a23269-9415-4a1f-b459-c6bb4a06cbae" containerID="ac14e6a45cf059eeee9968ac88e86f0eb6e0e8cf2c387a5ec7f48ff349eae62c" exitCode=60 Oct 30 14:45:05.764859 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:05.764784 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3b95dd-29889525-mkkbq" event={"ID":"62a23269-9415-4a1f-b459-c6bb4a06cbae","Type":"ContainerDied","Data":"ac14e6a45cf059eeee9968ac88e86f0eb6e0e8cf2c387a5ec7f48ff349eae62c"} Oct 30 14:45:05.765071 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:05.765056 4982 scope.go:122] "RemoveContainer" containerID="ac14e6a45cf059eeee9968ac88e86f0eb6e0e8cf2c387a5ec7f48ff349eae62c" Oct 30 14:45:05.765396 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:45:05.765370 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-3b95dd-29889525-mkkbq_odh-ai-gateway-infra(62a23269-9415-4a1f-b459-c6bb4a06cbae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3b95dd-29889525-mkkbq" podUID="62a23269-9415-4a1f-b459-c6bb4a06cbae" Oct 30 14:45:05.767110 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:05.767090 4982 generic.go:356] "Generic (PLEG): container finished" podID="c6dd7aa1-d123-4c72-889d-8a6371b7b483" containerID="038fe0c5c954dc5dcfba0045d19584a2e5f69b6134f4762c4e2c869ee0d79d63" exitCode=60 Oct 30 14:45:05.767200 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:05.767147 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889525-bfv9w" event={"ID":"c6dd7aa1-d123-4c72-889d-8a6371b7b483","Type":"ContainerDied","Data":"038fe0c5c954dc5dcfba0045d19584a2e5f69b6134f4762c4e2c869ee0d79d63"} Oct 30 14:45:05.767437 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:05.767417 4982 scope.go:122] "RemoveContainer" containerID="038fe0c5c954dc5dcfba0045d19584a2e5f69b6134f4762c4e2c869ee0d79d63" Oct 30 14:45:05.767755 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:45:05.767727 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-00926c-29889525-bfv9w_odh-ai-gateway-infra(c6dd7aa1-d123-4c72-889d-8a6371b7b483)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889525-bfv9w" podUID="c6dd7aa1-d123-4c72-889d-8a6371b7b483" Oct 30 14:45:05.769167 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:05.769148 4982 generic.go:356] "Generic (PLEG): container finished" podID="df8648aa-12d5-4d90-88ad-e7a1a8333142" containerID="3d79ca8a6c25e6831a8e4b69149069db43f80e630dd3420f467c784483fd2914" exitCode=60 Oct 30 14:45:05.769259 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:05.769235 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889525-hvqhb" event={"ID":"df8648aa-12d5-4d90-88ad-e7a1a8333142","Type":"ContainerDied","Data":"3d79ca8a6c25e6831a8e4b69149069db43f80e630dd3420f467c784483fd2914"} Oct 30 14:45:05.769451 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:05.769429 4982 scope.go:122] "RemoveContainer" containerID="3d79ca8a6c25e6831a8e4b69149069db43f80e630dd3420f467c784483fd2914" Oct 30 14:45:05.769768 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:45:05.769730 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-ca229c-29889525-hvqhb_odh-ai-gateway-infra(df8648aa-12d5-4d90-88ad-e7a1a8333142)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889525-hvqhb" podUID="df8648aa-12d5-4d90-88ad-e7a1a8333142" Oct 30 14:45:05.771864 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:05.771839 4982 generic.go:356] "Generic (PLEG): container finished" podID="381a4942-5434-499a-9d21-6e07dce9926b" containerID="282f3cba92a3573839f3da495c0ee08934a931ed2c88fb4972ebd094e232a904" exitCode=60 Oct 30 14:45:05.772235 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:05.772217 4982 scope.go:122] "RemoveContainer" containerID="282f3cba92a3573839f3da495c0ee08934a931ed2c88fb4972ebd094e232a904" Oct 30 14:45:05.773682 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:45:05.772487 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-46b13b-29889525-pjzwp_odh-ai-gateway-infra(381a4942-5434-499a-9d21-6e07dce9926b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889525-pjzwp" podUID="381a4942-5434-499a-9d21-6e07dce9926b" Oct 30 14:45:05.773682 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:05.772523 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889525-pjzwp" event={"ID":"381a4942-5434-499a-9d21-6e07dce9926b","Type":"ContainerDied","Data":"282f3cba92a3573839f3da495c0ee08934a931ed2c88fb4972ebd094e232a904"} Oct 30 14:45:05.773682 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:05.772588 4982 scope.go:122] "RemoveContainer" containerID="d86bd9dfb0de65a75313a9ac5a87787ae1e26349bf56bd5fde90a4b0278575a4" Oct 30 14:45:05.776129 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:05.776096 4982 generic.go:356] "Generic (PLEG): container finished" podID="d524b057-6219-43cf-a11b-5048f36a8dfd" containerID="52e69aae9b6a227d176f2c3c37177c43b1a3130ff1aaf4e937e9d1f82e0279d5" exitCode=60 Oct 30 14:45:05.776217 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:05.776145 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-33c209-29889525-fj5nd" event={"ID":"d524b057-6219-43cf-a11b-5048f36a8dfd","Type":"ContainerDied","Data":"52e69aae9b6a227d176f2c3c37177c43b1a3130ff1aaf4e937e9d1f82e0279d5"} Oct 30 14:45:05.776507 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:05.776486 4982 scope.go:122] "RemoveContainer" containerID="52e69aae9b6a227d176f2c3c37177c43b1a3130ff1aaf4e937e9d1f82e0279d5" Oct 30 14:45:05.777110 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:45:05.776782 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-33c209-29889525-fj5nd_odh-ai-gateway-infra(d524b057-6219-43cf-a11b-5048f36a8dfd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-33c209-29889525-fj5nd" podUID="d524b057-6219-43cf-a11b-5048f36a8dfd" Oct 30 14:45:05.778700 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:05.778532 4982 generic.go:356] "Generic (PLEG): container finished" podID="4e292b6c-6d65-41cd-a634-0a33f9cd86a4" containerID="5d4691d5b0d1f5d91dbb63d34bf1b195caccbd92a44c96df408728c7b19625aa" exitCode=60 Oct 30 14:45:05.778700 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:05.778557 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889525-hz26s" event={"ID":"4e292b6c-6d65-41cd-a634-0a33f9cd86a4","Type":"ContainerDied","Data":"5d4691d5b0d1f5d91dbb63d34bf1b195caccbd92a44c96df408728c7b19625aa"} Oct 30 14:45:05.778838 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:05.778828 4982 scope.go:122] "RemoveContainer" containerID="5d4691d5b0d1f5d91dbb63d34bf1b195caccbd92a44c96df408728c7b19625aa" Oct 30 14:45:05.779045 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:45:05.779027 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889525-hz26s_odh-ai-gateway-infra(4e292b6c-6d65-41cd-a634-0a33f9cd86a4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889525-hz26s" podUID="4e292b6c-6d65-41cd-a634-0a33f9cd86a4" Oct 30 14:45:05.791664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:05.787355 4982 generic.go:356] "Generic (PLEG): container finished" podID="620719c8-c4d6-4f56-a843-24137f57fec0" containerID="89a2c29a4dd95a6ba99405575ce22f9478035787f1e7817a9fa33d55450b2595" exitCode=60 Oct 30 14:45:05.791664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:05.787460 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2927fd-29889525-srgtf" event={"ID":"620719c8-c4d6-4f56-a843-24137f57fec0","Type":"ContainerDied","Data":"89a2c29a4dd95a6ba99405575ce22f9478035787f1e7817a9fa33d55450b2595"} Oct 30 14:45:05.791664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:05.787816 4982 scope.go:122] "RemoveContainer" containerID="89a2c29a4dd95a6ba99405575ce22f9478035787f1e7817a9fa33d55450b2595" Oct 30 14:45:05.791664 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:45:05.788067 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-2927fd-29889525-srgtf_odh-ai-gateway-infra(620719c8-c4d6-4f56-a843-24137f57fec0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2927fd-29889525-srgtf" podUID="620719c8-c4d6-4f56-a843-24137f57fec0" Oct 30 14:45:05.814659 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:05.814620 4982 scope.go:122] "RemoveContainer" containerID="760afe08b8c68ff0d1a54122dd4c686a60b180c105e513d03009c5598683bc82" Oct 30 14:45:05.828581 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:05.828557 4982 scope.go:122] "RemoveContainer" containerID="b7510d38f44fe8aa8302b4cbb3ebc6110e445efaee93a01bf991e5ca4d4eb2b7" Oct 30 14:45:05.843854 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:05.843831 4982 scope.go:122] "RemoveContainer" containerID="19edc87057e6a3459ba5f36ac05dd50df051d488b1a558246bc4c98e45d4e731" Oct 30 14:45:05.857055 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:05.857035 4982 scope.go:122] "RemoveContainer" containerID="af7855bd58c576f1cd14c604506706a26209517bc16e028cb8b7b9ce50d75b3b" Oct 30 14:45:05.869880 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:05.869863 4982 scope.go:122] "RemoveContainer" containerID="f5ebd6e0b20697ef9e6bf8171899c3a5d149034a8d8817c049a88159d6962b79" Oct 30 14:45:05.911300 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:05.911269 4982 scope.go:122] "RemoveContainer" containerID="48657d825eb647545d8860dbaedf21ba62d6c50009b1ebc011d53e6099530aa5" Oct 30 14:45:05.926315 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:05.926290 4982 scope.go:122] "RemoveContainer" containerID="16e60e85499ccb1a92746ef6604350e9072cc31d246817af4e436b604bd9a334" Oct 30 14:45:06.028411 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:06.028309 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:45:06.229674 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:06.227110 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:45:06.377099 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:06.377063 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:45:06.424013 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:06.423982 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd0cec81-8246-425e-8b19-8a3f2937dabc" path="/var/lib/kubelet/pods/bd0cec81-8246-425e-8b19-8a3f2937dabc/volumes" Oct 30 14:45:06.823892 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:06.823860 4982 scope.go:122] "RemoveContainer" containerID="6b33bfb4e0e2f2c2531897378fd2d0039317b537974038224dc04830428392d4" Oct 30 14:45:06.824973 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:45:06.824942 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889525-pjgvt_odh-ai-gateway-infra(1ca2abf9-60f0-489a-99fa-70a5f4d44def)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889525-pjgvt" podUID="1ca2abf9-60f0-489a-99fa-70a5f4d44def" Oct 30 14:45:06.828441 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:06.828381 4982 scope.go:122] "RemoveContainer" containerID="a69539550df24c2a57356c92ef0f8d52c29860416cd65c3ac0426bc91867d912" Oct 30 14:45:06.859658 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:45:06.847290 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-1d6347-29889525-dqf8x_odh-ai-gateway-infra(1a89d4da-0eef-472e-bf20-613e750dd598)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-1d6347-29889525-dqf8x" podUID="1a89d4da-0eef-472e-bf20-613e750dd598" Oct 30 14:45:06.875263 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:06.858355 4982 scope.go:122] "RemoveContainer" containerID="ac14e6a45cf059eeee9968ac88e86f0eb6e0e8cf2c387a5ec7f48ff349eae62c" Oct 30 14:45:06.875584 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:06.875563 4982 scope.go:122] "RemoveContainer" containerID="038fe0c5c954dc5dcfba0045d19584a2e5f69b6134f4762c4e2c869ee0d79d63" Oct 30 14:45:06.876140 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:45:06.876111 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-00926c-29889525-bfv9w_odh-ai-gateway-infra(c6dd7aa1-d123-4c72-889d-8a6371b7b483)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889525-bfv9w" podUID="c6dd7aa1-d123-4c72-889d-8a6371b7b483" Oct 30 14:45:06.876614 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:45:06.876588 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-3b95dd-29889525-mkkbq_odh-ai-gateway-infra(62a23269-9415-4a1f-b459-c6bb4a06cbae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3b95dd-29889525-mkkbq" podUID="62a23269-9415-4a1f-b459-c6bb4a06cbae" Oct 30 14:45:06.893971 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:06.893946 4982 scope.go:122] "RemoveContainer" containerID="3d79ca8a6c25e6831a8e4b69149069db43f80e630dd3420f467c784483fd2914" Oct 30 14:45:06.931382 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:45:06.931345 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-ca229c-29889525-hvqhb_odh-ai-gateway-infra(df8648aa-12d5-4d90-88ad-e7a1a8333142)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889525-hvqhb" podUID="df8648aa-12d5-4d90-88ad-e7a1a8333142" Oct 30 14:45:06.931657 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:06.931576 4982 scope.go:122] "RemoveContainer" containerID="282f3cba92a3573839f3da495c0ee08934a931ed2c88fb4972ebd094e232a904" Oct 30 14:45:06.932062 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:45:06.932036 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-46b13b-29889525-pjzwp_odh-ai-gateway-infra(381a4942-5434-499a-9d21-6e07dce9926b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889525-pjzwp" podUID="381a4942-5434-499a-9d21-6e07dce9926b" Oct 30 14:45:06.932213 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:06.932171 4982 scope.go:122] "RemoveContainer" containerID="52e69aae9b6a227d176f2c3c37177c43b1a3130ff1aaf4e937e9d1f82e0279d5" Oct 30 14:45:06.934937 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:45:06.932563 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-33c209-29889525-fj5nd_odh-ai-gateway-infra(d524b057-6219-43cf-a11b-5048f36a8dfd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-33c209-29889525-fj5nd" podUID="d524b057-6219-43cf-a11b-5048f36a8dfd" Oct 30 14:45:06.947346 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:06.947325 4982 scope.go:122] "RemoveContainer" containerID="5d4691d5b0d1f5d91dbb63d34bf1b195caccbd92a44c96df408728c7b19625aa" Oct 30 14:45:06.947873 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:45:06.947850 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889525-hz26s_odh-ai-gateway-infra(4e292b6c-6d65-41cd-a634-0a33f9cd86a4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889525-hz26s" podUID="4e292b6c-6d65-41cd-a634-0a33f9cd86a4" Oct 30 14:45:06.950771 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:06.949501 4982 scope.go:122] "RemoveContainer" containerID="89a2c29a4dd95a6ba99405575ce22f9478035787f1e7817a9fa33d55450b2595" Oct 30 14:45:06.964737 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:45:06.964389 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-2927fd-29889525-srgtf_odh-ai-gateway-infra(620719c8-c4d6-4f56-a843-24137f57fec0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2927fd-29889525-srgtf" podUID="620719c8-c4d6-4f56-a843-24137f57fec0" Oct 30 14:45:08.560182 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:08.560146 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:45:10.256795 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:10.256755 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f219c0fa-c9x9b"] Oct 30 14:45:10.257697 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:10.257626 4982 cpu_manager.go:414] "containerMap: removing container" podUID="bd0cec81-8246-425e-8b19-8a3f2937dabc" containerName="istio-proxy" Oct 30 14:45:10.257697 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:10.257675 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd0cec81-8246-425e-8b19-8a3f2937dabc" containerName="istio-proxy" Oct 30 14:45:10.257842 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:10.257827 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd0cec81-8246-425e-8b19-8a3f2937dabc" containerName="istio-proxy" Oct 30 14:45:10.261493 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:10.261476 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f219c0fa-c9x9b" Oct 30 14:45:10.284339 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:10.284311 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f219c0fa-c9x9b"] Oct 30 14:45:10.294608 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:10.294581 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:45:10.375280 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:10.375239 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9wbc\" (UniqueName: \"kubernetes.io/projected/8ef323f9-5aa4-48c4-826d-4d1b9ab83126-kube-api-access-c9wbc\") pod \"maas-api-revoke-sub-f219c0fa-c9x9b\" (UID: \"8ef323f9-5aa4-48c4-826d-4d1b9ab83126\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f219c0fa-c9x9b" Oct 30 14:45:10.375478 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:10.375301 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ef323f9-5aa4-48c4-826d-4d1b9ab83126-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f219c0fa-c9x9b\" (UID: \"8ef323f9-5aa4-48c4-826d-4d1b9ab83126\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f219c0fa-c9x9b" Oct 30 14:45:10.375478 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:10.375445 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ef323f9-5aa4-48c4-826d-4d1b9ab83126-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f219c0fa-c9x9b\" (UID: \"8ef323f9-5aa4-48c4-826d-4d1b9ab83126\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f219c0fa-c9x9b" Oct 30 14:45:10.385814 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:10.385789 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:45:10.476813 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:10.476782 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9wbc\" (UniqueName: \"kubernetes.io/projected/8ef323f9-5aa4-48c4-826d-4d1b9ab83126-kube-api-access-c9wbc\") pod \"maas-api-revoke-sub-f219c0fa-c9x9b\" (UID: \"8ef323f9-5aa4-48c4-826d-4d1b9ab83126\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f219c0fa-c9x9b" Oct 30 14:45:10.476998 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:10.476856 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ef323f9-5aa4-48c4-826d-4d1b9ab83126-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f219c0fa-c9x9b\" (UID: \"8ef323f9-5aa4-48c4-826d-4d1b9ab83126\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f219c0fa-c9x9b" Oct 30 14:45:10.476998 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:10.476922 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ef323f9-5aa4-48c4-826d-4d1b9ab83126-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f219c0fa-c9x9b\" (UID: \"8ef323f9-5aa4-48c4-826d-4d1b9ab83126\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f219c0fa-c9x9b" Oct 30 14:45:10.477871 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:10.477836 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ef323f9-5aa4-48c4-826d-4d1b9ab83126-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f219c0fa-c9x9b\" (UID: \"8ef323f9-5aa4-48c4-826d-4d1b9ab83126\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f219c0fa-c9x9b" Oct 30 14:45:10.479173 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:10.479148 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ef323f9-5aa4-48c4-826d-4d1b9ab83126-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f219c0fa-c9x9b\" (UID: \"8ef323f9-5aa4-48c4-826d-4d1b9ab83126\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f219c0fa-c9x9b" Oct 30 14:45:10.503007 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:10.502978 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9wbc\" (UniqueName: \"kubernetes.io/projected/8ef323f9-5aa4-48c4-826d-4d1b9ab83126-kube-api-access-c9wbc\") pod \"maas-api-revoke-sub-f219c0fa-c9x9b\" (UID: \"8ef323f9-5aa4-48c4-826d-4d1b9ab83126\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f219c0fa-c9x9b" Oct 30 14:45:10.542401 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:10.542328 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:45:10.580478 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:10.580450 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f219c0fa-c9x9b" Oct 30 14:45:10.773004 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:10.772968 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:45:10.900716 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:10.900683 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f219c0fa-c9x9b"] Oct 30 14:45:10.904907 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:45:10.904878 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ef323f9_5aa4_48c4_826d_4d1b9ab83126.slice/crio-d7596eef22e54ff4005cc5ee373954c772d033248eb765cbd76c5e5f8ac14580 WatchSource:0}: Error finding container d7596eef22e54ff4005cc5ee373954c772d033248eb765cbd76c5e5f8ac14580: Status 404 returned error can't find the container with id d7596eef22e54ff4005cc5ee373954c772d033248eb765cbd76c5e5f8ac14580 Oct 30 14:45:10.990407 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:10.990373 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f219c0fa-c9x9b" event={"ID":"8ef323f9-5aa4-48c4-826d-4d1b9ab83126","Type":"ContainerStarted","Data":"d7596eef22e54ff4005cc5ee373954c772d033248eb765cbd76c5e5f8ac14580"} Oct 30 14:45:10.992092 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:10.992067 4982 generic.go:356] "Generic (PLEG): container finished" podID="9b853a87-0adf-46f0-b45d-c17b69a68495" containerID="f40b22d496fc456deef3f35cbfb2bd4fce960f0bd73d35e1396e9f5ea8ab6e7a" exitCode=0 Oct 30 14:45:10.992200 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:10.992135 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27" event={"ID":"9b853a87-0adf-46f0-b45d-c17b69a68495","Type":"ContainerDied","Data":"f40b22d496fc456deef3f35cbfb2bd4fce960f0bd73d35e1396e9f5ea8ab6e7a"} Oct 30 14:45:10.992558 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:10.992539 4982 scope.go:122] "RemoveContainer" containerID="f40b22d496fc456deef3f35cbfb2bd4fce960f0bd73d35e1396e9f5ea8ab6e7a" Oct 30 14:45:10.994231 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:10.994202 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:45:10.994292 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:10.994272 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:45:10.994329 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:10.994305 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:45:12.018087 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:12.018051 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f219c0fa-c9x9b" event={"ID":"8ef323f9-5aa4-48c4-826d-4d1b9ab83126","Type":"ContainerStarted","Data":"d91d82838a966ad0d90ec6d3d95c548f7141768b29a80c2708d179ae60d5eb41"} Oct 30 14:45:12.020399 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:12.020374 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27" event={"ID":"9b853a87-0adf-46f0-b45d-c17b69a68495","Type":"ContainerStarted","Data":"07975312845b9ab2a9b567ef181257a9f0085811e94eea60ceda6f26185542dc"} Oct 30 14:45:12.062011 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:12.061961 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f219c0fa-c9x9b" podStartSLOduration=2.061946774 podStartE2EDuration="2.061946774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:45:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:45:12.061474178 +0000 UTC m=+3381.805608219" watchObservedRunningTime="2026-10-30 14:45:12.061946774 +0000 UTC m=+3381.806080816" Oct 30 14:45:12.928941 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:12.928904 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27" Oct 30 14:45:12.929879 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:12.929850 4982 patch_prober.go:29] interesting pod/e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Oct 30 14:45:12.930019 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:12.929904 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27" podUID="9b853a87-0adf-46f0-b45d-c17b69a68495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Oct 30 14:45:13.033660 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:13.031943 4982 generic.go:356] "Generic (PLEG): container finished" podID="8ef323f9-5aa4-48c4-826d-4d1b9ab83126" containerID="d91d82838a966ad0d90ec6d3d95c548f7141768b29a80c2708d179ae60d5eb41" exitCode=0 Oct 30 14:45:13.033660 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:13.032023 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f219c0fa-c9x9b" event={"ID":"8ef323f9-5aa4-48c4-826d-4d1b9ab83126","Type":"ContainerDied","Data":"d91d82838a966ad0d90ec6d3d95c548f7141768b29a80c2708d179ae60d5eb41"} Oct 30 14:45:13.517272 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:13.515182 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:45:13.929168 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:13.929123 4982 patch_prober.go:29] interesting pod/e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Oct 30 14:45:13.929380 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:13.929173 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27" podUID="9b853a87-0adf-46f0-b45d-c17b69a68495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Oct 30 14:45:14.277668 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:14.277626 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f219c0fa-c9x9b" Oct 30 14:45:14.456490 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:14.456459 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ef323f9-5aa4-48c4-826d-4d1b9ab83126-kube-api-access-c9wbc\" (UniqueName: \"kubernetes.io/projected/8ef323f9-5aa4-48c4-826d-4d1b9ab83126-kube-api-access-c9wbc\") pod \"8ef323f9-5aa4-48c4-826d-4d1b9ab83126\" (UID: \"8ef323f9-5aa4-48c4-826d-4d1b9ab83126\") " Oct 30 14:45:14.456702 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:14.456681 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ef323f9-5aa4-48c4-826d-4d1b9ab83126-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ef323f9-5aa4-48c4-826d-4d1b9ab83126-maas-api-xks-ca\") pod \"8ef323f9-5aa4-48c4-826d-4d1b9ab83126\" (UID: \"8ef323f9-5aa4-48c4-826d-4d1b9ab83126\") " Oct 30 14:45:14.456838 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:14.456809 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ef323f9-5aa4-48c4-826d-4d1b9ab83126-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ef323f9-5aa4-48c4-826d-4d1b9ab83126-maas-api-openshift-ca\") pod \"8ef323f9-5aa4-48c4-826d-4d1b9ab83126\" (UID: \"8ef323f9-5aa4-48c4-826d-4d1b9ab83126\") " Oct 30 14:45:14.457102 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:14.457076 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ef323f9-5aa4-48c4-826d-4d1b9ab83126-maas-api-openshift-ca" pod "8ef323f9-5aa4-48c4-826d-4d1b9ab83126" (UID: "8ef323f9-5aa4-48c4-826d-4d1b9ab83126"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:45:14.457276 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:14.457255 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ef323f9-5aa4-48c4-826d-4d1b9ab83126-maas-api-openshift-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:14.458862 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:14.458837 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ef323f9-5aa4-48c4-826d-4d1b9ab83126-maas-api-xks-ca" pod "8ef323f9-5aa4-48c4-826d-4d1b9ab83126" (UID: "8ef323f9-5aa4-48c4-826d-4d1b9ab83126"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:45:14.458938 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:14.458879 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ef323f9-5aa4-48c4-826d-4d1b9ab83126-kube-api-access-c9wbc" pod "8ef323f9-5aa4-48c4-826d-4d1b9ab83126" (UID: "8ef323f9-5aa4-48c4-826d-4d1b9ab83126"). InnerVolumeSpecName "kube-api-access-c9wbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:45:14.558677 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:14.558636 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ef323f9-5aa4-48c4-826d-4d1b9ab83126-maas-api-xks-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:14.558677 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:14.558676 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9wbc\" (UniqueName: \"kubernetes.io/projected/8ef323f9-5aa4-48c4-826d-4d1b9ab83126-kube-api-access-c9wbc\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:14.929040 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:14.928952 4982 patch_prober.go:29] interesting pod/e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Oct 30 14:45:14.929040 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:14.929016 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27" podUID="9b853a87-0adf-46f0-b45d-c17b69a68495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Oct 30 14:45:15.050750 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:15.050717 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f219c0fa-c9x9b" Oct 30 14:45:15.050894 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:15.050757 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f219c0fa-c9x9b" event={"ID":"8ef323f9-5aa4-48c4-826d-4d1b9ab83126","Type":"ContainerDied","Data":"d7596eef22e54ff4005cc5ee373954c772d033248eb765cbd76c5e5f8ac14580"} Oct 30 14:45:15.050894 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:15.050810 4982 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7596eef22e54ff4005cc5ee373954c772d033248eb765cbd76c5e5f8ac14580" Oct 30 14:45:15.302216 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:15.302121 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:45:15.404663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:15.403353 4982 scope.go:122] "RemoveContainer" containerID="282f3cba92a3573839f3da495c0ee08934a931ed2c88fb4972ebd094e232a904" Oct 30 14:45:15.404663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:15.404473 4982 scope.go:122] "RemoveContainer" containerID="6e02d67cd3c2aebc20ee841d4e7ca399272d86f5a56073da51cc679edd62bbd9" Oct 30 14:45:15.435878 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:15.435850 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" podUID="cbd754bd-771d-4007-8ac0-7e413dc24915" containerName="main" containerID="cri-o://b08205185b49f8f91e2fac0f1ee1b8adb70afe7d6201a1a78cf3b6718ac81bb6" gracePeriod=45 Oct 30 14:45:15.459192 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:15.459035 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8" podUID="f3093abe-9b1e-492a-8837-4bd42ea39d2d" containerName="main" containerID="cri-o://de198dcfa2afe64c56d7e44af73543b23100debca344cf2a95b7aecc602f296e" gracePeriod=45 Oct 30 14:45:15.873792 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:15.873766 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" Oct 30 14:45:15.876916 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:15.876882 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbd754bd-771d-4007-8ac0-7e413dc24915-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbd754bd-771d-4007-8ac0-7e413dc24915-tmp-dir\") pod \"cbd754bd-771d-4007-8ac0-7e413dc24915\" (UID: \"cbd754bd-771d-4007-8ac0-7e413dc24915\") " Oct 30 14:45:15.877146 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:15.877123 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbd754bd-771d-4007-8ac0-7e413dc24915-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbd754bd-771d-4007-8ac0-7e413dc24915-dshm\") pod \"cbd754bd-771d-4007-8ac0-7e413dc24915\" (UID: \"cbd754bd-771d-4007-8ac0-7e413dc24915\") " Oct 30 14:45:15.877234 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:15.877218 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbd754bd-771d-4007-8ac0-7e413dc24915-home\" (UniqueName: \"kubernetes.io/empty-dir/cbd754bd-771d-4007-8ac0-7e413dc24915-home\") pod \"cbd754bd-771d-4007-8ac0-7e413dc24915\" (UID: \"cbd754bd-771d-4007-8ac0-7e413dc24915\") " Oct 30 14:45:15.877280 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:15.877263 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbd754bd-771d-4007-8ac0-7e413dc24915-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbd754bd-771d-4007-8ac0-7e413dc24915-model-cache\") pod \"cbd754bd-771d-4007-8ac0-7e413dc24915\" (UID: \"cbd754bd-771d-4007-8ac0-7e413dc24915\") " Oct 30 14:45:15.877324 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:15.877300 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbd754bd-771d-4007-8ac0-7e413dc24915-kube-api-access-qdbl4\" (UniqueName: \"kubernetes.io/projected/cbd754bd-771d-4007-8ac0-7e413dc24915-kube-api-access-qdbl4\") pod \"cbd754bd-771d-4007-8ac0-7e413dc24915\" (UID: \"cbd754bd-771d-4007-8ac0-7e413dc24915\") " Oct 30 14:45:15.877324 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:15.877304 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbd754bd-771d-4007-8ac0-7e413dc24915-tmp-dir" pod "cbd754bd-771d-4007-8ac0-7e413dc24915" (UID: "cbd754bd-771d-4007-8ac0-7e413dc24915"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:45:15.877407 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:15.877330 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbd754bd-771d-4007-8ac0-7e413dc24915-tls-certs\" (UniqueName: \"kubernetes.io/secret/cbd754bd-771d-4007-8ac0-7e413dc24915-tls-certs\") pod \"cbd754bd-771d-4007-8ac0-7e413dc24915\" (UID: \"cbd754bd-771d-4007-8ac0-7e413dc24915\") " Oct 30 14:45:15.877447 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:15.877435 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbd754bd-771d-4007-8ac0-7e413dc24915-home" pod "cbd754bd-771d-4007-8ac0-7e413dc24915" (UID: "cbd754bd-771d-4007-8ac0-7e413dc24915"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:45:15.878014 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:15.877993 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbd754bd-771d-4007-8ac0-7e413dc24915-model-cache" pod "cbd754bd-771d-4007-8ac0-7e413dc24915" (UID: "cbd754bd-771d-4007-8ac0-7e413dc24915"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:45:15.878114 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:15.878054 4982 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbd754bd-771d-4007-8ac0-7e413dc24915-tmp-dir\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:15.878114 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:15.878077 4982 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbd754bd-771d-4007-8ac0-7e413dc24915-home\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:15.890202 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:15.884937 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8" Oct 30 14:45:15.908751 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:15.893162 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbd754bd-771d-4007-8ac0-7e413dc24915-tls-certs" pod "cbd754bd-771d-4007-8ac0-7e413dc24915" (UID: "cbd754bd-771d-4007-8ac0-7e413dc24915"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:45:15.939104 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:15.939049 4982 patch_prober.go:29] interesting pod/e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Oct 30 14:45:15.939363 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:15.939331 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27" podUID="9b853a87-0adf-46f0-b45d-c17b69a68495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Oct 30 14:45:15.999704 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:15.982593 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3093abe-9b1e-492a-8837-4bd42ea39d2d-tls-certs\" (UniqueName: \"kubernetes.io/secret/f3093abe-9b1e-492a-8837-4bd42ea39d2d-tls-certs\") pod \"f3093abe-9b1e-492a-8837-4bd42ea39d2d\" (UID: \"f3093abe-9b1e-492a-8837-4bd42ea39d2d\") " Oct 30 14:45:15.999704 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:15.982710 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3093abe-9b1e-492a-8837-4bd42ea39d2d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3093abe-9b1e-492a-8837-4bd42ea39d2d-model-cache\") pod \"f3093abe-9b1e-492a-8837-4bd42ea39d2d\" (UID: \"f3093abe-9b1e-492a-8837-4bd42ea39d2d\") " Oct 30 14:45:15.999704 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:15.982782 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3093abe-9b1e-492a-8837-4bd42ea39d2d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3093abe-9b1e-492a-8837-4bd42ea39d2d-dshm\") pod \"f3093abe-9b1e-492a-8837-4bd42ea39d2d\" (UID: \"f3093abe-9b1e-492a-8837-4bd42ea39d2d\") " Oct 30 14:45:15.999704 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:15.982837 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3093abe-9b1e-492a-8837-4bd42ea39d2d-kube-api-access-ncp2n\" (UniqueName: \"kubernetes.io/projected/f3093abe-9b1e-492a-8837-4bd42ea39d2d-kube-api-access-ncp2n\") pod \"f3093abe-9b1e-492a-8837-4bd42ea39d2d\" (UID: \"f3093abe-9b1e-492a-8837-4bd42ea39d2d\") " Oct 30 14:45:15.999704 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:15.982944 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3093abe-9b1e-492a-8837-4bd42ea39d2d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3093abe-9b1e-492a-8837-4bd42ea39d2d-tmp-dir\") pod \"f3093abe-9b1e-492a-8837-4bd42ea39d2d\" (UID: \"f3093abe-9b1e-492a-8837-4bd42ea39d2d\") " Oct 30 14:45:15.999704 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:15.983116 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3093abe-9b1e-492a-8837-4bd42ea39d2d-home\" (UniqueName: \"kubernetes.io/empty-dir/f3093abe-9b1e-492a-8837-4bd42ea39d2d-home\") pod \"f3093abe-9b1e-492a-8837-4bd42ea39d2d\" (UID: \"f3093abe-9b1e-492a-8837-4bd42ea39d2d\") " Oct 30 14:45:15.999704 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:15.985872 4982 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbd754bd-771d-4007-8ac0-7e413dc24915-model-cache\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:15.999704 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:15.986458 4982 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbd754bd-771d-4007-8ac0-7e413dc24915-tls-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:15.999704 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:15.986818 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3093abe-9b1e-492a-8837-4bd42ea39d2d-home" pod "f3093abe-9b1e-492a-8837-4bd42ea39d2d" (UID: "f3093abe-9b1e-492a-8837-4bd42ea39d2d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:45:15.999704 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:15.988244 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3093abe-9b1e-492a-8837-4bd42ea39d2d-tmp-dir" pod "f3093abe-9b1e-492a-8837-4bd42ea39d2d" (UID: "f3093abe-9b1e-492a-8837-4bd42ea39d2d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:45:15.999704 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:15.988312 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3093abe-9b1e-492a-8837-4bd42ea39d2d-model-cache" pod "f3093abe-9b1e-492a-8837-4bd42ea39d2d" (UID: "f3093abe-9b1e-492a-8837-4bd42ea39d2d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:45:15.999704 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:15.998325 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbd754bd-771d-4007-8ac0-7e413dc24915-dshm" pod "cbd754bd-771d-4007-8ac0-7e413dc24915" (UID: "cbd754bd-771d-4007-8ac0-7e413dc24915"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:45:16.030805 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:16.026173 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3093abe-9b1e-492a-8837-4bd42ea39d2d-dshm" pod "f3093abe-9b1e-492a-8837-4bd42ea39d2d" (UID: "f3093abe-9b1e-492a-8837-4bd42ea39d2d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:45:16.030805 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:16.026311 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3093abe-9b1e-492a-8837-4bd42ea39d2d-kube-api-access-ncp2n" pod "f3093abe-9b1e-492a-8837-4bd42ea39d2d" (UID: "f3093abe-9b1e-492a-8837-4bd42ea39d2d"). InnerVolumeSpecName "kube-api-access-ncp2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:45:16.030805 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:16.026368 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3093abe-9b1e-492a-8837-4bd42ea39d2d-tls-certs" pod "f3093abe-9b1e-492a-8837-4bd42ea39d2d" (UID: "f3093abe-9b1e-492a-8837-4bd42ea39d2d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:45:16.030805 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:16.026898 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbd754bd-771d-4007-8ac0-7e413dc24915-kube-api-access-qdbl4" pod "cbd754bd-771d-4007-8ac0-7e413dc24915" (UID: "cbd754bd-771d-4007-8ac0-7e413dc24915"). InnerVolumeSpecName "kube-api-access-qdbl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:45:16.067684 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:16.064873 4982 generic.go:356] "Generic (PLEG): container finished" podID="cbd754bd-771d-4007-8ac0-7e413dc24915" containerID="b08205185b49f8f91e2fac0f1ee1b8adb70afe7d6201a1a78cf3b6718ac81bb6" exitCode=0 Oct 30 14:45:16.067684 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:16.064938 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" Oct 30 14:45:16.067684 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:16.064953 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" event={"ID":"cbd754bd-771d-4007-8ac0-7e413dc24915","Type":"ContainerDied","Data":"b08205185b49f8f91e2fac0f1ee1b8adb70afe7d6201a1a78cf3b6718ac81bb6"} Oct 30 14:45:16.067684 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:16.064996 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn" event={"ID":"cbd754bd-771d-4007-8ac0-7e413dc24915","Type":"ContainerDied","Data":"b90ef2db5eefd1a9a37d1eee9825c3ca6559195829ad5a703bb1176e08f76ab2"} Oct 30 14:45:16.067684 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:16.065012 4982 scope.go:122] "RemoveContainer" containerID="b08205185b49f8f91e2fac0f1ee1b8adb70afe7d6201a1a78cf3b6718ac81bb6" Oct 30 14:45:16.067684 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:16.066744 4982 generic.go:356] "Generic (PLEG): container finished" podID="f3093abe-9b1e-492a-8837-4bd42ea39d2d" containerID="de198dcfa2afe64c56d7e44af73543b23100debca344cf2a95b7aecc602f296e" exitCode=0 Oct 30 14:45:16.067684 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:16.066864 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8" event={"ID":"f3093abe-9b1e-492a-8837-4bd42ea39d2d","Type":"ContainerDied","Data":"de198dcfa2afe64c56d7e44af73543b23100debca344cf2a95b7aecc602f296e"} Oct 30 14:45:16.067684 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:16.066891 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8" event={"ID":"f3093abe-9b1e-492a-8837-4bd42ea39d2d","Type":"ContainerDied","Data":"2e14dda523d100aa807d63035114e4a81c12777d9ada6aa87fa74d46c32144ef"} Oct 30 14:45:16.067684 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:16.066942 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8" Oct 30 14:45:16.078315 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:16.078298 4982 scope.go:122] "RemoveContainer" containerID="b08205185b49f8f91e2fac0f1ee1b8adb70afe7d6201a1a78cf3b6718ac81bb6" Oct 30 14:45:16.078649 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:45:16.078598 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b08205185b49f8f91e2fac0f1ee1b8adb70afe7d6201a1a78cf3b6718ac81bb6\": container with ID starting with b08205185b49f8f91e2fac0f1ee1b8adb70afe7d6201a1a78cf3b6718ac81bb6 not found: ID does not exist" containerID="b08205185b49f8f91e2fac0f1ee1b8adb70afe7d6201a1a78cf3b6718ac81bb6" Oct 30 14:45:16.078739 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:16.078631 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b08205185b49f8f91e2fac0f1ee1b8adb70afe7d6201a1a78cf3b6718ac81bb6"} err="failed to get container status \"b08205185b49f8f91e2fac0f1ee1b8adb70afe7d6201a1a78cf3b6718ac81bb6\": rpc error: code = NotFound desc = could not find container \"b08205185b49f8f91e2fac0f1ee1b8adb70afe7d6201a1a78cf3b6718ac81bb6\": container with ID starting with b08205185b49f8f91e2fac0f1ee1b8adb70afe7d6201a1a78cf3b6718ac81bb6 not found: ID does not exist" Oct 30 14:45:16.078739 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:16.078673 4982 scope.go:122] "RemoveContainer" containerID="de198dcfa2afe64c56d7e44af73543b23100debca344cf2a95b7aecc602f296e" Oct 30 14:45:16.090369 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:16.090338 4982 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3093abe-9b1e-492a-8837-4bd42ea39d2d-tmp-dir\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:16.090493 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:16.090374 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdbl4\" (UniqueName: \"kubernetes.io/projected/cbd754bd-771d-4007-8ac0-7e413dc24915-kube-api-access-qdbl4\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:16.090493 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:16.090387 4982 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3093abe-9b1e-492a-8837-4bd42ea39d2d-home\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:16.090493 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:16.090401 4982 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3093abe-9b1e-492a-8837-4bd42ea39d2d-tls-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:16.090493 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:16.090413 4982 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3093abe-9b1e-492a-8837-4bd42ea39d2d-model-cache\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:16.090493 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:16.090425 4982 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3093abe-9b1e-492a-8837-4bd42ea39d2d-dshm\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:16.090493 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:16.090438 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncp2n\" (UniqueName: \"kubernetes.io/projected/f3093abe-9b1e-492a-8837-4bd42ea39d2d-kube-api-access-ncp2n\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:16.090493 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:16.090451 4982 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbd754bd-771d-4007-8ac0-7e413dc24915-dshm\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:16.141100 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:16.140782 4982 scope.go:122] "RemoveContainer" containerID="de198dcfa2afe64c56d7e44af73543b23100debca344cf2a95b7aecc602f296e" Oct 30 14:45:16.141304 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:16.141250 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn"] Oct 30 14:45:16.141618 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:45:16.141541 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de198dcfa2afe64c56d7e44af73543b23100debca344cf2a95b7aecc602f296e\": container with ID starting with de198dcfa2afe64c56d7e44af73543b23100debca344cf2a95b7aecc602f296e not found: ID does not exist" containerID="de198dcfa2afe64c56d7e44af73543b23100debca344cf2a95b7aecc602f296e" Oct 30 14:45:16.141618 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:16.141575 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de198dcfa2afe64c56d7e44af73543b23100debca344cf2a95b7aecc602f296e"} err="failed to get container status \"de198dcfa2afe64c56d7e44af73543b23100debca344cf2a95b7aecc602f296e\": rpc error: code = NotFound desc = could not find container \"de198dcfa2afe64c56d7e44af73543b23100debca344cf2a95b7aecc602f296e\": container with ID starting with de198dcfa2afe64c56d7e44af73543b23100debca344cf2a95b7aecc602f296e not found: ID does not exist" Oct 30 14:45:16.144976 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:16.144599 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-ca229c/premium-w1-ca229c-kserve-545bb5d7b9-q2gvn"] Oct 30 14:45:16.148194 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:16.148175 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27" Oct 30 14:45:16.176580 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:16.176560 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8"] Oct 30 14:45:16.223665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:16.222175 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-ca229c/simulator-w1-ca229c-kserve-68bff4b9f6-fnwk8"] Oct 30 14:45:16.409718 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:16.396135 4982 scope.go:122] "RemoveContainer" containerID="038fe0c5c954dc5dcfba0045d19584a2e5f69b6134f4762c4e2c869ee0d79d63" Oct 30 14:45:16.409718 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:16.397036 4982 scope.go:122] "RemoveContainer" containerID="3d79ca8a6c25e6831a8e4b69149069db43f80e630dd3420f467c784483fd2914" Oct 30 14:45:16.429769 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:16.428729 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbd754bd-771d-4007-8ac0-7e413dc24915" path="/var/lib/kubelet/pods/cbd754bd-771d-4007-8ac0-7e413dc24915/volumes" Oct 30 14:45:16.429769 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:16.429307 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3093abe-9b1e-492a-8837-4bd42ea39d2d" path="/var/lib/kubelet/pods/f3093abe-9b1e-492a-8837-4bd42ea39d2d/volumes" Oct 30 14:45:16.929067 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:16.929029 4982 patch_prober.go:29] interesting pod/e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Oct 30 14:45:16.929232 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:16.929087 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27" podUID="9b853a87-0adf-46f0-b45d-c17b69a68495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Oct 30 14:45:17.077913 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:17.077874 4982 generic.go:356] "Generic (PLEG): container finished" podID="f95c53ac-ceef-47a4-abe7-aed7fba44238" containerID="e29b48ab7bac0c0ca4d4b30d4f9ead9a3b63f435202865cda8b59ab6b5682713" exitCode=60 Oct 30 14:45:17.078109 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:17.077966 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6e91df-29889525-cw745" event={"ID":"f95c53ac-ceef-47a4-abe7-aed7fba44238","Type":"ContainerDied","Data":"e29b48ab7bac0c0ca4d4b30d4f9ead9a3b63f435202865cda8b59ab6b5682713"} Oct 30 14:45:17.078109 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:17.078022 4982 scope.go:122] "RemoveContainer" containerID="6e02d67cd3c2aebc20ee841d4e7ca399272d86f5a56073da51cc679edd62bbd9" Oct 30 14:45:17.078550 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:17.078514 4982 scope.go:122] "RemoveContainer" containerID="e29b48ab7bac0c0ca4d4b30d4f9ead9a3b63f435202865cda8b59ab6b5682713" Oct 30 14:45:17.078999 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:45:17.078966 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-6e91df-29889525-cw745_odh-ai-gateway-infra(f95c53ac-ceef-47a4-abe7-aed7fba44238)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6e91df-29889525-cw745" podUID="f95c53ac-ceef-47a4-abe7-aed7fba44238" Oct 30 14:45:17.082252 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:17.082226 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889525-bfv9w" event={"ID":"c6dd7aa1-d123-4c72-889d-8a6371b7b483","Type":"ContainerStarted","Data":"9fe9a729d2a42778d80e324e2e3ded50f8eb8dc4e6e6832618ed4ff706c1be95"} Oct 30 14:45:17.087581 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:17.086935 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889525-hvqhb" event={"ID":"df8648aa-12d5-4d90-88ad-e7a1a8333142","Type":"ContainerStarted","Data":"e98fff1734be62c74a949bab458f4497b2c6326fcd6f6dd0f61535b3ee99b588"} Oct 30 14:45:17.089175 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:17.089156 4982 generic.go:356] "Generic (PLEG): container finished" podID="381a4942-5434-499a-9d21-6e07dce9926b" containerID="c06e91c7b16fe74434ac5a56d0f7d9f028cfeb3dc198da1248862b1bc1436144" exitCode=60 Oct 30 14:45:17.089297 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:17.089209 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889525-pjzwp" event={"ID":"381a4942-5434-499a-9d21-6e07dce9926b","Type":"ContainerDied","Data":"c06e91c7b16fe74434ac5a56d0f7d9f028cfeb3dc198da1248862b1bc1436144"} Oct 30 14:45:17.089717 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:17.089697 4982 scope.go:122] "RemoveContainer" containerID="c06e91c7b16fe74434ac5a56d0f7d9f028cfeb3dc198da1248862b1bc1436144" Oct 30 14:45:17.090021 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:45:17.089998 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-46b13b-29889525-pjzwp_odh-ai-gateway-infra(381a4942-5434-499a-9d21-6e07dce9926b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889525-pjzwp" podUID="381a4942-5434-499a-9d21-6e07dce9926b" Oct 30 14:45:17.097786 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:17.097768 4982 scope.go:122] "RemoveContainer" containerID="282f3cba92a3573839f3da495c0ee08934a931ed2c88fb4972ebd094e232a904" Oct 30 14:45:17.285014 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:17.284927 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:45:17.393008 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:17.392058 4982 scope.go:122] "RemoveContainer" containerID="ac14e6a45cf059eeee9968ac88e86f0eb6e0e8cf2c387a5ec7f48ff349eae62c" Oct 30 14:45:17.393008 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:17.392330 4982 scope.go:122] "RemoveContainer" containerID="6b33bfb4e0e2f2c2531897378fd2d0039317b537974038224dc04830428392d4" Oct 30 14:45:17.393477 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:17.393435 4982 scope.go:122] "RemoveContainer" containerID="52e69aae9b6a227d176f2c3c37177c43b1a3130ff1aaf4e937e9d1f82e0279d5" Oct 30 14:45:17.394023 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:17.393999 4982 scope.go:122] "RemoveContainer" containerID="a69539550df24c2a57356c92ef0f8d52c29860416cd65c3ac0426bc91867d912" Oct 30 14:45:17.433140 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:17.433105 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:45:17.577124 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:17.577098 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:45:17.935252 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:17.935218 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27" Oct 30 14:45:17.936357 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:17.936336 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27" Oct 30 14:45:18.108774 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:18.108731 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889525-pjgvt" event={"ID":"1ca2abf9-60f0-489a-99fa-70a5f4d44def","Type":"ContainerStarted","Data":"3f7bd7c245ff103fc3e67e81dd918edbb8812fff5df7108e36e617f8f12bf0ed"} Oct 30 14:45:18.112349 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:18.112323 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-1d6347-29889525-dqf8x" event={"ID":"1a89d4da-0eef-472e-bf20-613e750dd598","Type":"ContainerStarted","Data":"589b40a0ac5f4d5776a9f50a2eead9590f8581b1e5c4e0e31e95201f2f14aff3"} Oct 30 14:45:18.123576 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:18.123554 4982 generic.go:356] "Generic (PLEG): container finished" podID="c6dd7aa1-d123-4c72-889d-8a6371b7b483" containerID="9fe9a729d2a42778d80e324e2e3ded50f8eb8dc4e6e6832618ed4ff706c1be95" exitCode=60 Oct 30 14:45:18.123734 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:18.123684 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889525-bfv9w" event={"ID":"c6dd7aa1-d123-4c72-889d-8a6371b7b483","Type":"ContainerDied","Data":"9fe9a729d2a42778d80e324e2e3ded50f8eb8dc4e6e6832618ed4ff706c1be95"} Oct 30 14:45:18.123812 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:18.123757 4982 scope.go:122] "RemoveContainer" containerID="038fe0c5c954dc5dcfba0045d19584a2e5f69b6134f4762c4e2c869ee0d79d63" Oct 30 14:45:18.124397 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:18.124219 4982 scope.go:122] "RemoveContainer" containerID="9fe9a729d2a42778d80e324e2e3ded50f8eb8dc4e6e6832618ed4ff706c1be95" Oct 30 14:45:18.124742 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:45:18.124714 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-00926c-29889525-bfv9w_odh-ai-gateway-infra(c6dd7aa1-d123-4c72-889d-8a6371b7b483)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889525-bfv9w" podUID="c6dd7aa1-d123-4c72-889d-8a6371b7b483" Oct 30 14:45:18.127949 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:18.127926 4982 generic.go:356] "Generic (PLEG): container finished" podID="df8648aa-12d5-4d90-88ad-e7a1a8333142" containerID="e98fff1734be62c74a949bab458f4497b2c6326fcd6f6dd0f61535b3ee99b588" exitCode=60 Oct 30 14:45:18.128100 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:18.128048 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889525-hvqhb" event={"ID":"df8648aa-12d5-4d90-88ad-e7a1a8333142","Type":"ContainerDied","Data":"e98fff1734be62c74a949bab458f4497b2c6326fcd6f6dd0f61535b3ee99b588"} Oct 30 14:45:18.128523 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:18.128502 4982 scope.go:122] "RemoveContainer" containerID="e98fff1734be62c74a949bab458f4497b2c6326fcd6f6dd0f61535b3ee99b588" Oct 30 14:45:18.129022 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:45:18.128997 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-ca229c-29889525-hvqhb_odh-ai-gateway-infra(df8648aa-12d5-4d90-88ad-e7a1a8333142)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889525-hvqhb" podUID="df8648aa-12d5-4d90-88ad-e7a1a8333142" Oct 30 14:45:18.133087 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:18.133061 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-33c209-29889525-fj5nd" event={"ID":"d524b057-6219-43cf-a11b-5048f36a8dfd","Type":"ContainerStarted","Data":"bbe6216bf9d9803fc9978fb8baf707841830078d1170c01c60142f5cb9b2fcb7"} Oct 30 14:45:18.149191 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:18.143539 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6e91df-29889525-cw745"] Oct 30 14:45:18.155949 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:18.155913 4982 scope.go:122] "RemoveContainer" containerID="3d79ca8a6c25e6831a8e4b69149069db43f80e630dd3420f467c784483fd2914" Oct 30 14:45:18.192403 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:18.191155 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889525-bfv9w"] Oct 30 14:45:18.238690 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:18.236578 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889525-pjzwp"] Oct 30 14:45:18.288924 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:18.288893 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889525-hvqhb"] Oct 30 14:45:18.407314 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:18.406936 4982 scope.go:122] "RemoveContainer" containerID="5d4691d5b0d1f5d91dbb63d34bf1b195caccbd92a44c96df408728c7b19625aa" Oct 30 14:45:18.452048 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:18.452019 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6e91df-29889525-cw745" Oct 30 14:45:18.528800 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:18.528771 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889525-pjzwp" Oct 30 14:45:18.553335 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:18.553304 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/381a4942-5434-499a-9d21-6e07dce9926b-kube-api-access-qgnd5\" (UniqueName: \"kubernetes.io/projected/381a4942-5434-499a-9d21-6e07dce9926b-kube-api-access-qgnd5\") pod \"381a4942-5434-499a-9d21-6e07dce9926b\" (UID: \"381a4942-5434-499a-9d21-6e07dce9926b\") " Oct 30 14:45:18.553584 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:18.553561 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f95c53ac-ceef-47a4-abe7-aed7fba44238-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f95c53ac-ceef-47a4-abe7-aed7fba44238-maas-api-service-ca\") pod \"f95c53ac-ceef-47a4-abe7-aed7fba44238\" (UID: \"f95c53ac-ceef-47a4-abe7-aed7fba44238\") " Oct 30 14:45:18.553922 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:18.553902 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f95c53ac-ceef-47a4-abe7-aed7fba44238-kube-api-access-gtnph\" (UniqueName: \"kubernetes.io/projected/f95c53ac-ceef-47a4-abe7-aed7fba44238-kube-api-access-gtnph\") pod \"f95c53ac-ceef-47a4-abe7-aed7fba44238\" (UID: \"f95c53ac-ceef-47a4-abe7-aed7fba44238\") " Oct 30 14:45:18.554123 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:18.554104 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/381a4942-5434-499a-9d21-6e07dce9926b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/381a4942-5434-499a-9d21-6e07dce9926b-maas-api-service-ca\") pod \"381a4942-5434-499a-9d21-6e07dce9926b\" (UID: \"381a4942-5434-499a-9d21-6e07dce9926b\") " Oct 30 14:45:18.554427 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:18.554355 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f95c53ac-ceef-47a4-abe7-aed7fba44238-maas-api-service-ca" pod "f95c53ac-ceef-47a4-abe7-aed7fba44238" (UID: "f95c53ac-ceef-47a4-abe7-aed7fba44238"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:45:18.554926 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:18.554902 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f95c53ac-ceef-47a4-abe7-aed7fba44238-maas-api-service-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:18.555355 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:18.555333 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/381a4942-5434-499a-9d21-6e07dce9926b-maas-api-service-ca" pod "381a4942-5434-499a-9d21-6e07dce9926b" (UID: "381a4942-5434-499a-9d21-6e07dce9926b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:45:18.627806 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:18.627280 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f95c53ac-ceef-47a4-abe7-aed7fba44238-kube-api-access-gtnph" pod "f95c53ac-ceef-47a4-abe7-aed7fba44238" (UID: "f95c53ac-ceef-47a4-abe7-aed7fba44238"). InnerVolumeSpecName "kube-api-access-gtnph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:45:18.642874 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:18.642807 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/381a4942-5434-499a-9d21-6e07dce9926b-kube-api-access-qgnd5" pod "381a4942-5434-499a-9d21-6e07dce9926b" (UID: "381a4942-5434-499a-9d21-6e07dce9926b"). InnerVolumeSpecName "kube-api-access-qgnd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:45:18.656427 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:18.656390 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtnph\" (UniqueName: \"kubernetes.io/projected/f95c53ac-ceef-47a4-abe7-aed7fba44238-kube-api-access-gtnph\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:18.656575 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:18.656434 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/381a4942-5434-499a-9d21-6e07dce9926b-maas-api-service-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:18.656575 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:18.656448 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgnd5\" (UniqueName: \"kubernetes.io/projected/381a4942-5434-499a-9d21-6e07dce9926b-kube-api-access-qgnd5\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:19.145534 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.145501 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6e91df-29889525-cw745" event={"ID":"f95c53ac-ceef-47a4-abe7-aed7fba44238","Type":"ContainerDied","Data":"5f884511e91846e16eba8cb09cbfcb33b798b61df5b1c6388384205a5901f406"} Oct 30 14:45:19.145534 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.145519 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6e91df-29889525-cw745" Oct 30 14:45:19.145836 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.145564 4982 scope.go:122] "RemoveContainer" containerID="e29b48ab7bac0c0ca4d4b30d4f9ead9a3b63f435202865cda8b59ab6b5682713" Oct 30 14:45:19.147736 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.147717 4982 generic.go:356] "Generic (PLEG): container finished" podID="62a23269-9415-4a1f-b459-c6bb4a06cbae" containerID="975f4df0bbee5b2a80c0f5f1625d2fe38f53e65dbc988604cd86af1e612c2ef3" exitCode=60 Oct 30 14:45:19.147841 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.147799 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3b95dd-29889525-mkkbq" event={"ID":"62a23269-9415-4a1f-b459-c6bb4a06cbae","Type":"ContainerDied","Data":"975f4df0bbee5b2a80c0f5f1625d2fe38f53e65dbc988604cd86af1e612c2ef3"} Oct 30 14:45:19.148271 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.148250 4982 scope.go:122] "RemoveContainer" containerID="975f4df0bbee5b2a80c0f5f1625d2fe38f53e65dbc988604cd86af1e612c2ef3" Oct 30 14:45:19.148598 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:45:19.148569 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-3b95dd-29889525-mkkbq_odh-ai-gateway-infra(62a23269-9415-4a1f-b459-c6bb4a06cbae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3b95dd-29889525-mkkbq" podUID="62a23269-9415-4a1f-b459-c6bb4a06cbae" Oct 30 14:45:19.153279 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.153220 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889525-pjzwp" event={"ID":"381a4942-5434-499a-9d21-6e07dce9926b","Type":"ContainerDied","Data":"47c057534f9672c8d08c4452661c7ba5a1152f7cfe13f0cd98f69a8ae6c5a4cd"} Oct 30 14:45:19.153813 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.153783 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889525-pjzwp" Oct 30 14:45:19.157328 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.157307 4982 generic.go:356] "Generic (PLEG): container finished" podID="d524b057-6219-43cf-a11b-5048f36a8dfd" containerID="bbe6216bf9d9803fc9978fb8baf707841830078d1170c01c60142f5cb9b2fcb7" exitCode=60 Oct 30 14:45:19.157399 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.157381 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-33c209-29889525-fj5nd" event={"ID":"d524b057-6219-43cf-a11b-5048f36a8dfd","Type":"ContainerDied","Data":"bbe6216bf9d9803fc9978fb8baf707841830078d1170c01c60142f5cb9b2fcb7"} Oct 30 14:45:19.157680 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.157666 4982 scope.go:122] "RemoveContainer" containerID="bbe6216bf9d9803fc9978fb8baf707841830078d1170c01c60142f5cb9b2fcb7" Oct 30 14:45:19.158050 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:45:19.157949 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-33c209-29889525-fj5nd_odh-ai-gateway-infra(d524b057-6219-43cf-a11b-5048f36a8dfd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-33c209-29889525-fj5nd" podUID="d524b057-6219-43cf-a11b-5048f36a8dfd" Oct 30 14:45:19.160271 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.160159 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889525-hz26s" event={"ID":"4e292b6c-6d65-41cd-a634-0a33f9cd86a4","Type":"ContainerStarted","Data":"ecca7f9c92e10572108a9dcb6fd74b69da3ca4c0796f3474f6507d0d8b20431e"} Oct 30 14:45:19.160889 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.160824 4982 scope.go:122] "RemoveContainer" containerID="ac14e6a45cf059eeee9968ac88e86f0eb6e0e8cf2c387a5ec7f48ff349eae62c" Oct 30 14:45:19.164012 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.163991 4982 generic.go:356] "Generic (PLEG): container finished" podID="1ca2abf9-60f0-489a-99fa-70a5f4d44def" containerID="3f7bd7c245ff103fc3e67e81dd918edbb8812fff5df7108e36e617f8f12bf0ed" exitCode=60 Oct 30 14:45:19.164133 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.164056 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889525-pjgvt" event={"ID":"1ca2abf9-60f0-489a-99fa-70a5f4d44def","Type":"ContainerDied","Data":"3f7bd7c245ff103fc3e67e81dd918edbb8812fff5df7108e36e617f8f12bf0ed"} Oct 30 14:45:19.164352 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.164318 4982 scope.go:122] "RemoveContainer" containerID="3f7bd7c245ff103fc3e67e81dd918edbb8812fff5df7108e36e617f8f12bf0ed" Oct 30 14:45:19.164540 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:45:19.164513 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889525-pjgvt_odh-ai-gateway-infra(1ca2abf9-60f0-489a-99fa-70a5f4d44def)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889525-pjgvt" podUID="1ca2abf9-60f0-489a-99fa-70a5f4d44def" Oct 30 14:45:19.167604 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.167523 4982 generic.go:356] "Generic (PLEG): container finished" podID="1a89d4da-0eef-472e-bf20-613e750dd598" containerID="589b40a0ac5f4d5776a9f50a2eead9590f8581b1e5c4e0e31e95201f2f14aff3" exitCode=60 Oct 30 14:45:19.167604 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.167554 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-1d6347-29889525-dqf8x" event={"ID":"1a89d4da-0eef-472e-bf20-613e750dd598","Type":"ContainerDied","Data":"589b40a0ac5f4d5776a9f50a2eead9590f8581b1e5c4e0e31e95201f2f14aff3"} Oct 30 14:45:19.168402 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.168098 4982 scope.go:122] "RemoveContainer" containerID="589b40a0ac5f4d5776a9f50a2eead9590f8581b1e5c4e0e31e95201f2f14aff3" Oct 30 14:45:19.168402 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:45:19.168345 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-1d6347-29889525-dqf8x_odh-ai-gateway-infra(1a89d4da-0eef-472e-bf20-613e750dd598)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-1d6347-29889525-dqf8x" podUID="1a89d4da-0eef-472e-bf20-613e750dd598" Oct 30 14:45:19.178723 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.178694 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889525-pjgvt"] Oct 30 14:45:19.215237 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.214459 4982 scope.go:122] "RemoveContainer" containerID="c06e91c7b16fe74434ac5a56d0f7d9f028cfeb3dc198da1248862b1bc1436144" Oct 30 14:45:19.224691 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.224325 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-33c209-29889525-fj5nd"] Oct 30 14:45:19.235924 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.235415 4982 scope.go:122] "RemoveContainer" containerID="52e69aae9b6a227d176f2c3c37177c43b1a3130ff1aaf4e937e9d1f82e0279d5" Oct 30 14:45:19.255906 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.253358 4982 scope.go:122] "RemoveContainer" containerID="6b33bfb4e0e2f2c2531897378fd2d0039317b537974038224dc04830428392d4" Oct 30 14:45:19.292717 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.288711 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889525-pjzwp"] Oct 30 14:45:19.295657 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.295190 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-46b13b-29889525-pjzwp"] Oct 30 14:45:19.312508 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.311956 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:45:19.332984 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.332783 4982 scope.go:122] "RemoveContainer" containerID="a69539550df24c2a57356c92ef0f8d52c29860416cd65c3ac0426bc91867d912" Oct 30 14:45:19.394670 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.394623 4982 scope.go:122] "RemoveContainer" containerID="89a2c29a4dd95a6ba99405575ce22f9478035787f1e7817a9fa33d55450b2595" Oct 30 14:45:19.408283 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.408225 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889525-hvqhb" Oct 30 14:45:19.412697 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.412674 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-1d6347-29889525-dqf8x"] Oct 30 14:45:19.479618 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.479594 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df8648aa-12d5-4d90-88ad-e7a1a8333142-kube-api-access-9dqbc\" (UniqueName: \"kubernetes.io/projected/df8648aa-12d5-4d90-88ad-e7a1a8333142-kube-api-access-9dqbc\") pod \"df8648aa-12d5-4d90-88ad-e7a1a8333142\" (UID: \"df8648aa-12d5-4d90-88ad-e7a1a8333142\") " Oct 30 14:45:19.479924 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.479628 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df8648aa-12d5-4d90-88ad-e7a1a8333142-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df8648aa-12d5-4d90-88ad-e7a1a8333142-maas-api-service-ca\") pod \"df8648aa-12d5-4d90-88ad-e7a1a8333142\" (UID: \"df8648aa-12d5-4d90-88ad-e7a1a8333142\") " Oct 30 14:45:19.482916 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.480902 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df8648aa-12d5-4d90-88ad-e7a1a8333142-maas-api-service-ca" pod "df8648aa-12d5-4d90-88ad-e7a1a8333142" (UID: "df8648aa-12d5-4d90-88ad-e7a1a8333142"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:45:19.580842 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.580814 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df8648aa-12d5-4d90-88ad-e7a1a8333142-kube-api-access-9dqbc" pod "df8648aa-12d5-4d90-88ad-e7a1a8333142" (UID: "df8648aa-12d5-4d90-88ad-e7a1a8333142"). InnerVolumeSpecName "kube-api-access-9dqbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:45:19.584968 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.583872 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dqbc\" (UniqueName: \"kubernetes.io/projected/df8648aa-12d5-4d90-88ad-e7a1a8333142-kube-api-access-9dqbc\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:19.584968 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.583903 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df8648aa-12d5-4d90-88ad-e7a1a8333142-maas-api-service-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:19.617748 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.617722 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6e91df-29889525-cw745"] Oct 30 14:45:19.621863 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.621835 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6e91df-29889525-cw745"] Oct 30 14:45:19.678600 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.678580 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889525-bfv9w" Oct 30 14:45:19.770048 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.770015 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx"] Oct 30 14:45:19.770795 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.770777 4982 cpu_manager.go:414] "containerMap: removing container" podUID="cbd754bd-771d-4007-8ac0-7e413dc24915" containerName="main" Oct 30 14:45:19.770795 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.770797 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbd754bd-771d-4007-8ac0-7e413dc24915" containerName="main" Oct 30 14:45:19.770929 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.770864 4982 cpu_manager.go:414] "containerMap: removing container" podUID="381a4942-5434-499a-9d21-6e07dce9926b" containerName="cleanup" Oct 30 14:45:19.770929 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.770872 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="381a4942-5434-499a-9d21-6e07dce9926b" containerName="cleanup" Oct 30 14:45:19.770929 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.770894 4982 cpu_manager.go:414] "containerMap: removing container" podUID="df8648aa-12d5-4d90-88ad-e7a1a8333142" containerName="cleanup" Oct 30 14:45:19.770929 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.770903 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df8648aa-12d5-4d90-88ad-e7a1a8333142" containerName="cleanup" Oct 30 14:45:19.770929 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.770920 4982 cpu_manager.go:414] "containerMap: removing container" podUID="c6dd7aa1-d123-4c72-889d-8a6371b7b483" containerName="cleanup" Oct 30 14:45:19.771143 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.770930 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6dd7aa1-d123-4c72-889d-8a6371b7b483" containerName="cleanup" Oct 30 14:45:19.771143 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.770943 4982 cpu_manager.go:414] "containerMap: removing container" podUID="f95c53ac-ceef-47a4-abe7-aed7fba44238" containerName="cleanup" Oct 30 14:45:19.771143 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.770951 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f95c53ac-ceef-47a4-abe7-aed7fba44238" containerName="cleanup" Oct 30 14:45:19.771143 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.770975 4982 cpu_manager.go:414] "containerMap: removing container" podUID="381a4942-5434-499a-9d21-6e07dce9926b" containerName="cleanup" Oct 30 14:45:19.771143 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.770984 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="381a4942-5434-499a-9d21-6e07dce9926b" containerName="cleanup" Oct 30 14:45:19.771143 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.771004 4982 cpu_manager.go:414] "containerMap: removing container" podUID="c6dd7aa1-d123-4c72-889d-8a6371b7b483" containerName="cleanup" Oct 30 14:45:19.771143 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.771011 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6dd7aa1-d123-4c72-889d-8a6371b7b483" containerName="cleanup" Oct 30 14:45:19.771143 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.771041 4982 cpu_manager.go:414] "containerMap: removing container" podUID="c6dd7aa1-d123-4c72-889d-8a6371b7b483" containerName="cleanup" Oct 30 14:45:19.771143 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.771048 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6dd7aa1-d123-4c72-889d-8a6371b7b483" containerName="cleanup" Oct 30 14:45:19.771143 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.771066 4982 cpu_manager.go:414] "containerMap: removing container" podUID="8ef323f9-5aa4-48c4-826d-4d1b9ab83126" containerName="revoke-keys" Oct 30 14:45:19.771143 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.771074 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ef323f9-5aa4-48c4-826d-4d1b9ab83126" containerName="revoke-keys" Oct 30 14:45:19.771143 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.771093 4982 cpu_manager.go:414] "containerMap: removing container" podUID="f3093abe-9b1e-492a-8837-4bd42ea39d2d" containerName="main" Oct 30 14:45:19.771143 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.771099 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3093abe-9b1e-492a-8837-4bd42ea39d2d" containerName="main" Oct 30 14:45:19.771143 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.771116 4982 cpu_manager.go:414] "containerMap: removing container" podUID="381a4942-5434-499a-9d21-6e07dce9926b" containerName="cleanup" Oct 30 14:45:19.771143 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.771125 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="381a4942-5434-499a-9d21-6e07dce9926b" containerName="cleanup" Oct 30 14:45:19.771143 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.771134 4982 cpu_manager.go:414] "containerMap: removing container" podUID="df8648aa-12d5-4d90-88ad-e7a1a8333142" containerName="cleanup" Oct 30 14:45:19.771143 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.771141 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df8648aa-12d5-4d90-88ad-e7a1a8333142" containerName="cleanup" Oct 30 14:45:19.771143 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.771153 4982 cpu_manager.go:414] "containerMap: removing container" podUID="df8648aa-12d5-4d90-88ad-e7a1a8333142" containerName="cleanup" Oct 30 14:45:19.772013 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.771161 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df8648aa-12d5-4d90-88ad-e7a1a8333142" containerName="cleanup" Oct 30 14:45:19.772013 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.771171 4982 cpu_manager.go:414] "containerMap: removing container" podUID="f95c53ac-ceef-47a4-abe7-aed7fba44238" containerName="cleanup" Oct 30 14:45:19.772013 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.771178 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f95c53ac-ceef-47a4-abe7-aed7fba44238" containerName="cleanup" Oct 30 14:45:19.772013 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.771375 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6dd7aa1-d123-4c72-889d-8a6371b7b483" containerName="cleanup" Oct 30 14:45:19.772013 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.771386 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ef323f9-5aa4-48c4-826d-4d1b9ab83126" containerName="revoke-keys" Oct 30 14:45:19.772013 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.771400 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3093abe-9b1e-492a-8837-4bd42ea39d2d" containerName="main" Oct 30 14:45:19.772013 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.771411 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="381a4942-5434-499a-9d21-6e07dce9926b" containerName="cleanup" Oct 30 14:45:19.772013 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.771419 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="df8648aa-12d5-4d90-88ad-e7a1a8333142" containerName="cleanup" Oct 30 14:45:19.772013 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.771428 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="df8648aa-12d5-4d90-88ad-e7a1a8333142" containerName="cleanup" Oct 30 14:45:19.772013 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.771435 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="f95c53ac-ceef-47a4-abe7-aed7fba44238" containerName="cleanup" Oct 30 14:45:19.772013 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.771444 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="f95c53ac-ceef-47a4-abe7-aed7fba44238" containerName="cleanup" Oct 30 14:45:19.772013 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.771454 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbd754bd-771d-4007-8ac0-7e413dc24915" containerName="main" Oct 30 14:45:19.772013 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.771464 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6dd7aa1-d123-4c72-889d-8a6371b7b483" containerName="cleanup" Oct 30 14:45:19.772013 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.771475 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="f95c53ac-ceef-47a4-abe7-aed7fba44238" containerName="cleanup" Oct 30 14:45:19.772013 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.771487 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="381a4942-5434-499a-9d21-6e07dce9926b" containerName="cleanup" Oct 30 14:45:19.772013 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.771498 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6dd7aa1-d123-4c72-889d-8a6371b7b483" containerName="cleanup" Oct 30 14:45:19.775425 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.775404 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx" Oct 30 14:45:19.786931 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.786904 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6dd7aa1-d123-4c72-889d-8a6371b7b483-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6dd7aa1-d123-4c72-889d-8a6371b7b483-maas-api-service-ca\") pod \"c6dd7aa1-d123-4c72-889d-8a6371b7b483\" (UID: \"c6dd7aa1-d123-4c72-889d-8a6371b7b483\") " Oct 30 14:45:19.788041 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.787981 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6dd7aa1-d123-4c72-889d-8a6371b7b483-kube-api-access-9z7vp\" (UniqueName: \"kubernetes.io/projected/c6dd7aa1-d123-4c72-889d-8a6371b7b483-kube-api-access-9z7vp\") pod \"c6dd7aa1-d123-4c72-889d-8a6371b7b483\" (UID: \"c6dd7aa1-d123-4c72-889d-8a6371b7b483\") " Oct 30 14:45:19.790424 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.790383 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6dd7aa1-d123-4c72-889d-8a6371b7b483-maas-api-service-ca" pod "c6dd7aa1-d123-4c72-889d-8a6371b7b483" (UID: "c6dd7aa1-d123-4c72-889d-8a6371b7b483"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:45:19.797816 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.797785 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-f0e8b9-openshift-default-dockercfg-fhs75\"" Oct 30 14:45:19.798227 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.798193 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6dd7aa1-d123-4c72-889d-8a6371b7b483-kube-api-access-9z7vp" pod "c6dd7aa1-d123-4c72-889d-8a6371b7b483" (UID: "c6dd7aa1-d123-4c72-889d-8a6371b7b483"). InnerVolumeSpecName "kube-api-access-9z7vp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:45:19.814574 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.814546 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx"] Oct 30 14:45:19.891170 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.891131 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19c9ad1b-96d7-41de-a3cc-0960ef9c31b8-workload-certs\") pod \"e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx\" (UID: \"19c9ad1b-96d7-41de-a3cc-0960ef9c31b8\") " pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx" Oct 30 14:45:19.891337 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.891195 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c9ad1b-96d7-41de-a3cc-0960ef9c31b8-workload-socket\") pod \"e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx\" (UID: \"19c9ad1b-96d7-41de-a3cc-0960ef9c31b8\") " pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx" Oct 30 14:45:19.891337 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.891254 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19c9ad1b-96d7-41de-a3cc-0960ef9c31b8-istio-envoy\") pod \"e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx\" (UID: \"19c9ad1b-96d7-41de-a3cc-0960ef9c31b8\") " pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx" Oct 30 14:45:19.891337 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.891285 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19c9ad1b-96d7-41de-a3cc-0960ef9c31b8-istiod-ca-cert\") pod \"e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx\" (UID: \"19c9ad1b-96d7-41de-a3cc-0960ef9c31b8\") " pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx" Oct 30 14:45:19.891536 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.891340 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19c9ad1b-96d7-41de-a3cc-0960ef9c31b8-istio-podinfo\") pod \"e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx\" (UID: \"19c9ad1b-96d7-41de-a3cc-0960ef9c31b8\") " pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx" Oct 30 14:45:19.891536 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.891370 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19c9ad1b-96d7-41de-a3cc-0960ef9c31b8-istio-data\") pod \"e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx\" (UID: \"19c9ad1b-96d7-41de-a3cc-0960ef9c31b8\") " pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx" Oct 30 14:45:19.891757 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.891562 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvc7w\" (UniqueName: \"kubernetes.io/projected/19c9ad1b-96d7-41de-a3cc-0960ef9c31b8-kube-api-access-cvc7w\") pod \"e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx\" (UID: \"19c9ad1b-96d7-41de-a3cc-0960ef9c31b8\") " pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx" Oct 30 14:45:19.891757 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.891632 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c9ad1b-96d7-41de-a3cc-0960ef9c31b8-credential-socket\") pod \"e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx\" (UID: \"19c9ad1b-96d7-41de-a3cc-0960ef9c31b8\") " pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx" Oct 30 14:45:19.891757 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.891685 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19c9ad1b-96d7-41de-a3cc-0960ef9c31b8-istio-token\") pod \"e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx\" (UID: \"19c9ad1b-96d7-41de-a3cc-0960ef9c31b8\") " pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx" Oct 30 14:45:19.891940 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.891758 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9z7vp\" (UniqueName: \"kubernetes.io/projected/c6dd7aa1-d123-4c72-889d-8a6371b7b483-kube-api-access-9z7vp\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:19.891940 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.891774 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6dd7aa1-d123-4c72-889d-8a6371b7b483-maas-api-service-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:19.946953 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.946922 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-1d6347-7778f9d655-545gn"] Oct 30 14:45:19.993279 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.993251 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvc7w\" (UniqueName: \"kubernetes.io/projected/19c9ad1b-96d7-41de-a3cc-0960ef9c31b8-kube-api-access-cvc7w\") pod \"e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx\" (UID: \"19c9ad1b-96d7-41de-a3cc-0960ef9c31b8\") " pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx" Oct 30 14:45:19.993778 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.993754 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c9ad1b-96d7-41de-a3cc-0960ef9c31b8-credential-socket\") pod \"e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx\" (UID: \"19c9ad1b-96d7-41de-a3cc-0960ef9c31b8\") " pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx" Oct 30 14:45:19.994309 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.994292 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19c9ad1b-96d7-41de-a3cc-0960ef9c31b8-istio-token\") pod \"e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx\" (UID: \"19c9ad1b-96d7-41de-a3cc-0960ef9c31b8\") " pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx" Oct 30 14:45:19.994723 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.994702 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19c9ad1b-96d7-41de-a3cc-0960ef9c31b8-workload-certs\") pod \"e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx\" (UID: \"19c9ad1b-96d7-41de-a3cc-0960ef9c31b8\") " pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx" Oct 30 14:45:19.995140 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.995124 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c9ad1b-96d7-41de-a3cc-0960ef9c31b8-workload-socket\") pod \"e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx\" (UID: \"19c9ad1b-96d7-41de-a3cc-0960ef9c31b8\") " pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx" Oct 30 14:45:19.995603 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.995584 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19c9ad1b-96d7-41de-a3cc-0960ef9c31b8-istio-envoy\") pod \"e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx\" (UID: \"19c9ad1b-96d7-41de-a3cc-0960ef9c31b8\") " pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx" Oct 30 14:45:19.995763 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.995747 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19c9ad1b-96d7-41de-a3cc-0960ef9c31b8-istiod-ca-cert\") pod \"e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx\" (UID: \"19c9ad1b-96d7-41de-a3cc-0960ef9c31b8\") " pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx" Oct 30 14:45:19.996601 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.996583 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19c9ad1b-96d7-41de-a3cc-0960ef9c31b8-istio-podinfo\") pod \"e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx\" (UID: \"19c9ad1b-96d7-41de-a3cc-0960ef9c31b8\") " pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx" Oct 30 14:45:19.997149 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.997131 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19c9ad1b-96d7-41de-a3cc-0960ef9c31b8-istio-data\") pod \"e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx\" (UID: \"19c9ad1b-96d7-41de-a3cc-0960ef9c31b8\") " pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx" Oct 30 14:45:19.997543 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.994236 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c9ad1b-96d7-41de-a3cc-0960ef9c31b8-credential-socket\") pod \"e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx\" (UID: \"19c9ad1b-96d7-41de-a3cc-0960ef9c31b8\") " pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx" Oct 30 14:45:19.997728 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.996496 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19c9ad1b-96d7-41de-a3cc-0960ef9c31b8-istiod-ca-cert\") pod \"e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx\" (UID: \"19c9ad1b-96d7-41de-a3cc-0960ef9c31b8\") " pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx" Oct 30 14:45:19.997845 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.995505 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c9ad1b-96d7-41de-a3cc-0960ef9c31b8-workload-socket\") pod \"e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx\" (UID: \"19c9ad1b-96d7-41de-a3cc-0960ef9c31b8\") " pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx" Oct 30 14:45:19.997937 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.995059 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19c9ad1b-96d7-41de-a3cc-0960ef9c31b8-workload-certs\") pod \"e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx\" (UID: \"19c9ad1b-96d7-41de-a3cc-0960ef9c31b8\") " pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx" Oct 30 14:45:19.998018 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.997534 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19c9ad1b-96d7-41de-a3cc-0960ef9c31b8-istio-data\") pod \"e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx\" (UID: \"19c9ad1b-96d7-41de-a3cc-0960ef9c31b8\") " pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx" Oct 30 14:45:19.999445 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:19.999419 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19c9ad1b-96d7-41de-a3cc-0960ef9c31b8-istio-envoy\") pod \"e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx\" (UID: \"19c9ad1b-96d7-41de-a3cc-0960ef9c31b8\") " pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx" Oct 30 14:45:20.000422 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:20.000363 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19c9ad1b-96d7-41de-a3cc-0960ef9c31b8-istio-podinfo\") pod \"e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx\" (UID: \"19c9ad1b-96d7-41de-a3cc-0960ef9c31b8\") " pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx" Oct 30 14:45:20.033593 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:20.032988 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19c9ad1b-96d7-41de-a3cc-0960ef9c31b8-istio-token\") pod \"e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx\" (UID: \"19c9ad1b-96d7-41de-a3cc-0960ef9c31b8\") " pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx" Oct 30 14:45:20.033593 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:20.033549 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvc7w\" (UniqueName: \"kubernetes.io/projected/19c9ad1b-96d7-41de-a3cc-0960ef9c31b8-kube-api-access-cvc7w\") pod \"e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx\" (UID: \"19c9ad1b-96d7-41de-a3cc-0960ef9c31b8\") " pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx" Oct 30 14:45:20.092456 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:20.092409 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:45:20.098382 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:20.098360 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx" Oct 30 14:45:20.216529 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:20.212326 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2927fd-29889525-srgtf" event={"ID":"620719c8-c4d6-4f56-a843-24137f57fec0","Type":"ContainerStarted","Data":"de03e6525209e0e0e411dc40f9b88c81e9a8fb333c17ae82da8d82ae183d809d"} Oct 30 14:45:20.225604 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:20.225510 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889525-bfv9w" event={"ID":"c6dd7aa1-d123-4c72-889d-8a6371b7b483","Type":"ContainerDied","Data":"755213e9eb86d7bf34026af7c1d812e890a6057d105ffba449ad8ff1215b1d87"} Oct 30 14:45:20.225835 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:20.225759 4982 scope.go:122] "RemoveContainer" containerID="9fe9a729d2a42778d80e324e2e3ded50f8eb8dc4e6e6832618ed4ff706c1be95" Oct 30 14:45:20.226087 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:20.225988 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889525-bfv9w" Oct 30 14:45:20.238827 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:20.231063 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889525-hvqhb" event={"ID":"df8648aa-12d5-4d90-88ad-e7a1a8333142","Type":"ContainerDied","Data":"fcce525882d7299a22482a99c1f77860af60a58562c32271df28bcd25a66f487"} Oct 30 14:45:20.238827 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:20.231157 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889525-hvqhb" Oct 30 14:45:20.243624 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:20.242354 4982 generic.go:356] "Generic (PLEG): container finished" podID="4e292b6c-6d65-41cd-a634-0a33f9cd86a4" containerID="ecca7f9c92e10572108a9dcb6fd74b69da3ca4c0796f3474f6507d0d8b20431e" exitCode=60 Oct 30 14:45:20.243624 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:20.242461 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889525-hz26s" event={"ID":"4e292b6c-6d65-41cd-a634-0a33f9cd86a4","Type":"ContainerDied","Data":"ecca7f9c92e10572108a9dcb6fd74b69da3ca4c0796f3474f6507d0d8b20431e"} Oct 30 14:45:20.243624 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:20.242928 4982 scope.go:122] "RemoveContainer" containerID="ecca7f9c92e10572108a9dcb6fd74b69da3ca4c0796f3474f6507d0d8b20431e" Oct 30 14:45:20.243624 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:45:20.243243 4982 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889525-hz26s_odh-ai-gateway-infra(4e292b6c-6d65-41cd-a634-0a33f9cd86a4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889525-hz26s" podUID="4e292b6c-6d65-41cd-a634-0a33f9cd86a4" Oct 30 14:45:20.248909 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:20.248843 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:45:20.277398 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:20.277375 4982 scope.go:122] "RemoveContainer" containerID="e98fff1734be62c74a949bab458f4497b2c6326fcd6f6dd0f61535b3ee99b588" Oct 30 14:45:20.334607 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:20.334580 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889525-hz26s"] Oct 30 14:45:20.345808 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:20.343894 4982 scope.go:122] "RemoveContainer" containerID="5d4691d5b0d1f5d91dbb63d34bf1b195caccbd92a44c96df408728c7b19625aa" Oct 30 14:45:20.409012 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:20.408980 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="381a4942-5434-499a-9d21-6e07dce9926b" path="/var/lib/kubelet/pods/381a4942-5434-499a-9d21-6e07dce9926b/volumes" Oct 30 14:45:20.409906 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:20.409884 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f95c53ac-ceef-47a4-abe7-aed7fba44238" path="/var/lib/kubelet/pods/f95c53ac-ceef-47a4-abe7-aed7fba44238/volumes" Oct 30 14:45:20.426136 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:20.426086 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889525-hvqhb"] Oct 30 14:45:20.447789 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:45:20.447766 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19c9ad1b_96d7_41de_a3cc_0960ef9c31b8.slice/crio-92b27f1b26638f5ec435a3333ba39e849a39db954688f7492490450f14c3a5f2 WatchSource:0}: Error finding container 92b27f1b26638f5ec435a3333ba39e849a39db954688f7492490450f14c3a5f2: Status 404 returned error can't find the container with id 92b27f1b26638f5ec435a3333ba39e849a39db954688f7492490450f14c3a5f2 Oct 30 14:45:20.451903 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:20.451852 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:45:20.452102 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:20.452076 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:45:20.452248 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:20.452226 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:45:20.461737 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:20.459711 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889525-hvqhb"] Oct 30 14:45:20.463000 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:20.462975 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx"] Oct 30 14:45:20.564788 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:20.564764 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889525-pjgvt" Oct 30 14:45:20.578367 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:20.578342 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-1d6347-29889525-dqf8x" Oct 30 14:45:20.598764 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:20.584979 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-33c209-29889525-fj5nd" Oct 30 14:45:20.617686 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:20.607331 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889525-bfv9w"] Oct 30 14:45:20.617686 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:20.608091 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3b95dd-29889525-mkkbq"] Oct 30 14:45:20.617686 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:20.616058 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889525-bfv9w"] Oct 30 14:45:20.659663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:20.653529 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ca2abf9-60f0-489a-99fa-70a5f4d44def-kube-api-access-544ll\" (UniqueName: \"kubernetes.io/projected/1ca2abf9-60f0-489a-99fa-70a5f4d44def-kube-api-access-544ll\") pod \"1ca2abf9-60f0-489a-99fa-70a5f4d44def\" (UID: \"1ca2abf9-60f0-489a-99fa-70a5f4d44def\") " Oct 30 14:45:20.659663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:20.653713 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d524b057-6219-43cf-a11b-5048f36a8dfd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d524b057-6219-43cf-a11b-5048f36a8dfd-maas-api-service-ca\") pod \"d524b057-6219-43cf-a11b-5048f36a8dfd\" (UID: \"d524b057-6219-43cf-a11b-5048f36a8dfd\") " Oct 30 14:45:20.659663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:20.653758 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a89d4da-0eef-472e-bf20-613e750dd598-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a89d4da-0eef-472e-bf20-613e750dd598-maas-api-service-ca\") pod \"1a89d4da-0eef-472e-bf20-613e750dd598\" (UID: \"1a89d4da-0eef-472e-bf20-613e750dd598\") " Oct 30 14:45:20.659663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:20.653804 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a89d4da-0eef-472e-bf20-613e750dd598-kube-api-access-hcs2t\" (UniqueName: \"kubernetes.io/projected/1a89d4da-0eef-472e-bf20-613e750dd598-kube-api-access-hcs2t\") pod \"1a89d4da-0eef-472e-bf20-613e750dd598\" (UID: \"1a89d4da-0eef-472e-bf20-613e750dd598\") " Oct 30 14:45:20.659663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:20.653907 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ca2abf9-60f0-489a-99fa-70a5f4d44def-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ca2abf9-60f0-489a-99fa-70a5f4d44def-maas-api-service-ca\") pod \"1ca2abf9-60f0-489a-99fa-70a5f4d44def\" (UID: \"1ca2abf9-60f0-489a-99fa-70a5f4d44def\") " Oct 30 14:45:20.659663 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:20.653974 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d524b057-6219-43cf-a11b-5048f36a8dfd-kube-api-access-4d8v5\" (UniqueName: \"kubernetes.io/projected/d524b057-6219-43cf-a11b-5048f36a8dfd-kube-api-access-4d8v5\") pod \"d524b057-6219-43cf-a11b-5048f36a8dfd\" (UID: \"d524b057-6219-43cf-a11b-5048f36a8dfd\") " Oct 30 14:45:20.692634 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:20.692275 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ca2abf9-60f0-489a-99fa-70a5f4d44def-maas-api-service-ca" pod "1ca2abf9-60f0-489a-99fa-70a5f4d44def" (UID: "1ca2abf9-60f0-489a-99fa-70a5f4d44def"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:45:20.694909 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:20.694882 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d524b057-6219-43cf-a11b-5048f36a8dfd-maas-api-service-ca" pod "d524b057-6219-43cf-a11b-5048f36a8dfd" (UID: "d524b057-6219-43cf-a11b-5048f36a8dfd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:45:20.695346 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:20.695317 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a89d4da-0eef-472e-bf20-613e750dd598-maas-api-service-ca" pod "1a89d4da-0eef-472e-bf20-613e750dd598" (UID: "1a89d4da-0eef-472e-bf20-613e750dd598"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:45:20.701563 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:20.701521 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a89d4da-0eef-472e-bf20-613e750dd598-kube-api-access-hcs2t" pod "1a89d4da-0eef-472e-bf20-613e750dd598" (UID: "1a89d4da-0eef-472e-bf20-613e750dd598"). InnerVolumeSpecName "kube-api-access-hcs2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:45:20.701952 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:20.701911 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d524b057-6219-43cf-a11b-5048f36a8dfd-kube-api-access-4d8v5" pod "d524b057-6219-43cf-a11b-5048f36a8dfd" (UID: "d524b057-6219-43cf-a11b-5048f36a8dfd"). InnerVolumeSpecName "kube-api-access-4d8v5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:45:20.702184 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:20.702150 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ca2abf9-60f0-489a-99fa-70a5f4d44def-kube-api-access-544ll" pod "1ca2abf9-60f0-489a-99fa-70a5f4d44def" (UID: "1ca2abf9-60f0-489a-99fa-70a5f4d44def"). InnerVolumeSpecName "kube-api-access-544ll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:45:20.798961 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:20.797050 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d524b057-6219-43cf-a11b-5048f36a8dfd-maas-api-service-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:20.798961 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:20.797090 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a89d4da-0eef-472e-bf20-613e750dd598-maas-api-service-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:20.798961 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:20.797105 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcs2t\" (UniqueName: \"kubernetes.io/projected/1a89d4da-0eef-472e-bf20-613e750dd598-kube-api-access-hcs2t\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:20.798961 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:20.797120 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ca2abf9-60f0-489a-99fa-70a5f4d44def-maas-api-service-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:20.798961 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:20.797134 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4d8v5\" (UniqueName: \"kubernetes.io/projected/d524b057-6219-43cf-a11b-5048f36a8dfd-kube-api-access-4d8v5\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:20.798961 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:20.797147 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-544ll\" (UniqueName: \"kubernetes.io/projected/1ca2abf9-60f0-489a-99fa-70a5f4d44def-kube-api-access-544ll\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:20.951199 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:20.951133 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3b95dd-29889525-mkkbq" Oct 30 14:45:21.005198 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:21.005166 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62a23269-9415-4a1f-b459-c6bb4a06cbae-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62a23269-9415-4a1f-b459-c6bb4a06cbae-maas-api-service-ca\") pod \"62a23269-9415-4a1f-b459-c6bb4a06cbae\" (UID: \"62a23269-9415-4a1f-b459-c6bb4a06cbae\") " Oct 30 14:45:21.005394 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:21.005377 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62a23269-9415-4a1f-b459-c6bb4a06cbae-kube-api-access-5zmgz\" (UniqueName: \"kubernetes.io/projected/62a23269-9415-4a1f-b459-c6bb4a06cbae-kube-api-access-5zmgz\") pod \"62a23269-9415-4a1f-b459-c6bb4a06cbae\" (UID: \"62a23269-9415-4a1f-b459-c6bb4a06cbae\") " Oct 30 14:45:21.005535 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:21.005512 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62a23269-9415-4a1f-b459-c6bb4a06cbae-maas-api-service-ca" pod "62a23269-9415-4a1f-b459-c6bb4a06cbae" (UID: "62a23269-9415-4a1f-b459-c6bb4a06cbae"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:45:21.006553 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:21.006527 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62a23269-9415-4a1f-b459-c6bb4a06cbae-maas-api-service-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:21.008382 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:21.008353 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62a23269-9415-4a1f-b459-c6bb4a06cbae-kube-api-access-5zmgz" pod "62a23269-9415-4a1f-b459-c6bb4a06cbae" (UID: "62a23269-9415-4a1f-b459-c6bb4a06cbae"). InnerVolumeSpecName "kube-api-access-5zmgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:45:21.107873 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:21.107840 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zmgz\" (UniqueName: \"kubernetes.io/projected/62a23269-9415-4a1f-b459-c6bb4a06cbae-kube-api-access-5zmgz\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:21.258118 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:21.258035 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889525-pjgvt" event={"ID":"1ca2abf9-60f0-489a-99fa-70a5f4d44def","Type":"ContainerDied","Data":"155cc567fff2ee58a2801da7051c853dea7bc81c01b3f1d038a64a452dc989fe"} Oct 30 14:45:21.258272 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:21.258143 4982 scope.go:122] "RemoveContainer" containerID="3f7bd7c245ff103fc3e67e81dd918edbb8812fff5df7108e36e617f8f12bf0ed" Oct 30 14:45:21.258272 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:21.258046 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889525-pjgvt" Oct 30 14:45:21.260541 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:21.260059 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-1d6347-29889525-dqf8x" Oct 30 14:45:21.260541 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:21.260201 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-1d6347-29889525-dqf8x" event={"ID":"1a89d4da-0eef-472e-bf20-613e750dd598","Type":"ContainerDied","Data":"205f09f335e3eebd461c5d161f33b3609f4b10cb6fed3e83a1e08deee161043c"} Oct 30 14:45:21.270632 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:21.270603 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx" event={"ID":"19c9ad1b-96d7-41de-a3cc-0960ef9c31b8","Type":"ContainerStarted","Data":"0343b9bee9368249e29c13772a84c585addf6942ee565d663c9f8ff2541e86b5"} Oct 30 14:45:21.270771 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:21.270690 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx" event={"ID":"19c9ad1b-96d7-41de-a3cc-0960ef9c31b8","Type":"ContainerStarted","Data":"92b27f1b26638f5ec435a3333ba39e849a39db954688f7492490450f14c3a5f2"} Oct 30 14:45:21.272680 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:21.272660 4982 scope.go:122] "RemoveContainer" containerID="589b40a0ac5f4d5776a9f50a2eead9590f8581b1e5c4e0e31e95201f2f14aff3" Oct 30 14:45:21.273324 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:21.273260 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3b95dd-29889525-mkkbq" event={"ID":"62a23269-9415-4a1f-b459-c6bb4a06cbae","Type":"ContainerDied","Data":"5926fac640544ad205bd99f106244a0e7d4e5d96a8c4463abfc380b1b238498f"} Oct 30 14:45:21.275135 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:21.273704 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3b95dd-29889525-mkkbq" Oct 30 14:45:21.303701 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:21.280393 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-33c209-29889525-fj5nd" event={"ID":"d524b057-6219-43cf-a11b-5048f36a8dfd","Type":"ContainerDied","Data":"11a42ddea3207c9a8f4051305b59c109a5e1d8c8b40f59c072a8f01657be6ea7"} Oct 30 14:45:21.303701 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:21.280474 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2927fd-29889525-srgtf"] Oct 30 14:45:21.303701 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:21.280496 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-33c209-29889525-fj5nd" Oct 30 14:45:21.303701 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:21.291085 4982 generic.go:356] "Generic (PLEG): container finished" podID="620719c8-c4d6-4f56-a843-24137f57fec0" containerID="de03e6525209e0e0e411dc40f9b88c81e9a8fb333c17ae82da8d82ae183d809d" exitCode=60 Oct 30 14:45:21.303701 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:21.291162 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2927fd-29889525-srgtf" event={"ID":"620719c8-c4d6-4f56-a843-24137f57fec0","Type":"ContainerDied","Data":"de03e6525209e0e0e411dc40f9b88c81e9a8fb333c17ae82da8d82ae183d809d"} Oct 30 14:45:21.303701 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:21.299249 4982 scope.go:122] "RemoveContainer" containerID="975f4df0bbee5b2a80c0f5f1625d2fe38f53e65dbc988604cd86af1e612c2ef3" Oct 30 14:45:21.324607 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:21.323960 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-1d6347-29889525-dqf8x"] Oct 30 14:45:21.332258 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:21.332240 4982 scope.go:122] "RemoveContainer" containerID="bbe6216bf9d9803fc9978fb8baf707841830078d1170c01c60142f5cb9b2fcb7" Oct 30 14:45:21.341851 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:21.341827 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-1d6347-29889525-dqf8x"] Oct 30 14:45:21.347362 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:21.347341 4982 scope.go:122] "RemoveContainer" containerID="89a2c29a4dd95a6ba99405575ce22f9478035787f1e7817a9fa33d55450b2595" Oct 30 14:45:21.384469 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:21.381476 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889525-pjgvt"] Oct 30 14:45:21.393959 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:21.393934 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889525-pjgvt"] Oct 30 14:45:21.442924 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:21.442880 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:45:21.449775 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:21.446122 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx" podStartSLOduration=2.446091638 podStartE2EDuration="2.446091638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:45:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:45:21.442392876 +0000 UTC m=+3391.186526917" watchObservedRunningTime="2026-10-30 14:45:21.446091638 +0000 UTC m=+3391.190225689" Oct 30 14:45:21.520422 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:21.520345 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz"] Oct 30 14:45:21.524307 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:21.524232 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz" podUID="1b30286e-1e19-4330-94e0-15decca18f76" containerName="istio-proxy" containerID="cri-o://b5838525fa8be1cbed04620f11312410d6454138abb1249cdb958417ecb25d78" gracePeriod=30 Oct 30 14:45:21.565507 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:21.565473 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-33c209-29889525-fj5nd"] Oct 30 14:45:21.575428 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:21.575403 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-33c209-29889525-fj5nd"] Oct 30 14:45:21.587434 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:21.587403 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889525-hz26s" Oct 30 14:45:21.590169 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:21.590144 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2927fd-29889525-srgtf" Oct 30 14:45:21.620303 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:21.620266 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3b95dd-29889525-mkkbq"] Oct 30 14:45:21.630424 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:21.629201 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e292b6c-6d65-41cd-a634-0a33f9cd86a4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e292b6c-6d65-41cd-a634-0a33f9cd86a4-maas-api-service-ca\") pod \"4e292b6c-6d65-41cd-a634-0a33f9cd86a4\" (UID: \"4e292b6c-6d65-41cd-a634-0a33f9cd86a4\") " Oct 30 14:45:21.630424 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:21.629420 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e292b6c-6d65-41cd-a634-0a33f9cd86a4-kube-api-access-7pkt4\" (UniqueName: \"kubernetes.io/projected/4e292b6c-6d65-41cd-a634-0a33f9cd86a4-kube-api-access-7pkt4\") pod \"4e292b6c-6d65-41cd-a634-0a33f9cd86a4\" (UID: \"4e292b6c-6d65-41cd-a634-0a33f9cd86a4\") " Oct 30 14:45:21.630424 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:21.629537 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/620719c8-c4d6-4f56-a843-24137f57fec0-kube-api-access-q4fwb\" (UniqueName: \"kubernetes.io/projected/620719c8-c4d6-4f56-a843-24137f57fec0-kube-api-access-q4fwb\") pod \"620719c8-c4d6-4f56-a843-24137f57fec0\" (UID: \"620719c8-c4d6-4f56-a843-24137f57fec0\") " Oct 30 14:45:21.630424 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:21.629711 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/620719c8-c4d6-4f56-a843-24137f57fec0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/620719c8-c4d6-4f56-a843-24137f57fec0-maas-api-service-ca\") pod \"620719c8-c4d6-4f56-a843-24137f57fec0\" (UID: \"620719c8-c4d6-4f56-a843-24137f57fec0\") " Oct 30 14:45:21.630768 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:21.630576 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3b95dd-29889525-mkkbq"] Oct 30 14:45:21.631057 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:21.631033 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e292b6c-6d65-41cd-a634-0a33f9cd86a4-maas-api-service-ca" pod "4e292b6c-6d65-41cd-a634-0a33f9cd86a4" (UID: "4e292b6c-6d65-41cd-a634-0a33f9cd86a4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:45:21.631153 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:21.631032 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/620719c8-c4d6-4f56-a843-24137f57fec0-maas-api-service-ca" pod "620719c8-c4d6-4f56-a843-24137f57fec0" (UID: "620719c8-c4d6-4f56-a843-24137f57fec0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:45:21.633307 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:21.633281 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/620719c8-c4d6-4f56-a843-24137f57fec0-kube-api-access-q4fwb" pod "620719c8-c4d6-4f56-a843-24137f57fec0" (UID: "620719c8-c4d6-4f56-a843-24137f57fec0"). InnerVolumeSpecName "kube-api-access-q4fwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:45:21.633526 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:21.633502 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e292b6c-6d65-41cd-a634-0a33f9cd86a4-kube-api-access-7pkt4" pod "4e292b6c-6d65-41cd-a634-0a33f9cd86a4" (UID: "4e292b6c-6d65-41cd-a634-0a33f9cd86a4"). InnerVolumeSpecName "kube-api-access-7pkt4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:45:21.732606 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:21.732568 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/620719c8-c4d6-4f56-a843-24137f57fec0-maas-api-service-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:21.732606 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:21.732606 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e292b6c-6d65-41cd-a634-0a33f9cd86a4-maas-api-service-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:21.732942 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:21.732621 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pkt4\" (UniqueName: \"kubernetes.io/projected/4e292b6c-6d65-41cd-a634-0a33f9cd86a4-kube-api-access-7pkt4\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:21.732942 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:21.732651 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4fwb\" (UniqueName: \"kubernetes.io/projected/620719c8-c4d6-4f56-a843-24137f57fec0-kube-api-access-q4fwb\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:22.103662 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:22.102969 4982 patch_prober.go:29] interesting pod/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" start-of-body= Oct 30 14:45:22.103662 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:22.103032 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx" podUID="19c9ad1b-96d7-41de-a3cc-0960ef9c31b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" Oct 30 14:45:22.103662 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:22.103294 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx" Oct 30 14:45:22.323221 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:22.323175 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889525-hz26s" Oct 30 14:45:22.323885 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:22.323544 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889525-hz26s" event={"ID":"4e292b6c-6d65-41cd-a634-0a33f9cd86a4","Type":"ContainerDied","Data":"40f96a6d6d4aeadc631454a6fe3e22c8bdf7bfffba7c30bf01538b66eab92686"} Oct 30 14:45:22.324130 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:22.324110 4982 scope.go:122] "RemoveContainer" containerID="ecca7f9c92e10572108a9dcb6fd74b69da3ca4c0796f3474f6507d0d8b20431e" Oct 30 14:45:22.330179 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:22.330149 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2927fd-29889525-srgtf" event={"ID":"620719c8-c4d6-4f56-a843-24137f57fec0","Type":"ContainerDied","Data":"0f1b5dc625d4ce4791dd4f5a4c0d26a1907a9c8d9c60cc5c5133464a8669ff92"} Oct 30 14:45:22.330301 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:22.330161 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2927fd-29889525-srgtf" Oct 30 14:45:22.349667 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:22.349617 4982 scope.go:122] "RemoveContainer" containerID="de03e6525209e0e0e411dc40f9b88c81e9a8fb333c17ae82da8d82ae183d809d" Oct 30 14:45:22.434292 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:22.434246 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a89d4da-0eef-472e-bf20-613e750dd598" path="/var/lib/kubelet/pods/1a89d4da-0eef-472e-bf20-613e750dd598/volumes" Oct 30 14:45:22.435062 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:22.435033 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ca2abf9-60f0-489a-99fa-70a5f4d44def" path="/var/lib/kubelet/pods/1ca2abf9-60f0-489a-99fa-70a5f4d44def/volumes" Oct 30 14:45:22.435740 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:22.435718 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62a23269-9415-4a1f-b459-c6bb4a06cbae" path="/var/lib/kubelet/pods/62a23269-9415-4a1f-b459-c6bb4a06cbae/volumes" Oct 30 14:45:22.436882 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:22.436858 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6dd7aa1-d123-4c72-889d-8a6371b7b483" path="/var/lib/kubelet/pods/c6dd7aa1-d123-4c72-889d-8a6371b7b483/volumes" Oct 30 14:45:22.437569 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:22.437528 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d524b057-6219-43cf-a11b-5048f36a8dfd" path="/var/lib/kubelet/pods/d524b057-6219-43cf-a11b-5048f36a8dfd/volumes" Oct 30 14:45:22.438202 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:22.438178 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df8648aa-12d5-4d90-88ad-e7a1a8333142" path="/var/lib/kubelet/pods/df8648aa-12d5-4d90-88ad-e7a1a8333142/volumes" Oct 30 14:45:22.439368 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:22.439348 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889525-hz26s"] Oct 30 14:45:22.439494 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:22.439485 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889525-hz26s"] Oct 30 14:45:22.456161 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:22.456139 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2927fd-29889525-srgtf"] Oct 30 14:45:22.462452 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:22.462432 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2927fd-29889525-srgtf"] Oct 30 14:45:23.103001 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:23.102964 4982 patch_prober.go:29] interesting pod/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" start-of-body= Oct 30 14:45:23.104318 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:23.104276 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx" podUID="19c9ad1b-96d7-41de-a3cc-0960ef9c31b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" Oct 30 14:45:23.928708 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:23.928669 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:45:24.105664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:24.104779 4982 patch_prober.go:29] interesting pod/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" start-of-body= Oct 30 14:45:24.105664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:24.104850 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx" podUID="19c9ad1b-96d7-41de-a3cc-0960ef9c31b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" Oct 30 14:45:24.424660 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:24.424218 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e292b6c-6d65-41cd-a634-0a33f9cd86a4" path="/var/lib/kubelet/pods/4e292b6c-6d65-41cd-a634-0a33f9cd86a4/volumes" Oct 30 14:45:24.425659 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:24.425075 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="620719c8-c4d6-4f56-a843-24137f57fec0" path="/var/lib/kubelet/pods/620719c8-c4d6-4f56-a843-24137f57fec0/volumes" Oct 30 14:45:24.947713 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:24.947673 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-1d6347-7778f9d655-545gn" podUID="d15be33a-a221-4357-8b19-609ba138d557" containerName="payload-pre-processing" containerID="cri-o://5a2145c7f3ea47c29161cb564a34d4be342b22ef2cc1abf97fa0c96662f1a6e1" gracePeriod=25 Oct 30 14:45:25.099392 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:25.098978 4982 patch_prober.go:29] interesting pod/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" start-of-body= Oct 30 14:45:25.099392 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:25.099035 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx" podUID="19c9ad1b-96d7-41de-a3cc-0960ef9c31b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" Oct 30 14:45:25.734515 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:25.734478 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:45:26.106996 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:26.106958 4982 patch_prober.go:29] interesting pod/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" start-of-body= Oct 30 14:45:26.107303 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:26.107274 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx" podUID="19c9ad1b-96d7-41de-a3cc-0960ef9c31b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" Oct 30 14:45:26.858252 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:26.858223 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz" Oct 30 14:45:26.931692 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:26.930452 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b30286e-1e19-4330-94e0-15decca18f76-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b30286e-1e19-4330-94e0-15decca18f76-workload-socket\") pod \"1b30286e-1e19-4330-94e0-15decca18f76\" (UID: \"1b30286e-1e19-4330-94e0-15decca18f76\") " Oct 30 14:45:26.931692 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:26.930810 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b30286e-1e19-4330-94e0-15decca18f76-workload-socket" pod "1b30286e-1e19-4330-94e0-15decca18f76" (UID: "1b30286e-1e19-4330-94e0-15decca18f76"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:45:26.933228 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:26.933202 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b30286e-1e19-4330-94e0-15decca18f76-kube-api-access-8zkhj\" (UniqueName: \"kubernetes.io/projected/1b30286e-1e19-4330-94e0-15decca18f76-kube-api-access-8zkhj\") pod \"1b30286e-1e19-4330-94e0-15decca18f76\" (UID: \"1b30286e-1e19-4330-94e0-15decca18f76\") " Oct 30 14:45:26.933649 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:26.933275 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b30286e-1e19-4330-94e0-15decca18f76-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b30286e-1e19-4330-94e0-15decca18f76-credential-socket\") pod \"1b30286e-1e19-4330-94e0-15decca18f76\" (UID: \"1b30286e-1e19-4330-94e0-15decca18f76\") " Oct 30 14:45:26.933649 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:26.933324 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b30286e-1e19-4330-94e0-15decca18f76-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b30286e-1e19-4330-94e0-15decca18f76-istiod-ca-cert\") pod \"1b30286e-1e19-4330-94e0-15decca18f76\" (UID: \"1b30286e-1e19-4330-94e0-15decca18f76\") " Oct 30 14:45:26.933649 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:26.933379 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b30286e-1e19-4330-94e0-15decca18f76-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b30286e-1e19-4330-94e0-15decca18f76-workload-certs\") pod \"1b30286e-1e19-4330-94e0-15decca18f76\" (UID: \"1b30286e-1e19-4330-94e0-15decca18f76\") " Oct 30 14:45:26.933649 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:26.933498 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b30286e-1e19-4330-94e0-15decca18f76-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b30286e-1e19-4330-94e0-15decca18f76-istio-envoy\") pod \"1b30286e-1e19-4330-94e0-15decca18f76\" (UID: \"1b30286e-1e19-4330-94e0-15decca18f76\") " Oct 30 14:45:26.933649 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:26.933533 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1b30286e-1e19-4330-94e0-15decca18f76-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b30286e-1e19-4330-94e0-15decca18f76-istio-podinfo\") pod \"1b30286e-1e19-4330-94e0-15decca18f76\" (UID: \"1b30286e-1e19-4330-94e0-15decca18f76\") " Oct 30 14:45:26.933649 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:26.933558 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b30286e-1e19-4330-94e0-15decca18f76-istio-token\" (UniqueName: \"kubernetes.io/projected/1b30286e-1e19-4330-94e0-15decca18f76-istio-token\") pod \"1b30286e-1e19-4330-94e0-15decca18f76\" (UID: \"1b30286e-1e19-4330-94e0-15decca18f76\") " Oct 30 14:45:26.933649 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:26.933595 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b30286e-1e19-4330-94e0-15decca18f76-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b30286e-1e19-4330-94e0-15decca18f76-istio-data\") pod \"1b30286e-1e19-4330-94e0-15decca18f76\" (UID: \"1b30286e-1e19-4330-94e0-15decca18f76\") " Oct 30 14:45:26.934220 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:26.934194 4982 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b30286e-1e19-4330-94e0-15decca18f76-workload-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:26.934438 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:26.934414 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b30286e-1e19-4330-94e0-15decca18f76-istio-data" pod "1b30286e-1e19-4330-94e0-15decca18f76" (UID: "1b30286e-1e19-4330-94e0-15decca18f76"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:45:26.935192 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:26.935165 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b30286e-1e19-4330-94e0-15decca18f76-workload-certs" pod "1b30286e-1e19-4330-94e0-15decca18f76" (UID: "1b30286e-1e19-4330-94e0-15decca18f76"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:45:26.935378 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:26.935356 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b30286e-1e19-4330-94e0-15decca18f76-credential-socket" pod "1b30286e-1e19-4330-94e0-15decca18f76" (UID: "1b30286e-1e19-4330-94e0-15decca18f76"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:45:26.935719 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:26.935679 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b30286e-1e19-4330-94e0-15decca18f76-istiod-ca-cert" pod "1b30286e-1e19-4330-94e0-15decca18f76" (UID: "1b30286e-1e19-4330-94e0-15decca18f76"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:45:26.940040 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:26.940013 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b30286e-1e19-4330-94e0-15decca18f76-istio-podinfo" pod "1b30286e-1e19-4330-94e0-15decca18f76" (UID: "1b30286e-1e19-4330-94e0-15decca18f76"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:45:26.940184 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:26.940037 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b30286e-1e19-4330-94e0-15decca18f76-istio-envoy" pod "1b30286e-1e19-4330-94e0-15decca18f76" (UID: "1b30286e-1e19-4330-94e0-15decca18f76"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:45:26.940184 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:26.940128 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b30286e-1e19-4330-94e0-15decca18f76-kube-api-access-8zkhj" pod "1b30286e-1e19-4330-94e0-15decca18f76" (UID: "1b30286e-1e19-4330-94e0-15decca18f76"). InnerVolumeSpecName "kube-api-access-8zkhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:45:26.942596 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:26.942562 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b30286e-1e19-4330-94e0-15decca18f76-istio-token" pod "1b30286e-1e19-4330-94e0-15decca18f76" (UID: "1b30286e-1e19-4330-94e0-15decca18f76"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:45:27.038673 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:27.038602 4982 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b30286e-1e19-4330-94e0-15decca18f76-istio-envoy\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:27.038673 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:27.038660 4982 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b30286e-1e19-4330-94e0-15decca18f76-istio-podinfo\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:27.038673 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:27.038676 4982 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b30286e-1e19-4330-94e0-15decca18f76-istio-token\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:27.039003 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:27.038688 4982 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b30286e-1e19-4330-94e0-15decca18f76-istio-data\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:27.039003 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:27.038700 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zkhj\" (UniqueName: \"kubernetes.io/projected/1b30286e-1e19-4330-94e0-15decca18f76-kube-api-access-8zkhj\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:27.039003 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:27.038730 4982 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b30286e-1e19-4330-94e0-15decca18f76-credential-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:27.039003 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:27.038781 4982 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b30286e-1e19-4330-94e0-15decca18f76-istiod-ca-cert\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:27.039003 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:27.038796 4982 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b30286e-1e19-4330-94e0-15decca18f76-workload-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:27.099950 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:27.099867 4982 patch_prober.go:29] interesting pod/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" start-of-body= Oct 30 14:45:27.099950 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:27.099923 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx" podUID="19c9ad1b-96d7-41de-a3cc-0960ef9c31b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" Oct 30 14:45:27.429910 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:27.429830 4982 generic.go:356] "Generic (PLEG): container finished" podID="1b30286e-1e19-4330-94e0-15decca18f76" containerID="b5838525fa8be1cbed04620f11312410d6454138abb1249cdb958417ecb25d78" exitCode=0 Oct 30 14:45:27.429910 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:27.429883 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz" Oct 30 14:45:27.430137 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:27.429904 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz" event={"ID":"1b30286e-1e19-4330-94e0-15decca18f76","Type":"ContainerDied","Data":"b5838525fa8be1cbed04620f11312410d6454138abb1249cdb958417ecb25d78"} Oct 30 14:45:27.430137 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:27.429956 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz" event={"ID":"1b30286e-1e19-4330-94e0-15decca18f76","Type":"ContainerDied","Data":"b0a994dcb95c6e597300597adce9d37864dec4a52ff481c9985107bbaa08589c"} Oct 30 14:45:27.430137 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:27.429977 4982 scope.go:122] "RemoveContainer" containerID="b5838525fa8be1cbed04620f11312410d6454138abb1249cdb958417ecb25d78" Oct 30 14:45:27.441995 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:27.441979 4982 scope.go:122] "RemoveContainer" containerID="b5838525fa8be1cbed04620f11312410d6454138abb1249cdb958417ecb25d78" Oct 30 14:45:27.442282 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:45:27.442258 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5838525fa8be1cbed04620f11312410d6454138abb1249cdb958417ecb25d78\": container with ID starting with b5838525fa8be1cbed04620f11312410d6454138abb1249cdb958417ecb25d78 not found: ID does not exist" containerID="b5838525fa8be1cbed04620f11312410d6454138abb1249cdb958417ecb25d78" Oct 30 14:45:27.442387 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:27.442285 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5838525fa8be1cbed04620f11312410d6454138abb1249cdb958417ecb25d78"} err="failed to get container status \"b5838525fa8be1cbed04620f11312410d6454138abb1249cdb958417ecb25d78\": rpc error: code = NotFound desc = could not find container \"b5838525fa8be1cbed04620f11312410d6454138abb1249cdb958417ecb25d78\": container with ID starting with b5838525fa8be1cbed04620f11312410d6454138abb1249cdb958417ecb25d78 not found: ID does not exist" Oct 30 14:45:27.491477 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:27.491435 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz"] Oct 30 14:45:27.504128 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:27.504091 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-7dxfz"] Oct 30 14:45:27.542745 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:27.542714 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:45:28.012252 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:45:28.012210 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:45:28.106436 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:28.106406 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx" Oct 30 14:45:28.106910 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:28.106889 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx" Oct 30 14:45:28.107671 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:28.107620 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx" Oct 30 14:45:28.401761 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:28.401730 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b30286e-1e19-4330-94e0-15decca18f76" path="/var/lib/kubelet/pods/1b30286e-1e19-4330-94e0-15decca18f76/volumes" Oct 30 14:45:29.401435 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:29.401403 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:45:31.268014 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:31.267963 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:45:32.106607 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:32.106568 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-1d6347-7778f9d655-545gn" podUID="d15be33a-a221-4357-8b19-609ba138d557" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:45:32.178598 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:32.178548 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:45:32.963949 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:32.963915 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr"] Oct 30 14:45:32.964365 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:32.964154 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr" podUID="9e1d6dd2-aa38-431e-aacc-7ba23fd2310d" containerName="istio-proxy" containerID="cri-o://8622eda68b8255c5719ae056f4e68765fb911b6188ba3b001f5b9b0d3f4eebed" gracePeriod=30 Oct 30 14:45:33.118408 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:33.118366 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4be240-58b8ddbdcb-n8p5d"] Oct 30 14:45:33.158628 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:33.158597 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4be240-84fd4f496c-7z9qv"] Oct 30 14:45:33.282198 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:33.282089 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:45:33.427119 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:33.427074 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:45:35.203424 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:35.203388 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:45:37.442811 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:37.442751 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:45:37.598167 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:37.598130 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:45:37.747339 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:37.747245 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:45:38.119064 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.119020 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-4be240-58b8ddbdcb-n8p5d" podUID="e2d18448-7965-46d9-bc11-dd64afe3bad7" containerName="payload-processing" containerID="cri-o://56e14dacc2dcc1dfc7ac5b261420d4a914e29551e4cde0ecf805e88fbf8135d6" gracePeriod=25 Oct 30 14:45:38.175211 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.166174 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4be240-84fd4f496c-7z9qv" podUID="bb284e7d-7925-45d6-892e-622f297e32be" containerName="payload-pre-processing" containerID="cri-o://55aa5085e8d64cb01b208e53e9c668a10276aa2a23ff41d89f2c10c181fabe57" gracePeriod=25 Oct 30 14:45:38.333741 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.333322 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr" Oct 30 14:45:38.439852 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.438759 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4be240-58b8ddbdcb-n8p5d" Oct 30 14:45:38.509100 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.508899 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e1d6dd2-aa38-431e-aacc-7ba23fd2310d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e1d6dd2-aa38-431e-aacc-7ba23fd2310d-workload-certs\") pod \"9e1d6dd2-aa38-431e-aacc-7ba23fd2310d\" (UID: \"9e1d6dd2-aa38-431e-aacc-7ba23fd2310d\") " Oct 30 14:45:38.509100 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.509026 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e1d6dd2-aa38-431e-aacc-7ba23fd2310d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e1d6dd2-aa38-431e-aacc-7ba23fd2310d-workload-socket\") pod \"9e1d6dd2-aa38-431e-aacc-7ba23fd2310d\" (UID: \"9e1d6dd2-aa38-431e-aacc-7ba23fd2310d\") " Oct 30 14:45:38.509100 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.509063 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e1d6dd2-aa38-431e-aacc-7ba23fd2310d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e1d6dd2-aa38-431e-aacc-7ba23fd2310d-credential-socket\") pod \"9e1d6dd2-aa38-431e-aacc-7ba23fd2310d\" (UID: \"9e1d6dd2-aa38-431e-aacc-7ba23fd2310d\") " Oct 30 14:45:38.509617 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.509145 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e1d6dd2-aa38-431e-aacc-7ba23fd2310d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e1d6dd2-aa38-431e-aacc-7ba23fd2310d-istio-envoy\") pod \"9e1d6dd2-aa38-431e-aacc-7ba23fd2310d\" (UID: \"9e1d6dd2-aa38-431e-aacc-7ba23fd2310d\") " Oct 30 14:45:38.509617 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.509190 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e1d6dd2-aa38-431e-aacc-7ba23fd2310d-istio-token\" (UniqueName: \"kubernetes.io/projected/9e1d6dd2-aa38-431e-aacc-7ba23fd2310d-istio-token\") pod \"9e1d6dd2-aa38-431e-aacc-7ba23fd2310d\" (UID: \"9e1d6dd2-aa38-431e-aacc-7ba23fd2310d\") " Oct 30 14:45:38.509617 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.509234 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e1d6dd2-aa38-431e-aacc-7ba23fd2310d-kube-api-access-zdmpk\" (UniqueName: \"kubernetes.io/projected/9e1d6dd2-aa38-431e-aacc-7ba23fd2310d-kube-api-access-zdmpk\") pod \"9e1d6dd2-aa38-431e-aacc-7ba23fd2310d\" (UID: \"9e1d6dd2-aa38-431e-aacc-7ba23fd2310d\") " Oct 30 14:45:38.509617 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.509309 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e1d6dd2-aa38-431e-aacc-7ba23fd2310d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e1d6dd2-aa38-431e-aacc-7ba23fd2310d-istiod-ca-cert\") pod \"9e1d6dd2-aa38-431e-aacc-7ba23fd2310d\" (UID: \"9e1d6dd2-aa38-431e-aacc-7ba23fd2310d\") " Oct 30 14:45:38.509617 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.509337 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e1d6dd2-aa38-431e-aacc-7ba23fd2310d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e1d6dd2-aa38-431e-aacc-7ba23fd2310d-istio-podinfo\") pod \"9e1d6dd2-aa38-431e-aacc-7ba23fd2310d\" (UID: \"9e1d6dd2-aa38-431e-aacc-7ba23fd2310d\") " Oct 30 14:45:38.509617 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.509395 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e1d6dd2-aa38-431e-aacc-7ba23fd2310d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e1d6dd2-aa38-431e-aacc-7ba23fd2310d-istio-data\") pod \"9e1d6dd2-aa38-431e-aacc-7ba23fd2310d\" (UID: \"9e1d6dd2-aa38-431e-aacc-7ba23fd2310d\") " Oct 30 14:45:38.510851 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.510650 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e1d6dd2-aa38-431e-aacc-7ba23fd2310d-workload-socket" pod "9e1d6dd2-aa38-431e-aacc-7ba23fd2310d" (UID: "9e1d6dd2-aa38-431e-aacc-7ba23fd2310d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:45:38.510950 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.510896 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e1d6dd2-aa38-431e-aacc-7ba23fd2310d-credential-socket" pod "9e1d6dd2-aa38-431e-aacc-7ba23fd2310d" (UID: "9e1d6dd2-aa38-431e-aacc-7ba23fd2310d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:45:38.511085 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.511057 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e1d6dd2-aa38-431e-aacc-7ba23fd2310d-istiod-ca-cert" pod "9e1d6dd2-aa38-431e-aacc-7ba23fd2310d" (UID: "9e1d6dd2-aa38-431e-aacc-7ba23fd2310d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:45:38.511284 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.511261 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e1d6dd2-aa38-431e-aacc-7ba23fd2310d-istio-data" pod "9e1d6dd2-aa38-431e-aacc-7ba23fd2310d" (UID: "9e1d6dd2-aa38-431e-aacc-7ba23fd2310d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:45:38.511355 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.511294 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e1d6dd2-aa38-431e-aacc-7ba23fd2310d-workload-certs" pod "9e1d6dd2-aa38-431e-aacc-7ba23fd2310d" (UID: "9e1d6dd2-aa38-431e-aacc-7ba23fd2310d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:45:38.514045 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.514023 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e1d6dd2-aa38-431e-aacc-7ba23fd2310d-istio-token" pod "9e1d6dd2-aa38-431e-aacc-7ba23fd2310d" (UID: "9e1d6dd2-aa38-431e-aacc-7ba23fd2310d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:45:38.514150 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.514102 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e1d6dd2-aa38-431e-aacc-7ba23fd2310d-istio-podinfo" pod "9e1d6dd2-aa38-431e-aacc-7ba23fd2310d" (UID: "9e1d6dd2-aa38-431e-aacc-7ba23fd2310d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:45:38.520027 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.520002 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e1d6dd2-aa38-431e-aacc-7ba23fd2310d-istio-envoy" pod "9e1d6dd2-aa38-431e-aacc-7ba23fd2310d" (UID: "9e1d6dd2-aa38-431e-aacc-7ba23fd2310d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:45:38.521241 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.521210 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e1d6dd2-aa38-431e-aacc-7ba23fd2310d-kube-api-access-zdmpk" pod "9e1d6dd2-aa38-431e-aacc-7ba23fd2310d" (UID: "9e1d6dd2-aa38-431e-aacc-7ba23fd2310d"). InnerVolumeSpecName "kube-api-access-zdmpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:45:38.522937 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.522917 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4be240-84fd4f496c-7z9qv" Oct 30 14:45:38.533101 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.533075 4982 generic.go:356] "Generic (PLEG): container finished" podID="9e1d6dd2-aa38-431e-aacc-7ba23fd2310d" containerID="8622eda68b8255c5719ae056f4e68765fb911b6188ba3b001f5b9b0d3f4eebed" exitCode=0 Oct 30 14:45:38.533189 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.533116 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr" Oct 30 14:45:38.533189 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.533156 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr" event={"ID":"9e1d6dd2-aa38-431e-aacc-7ba23fd2310d","Type":"ContainerDied","Data":"8622eda68b8255c5719ae056f4e68765fb911b6188ba3b001f5b9b0d3f4eebed"} Oct 30 14:45:38.533289 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.533213 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr" event={"ID":"9e1d6dd2-aa38-431e-aacc-7ba23fd2310d","Type":"ContainerDied","Data":"50e8bcef0e439121f4c6bcb70a5ed3c5491daad888a1808e94c3bbf631444202"} Oct 30 14:45:38.533289 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.533244 4982 scope.go:122] "RemoveContainer" containerID="8622eda68b8255c5719ae056f4e68765fb911b6188ba3b001f5b9b0d3f4eebed" Oct 30 14:45:38.534758 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.534735 4982 generic.go:356] "Generic (PLEG): container finished" podID="bb284e7d-7925-45d6-892e-622f297e32be" containerID="55aa5085e8d64cb01b208e53e9c668a10276aa2a23ff41d89f2c10c181fabe57" exitCode=0 Oct 30 14:45:38.534862 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.534821 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4be240-84fd4f496c-7z9qv" event={"ID":"bb284e7d-7925-45d6-892e-622f297e32be","Type":"ContainerDied","Data":"55aa5085e8d64cb01b208e53e9c668a10276aa2a23ff41d89f2c10c181fabe57"} Oct 30 14:45:38.534929 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.534869 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4be240-84fd4f496c-7z9qv" event={"ID":"bb284e7d-7925-45d6-892e-622f297e32be","Type":"ContainerDied","Data":"7902d29b33d06354568591335cfe50988e7e236d87e5b93d22007be780e4e55b"} Oct 30 14:45:38.535049 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.535030 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4be240-84fd4f496c-7z9qv" Oct 30 14:45:38.536944 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.536797 4982 generic.go:356] "Generic (PLEG): container finished" podID="e2d18448-7965-46d9-bc11-dd64afe3bad7" containerID="56e14dacc2dcc1dfc7ac5b261420d4a914e29551e4cde0ecf805e88fbf8135d6" exitCode=0 Oct 30 14:45:38.536944 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.536833 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4be240-58b8ddbdcb-n8p5d" event={"ID":"e2d18448-7965-46d9-bc11-dd64afe3bad7","Type":"ContainerDied","Data":"56e14dacc2dcc1dfc7ac5b261420d4a914e29551e4cde0ecf805e88fbf8135d6"} Oct 30 14:45:38.536944 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.536867 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4be240-58b8ddbdcb-n8p5d" event={"ID":"e2d18448-7965-46d9-bc11-dd64afe3bad7","Type":"ContainerDied","Data":"ece2d5b0bcb5ea9ccb5a876ab24669b2f050ecf9c59ffcd31582aa08f1c8185a"} Oct 30 14:45:38.536944 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.536932 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4be240-58b8ddbdcb-n8p5d" Oct 30 14:45:38.546593 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.546561 4982 scope.go:122] "RemoveContainer" containerID="8622eda68b8255c5719ae056f4e68765fb911b6188ba3b001f5b9b0d3f4eebed" Oct 30 14:45:38.547096 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:45:38.547068 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8622eda68b8255c5719ae056f4e68765fb911b6188ba3b001f5b9b0d3f4eebed\": container with ID starting with 8622eda68b8255c5719ae056f4e68765fb911b6188ba3b001f5b9b0d3f4eebed not found: ID does not exist" containerID="8622eda68b8255c5719ae056f4e68765fb911b6188ba3b001f5b9b0d3f4eebed" Oct 30 14:45:38.547199 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.547105 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8622eda68b8255c5719ae056f4e68765fb911b6188ba3b001f5b9b0d3f4eebed"} err="failed to get container status \"8622eda68b8255c5719ae056f4e68765fb911b6188ba3b001f5b9b0d3f4eebed\": rpc error: code = NotFound desc = could not find container \"8622eda68b8255c5719ae056f4e68765fb911b6188ba3b001f5b9b0d3f4eebed\": container with ID starting with 8622eda68b8255c5719ae056f4e68765fb911b6188ba3b001f5b9b0d3f4eebed not found: ID does not exist" Oct 30 14:45:38.547199 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.547126 4982 scope.go:122] "RemoveContainer" containerID="55aa5085e8d64cb01b208e53e9c668a10276aa2a23ff41d89f2c10c181fabe57" Oct 30 14:45:38.560511 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.560493 4982 scope.go:122] "RemoveContainer" containerID="55aa5085e8d64cb01b208e53e9c668a10276aa2a23ff41d89f2c10c181fabe57" Oct 30 14:45:38.563422 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:45:38.561515 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55aa5085e8d64cb01b208e53e9c668a10276aa2a23ff41d89f2c10c181fabe57\": container with ID starting with 55aa5085e8d64cb01b208e53e9c668a10276aa2a23ff41d89f2c10c181fabe57 not found: ID does not exist" containerID="55aa5085e8d64cb01b208e53e9c668a10276aa2a23ff41d89f2c10c181fabe57" Oct 30 14:45:38.563422 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.561549 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55aa5085e8d64cb01b208e53e9c668a10276aa2a23ff41d89f2c10c181fabe57"} err="failed to get container status \"55aa5085e8d64cb01b208e53e9c668a10276aa2a23ff41d89f2c10c181fabe57\": rpc error: code = NotFound desc = could not find container \"55aa5085e8d64cb01b208e53e9c668a10276aa2a23ff41d89f2c10c181fabe57\": container with ID starting with 55aa5085e8d64cb01b208e53e9c668a10276aa2a23ff41d89f2c10c181fabe57 not found: ID does not exist" Oct 30 14:45:38.563422 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.561570 4982 scope.go:122] "RemoveContainer" containerID="56e14dacc2dcc1dfc7ac5b261420d4a914e29551e4cde0ecf805e88fbf8135d6" Oct 30 14:45:38.578076 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.577780 4982 scope.go:122] "RemoveContainer" containerID="56e14dacc2dcc1dfc7ac5b261420d4a914e29551e4cde0ecf805e88fbf8135d6" Oct 30 14:45:38.581872 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:45:38.581837 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56e14dacc2dcc1dfc7ac5b261420d4a914e29551e4cde0ecf805e88fbf8135d6\": container with ID starting with 56e14dacc2dcc1dfc7ac5b261420d4a914e29551e4cde0ecf805e88fbf8135d6 not found: ID does not exist" containerID="56e14dacc2dcc1dfc7ac5b261420d4a914e29551e4cde0ecf805e88fbf8135d6" Oct 30 14:45:38.581973 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.581881 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56e14dacc2dcc1dfc7ac5b261420d4a914e29551e4cde0ecf805e88fbf8135d6"} err="failed to get container status \"56e14dacc2dcc1dfc7ac5b261420d4a914e29551e4cde0ecf805e88fbf8135d6\": rpc error: code = NotFound desc = could not find container \"56e14dacc2dcc1dfc7ac5b261420d4a914e29551e4cde0ecf805e88fbf8135d6\": container with ID starting with 56e14dacc2dcc1dfc7ac5b261420d4a914e29551e4cde0ecf805e88fbf8135d6 not found: ID does not exist" Oct 30 14:45:38.610598 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.610529 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2d18448-7965-46d9-bc11-dd64afe3bad7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2d18448-7965-46d9-bc11-dd64afe3bad7-plugins-config-volume\") pod \"e2d18448-7965-46d9-bc11-dd64afe3bad7\" (UID: \"e2d18448-7965-46d9-bc11-dd64afe3bad7\") " Oct 30 14:45:38.610833 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.610813 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2d18448-7965-46d9-bc11-dd64afe3bad7-kube-api-access-pq8bz\" (UniqueName: \"kubernetes.io/projected/e2d18448-7965-46d9-bc11-dd64afe3bad7-kube-api-access-pq8bz\") pod \"e2d18448-7965-46d9-bc11-dd64afe3bad7\" (UID: \"e2d18448-7965-46d9-bc11-dd64afe3bad7\") " Oct 30 14:45:38.611057 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.611031 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2d18448-7965-46d9-bc11-dd64afe3bad7-plugins-config-volume" pod "e2d18448-7965-46d9-bc11-dd64afe3bad7" (UID: "e2d18448-7965-46d9-bc11-dd64afe3bad7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:45:38.611695 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.611672 4982 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e1d6dd2-aa38-431e-aacc-7ba23fd2310d-workload-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:38.611811 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.611790 4982 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e1d6dd2-aa38-431e-aacc-7ba23fd2310d-credential-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:38.611883 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.611817 4982 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e1d6dd2-aa38-431e-aacc-7ba23fd2310d-istio-envoy\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:38.611883 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.611847 4982 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e1d6dd2-aa38-431e-aacc-7ba23fd2310d-istio-token\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:38.611883 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.611863 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdmpk\" (UniqueName: \"kubernetes.io/projected/9e1d6dd2-aa38-431e-aacc-7ba23fd2310d-kube-api-access-zdmpk\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:38.611883 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.611877 4982 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e1d6dd2-aa38-431e-aacc-7ba23fd2310d-istiod-ca-cert\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:38.612085 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.611891 4982 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e1d6dd2-aa38-431e-aacc-7ba23fd2310d-istio-podinfo\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:38.612085 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.611918 4982 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e1d6dd2-aa38-431e-aacc-7ba23fd2310d-istio-data\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:38.612085 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.611932 4982 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e1d6dd2-aa38-431e-aacc-7ba23fd2310d-workload-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:38.612085 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.611946 4982 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2d18448-7965-46d9-bc11-dd64afe3bad7-plugins-config-volume\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:38.614235 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.614211 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2d18448-7965-46d9-bc11-dd64afe3bad7-kube-api-access-pq8bz" pod "e2d18448-7965-46d9-bc11-dd64afe3bad7" (UID: "e2d18448-7965-46d9-bc11-dd64afe3bad7"). InnerVolumeSpecName "kube-api-access-pq8bz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:45:38.647711 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.647688 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr"] Oct 30 14:45:38.653730 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.653709 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e371c3-openshift-default-699697d895-p9kqr"] Oct 30 14:45:38.712564 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.712542 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb284e7d-7925-45d6-892e-622f297e32be-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb284e7d-7925-45d6-892e-622f297e32be-plugins-config-volume\") pod \"bb284e7d-7925-45d6-892e-622f297e32be\" (UID: \"bb284e7d-7925-45d6-892e-622f297e32be\") " Oct 30 14:45:38.712775 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.712757 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb284e7d-7925-45d6-892e-622f297e32be-kube-api-access-pvt5n\" (UniqueName: \"kubernetes.io/projected/bb284e7d-7925-45d6-892e-622f297e32be-kube-api-access-pvt5n\") pod \"bb284e7d-7925-45d6-892e-622f297e32be\" (UID: \"bb284e7d-7925-45d6-892e-622f297e32be\") " Oct 30 14:45:38.712961 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.712941 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb284e7d-7925-45d6-892e-622f297e32be-plugins-config-volume" pod "bb284e7d-7925-45d6-892e-622f297e32be" (UID: "bb284e7d-7925-45d6-892e-622f297e32be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:45:38.713297 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.713278 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pq8bz\" (UniqueName: \"kubernetes.io/projected/e2d18448-7965-46d9-bc11-dd64afe3bad7-kube-api-access-pq8bz\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:38.713377 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.713300 4982 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb284e7d-7925-45d6-892e-622f297e32be-plugins-config-volume\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:38.714887 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.714860 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb284e7d-7925-45d6-892e-622f297e32be-kube-api-access-pvt5n" pod "bb284e7d-7925-45d6-892e-622f297e32be" (UID: "bb284e7d-7925-45d6-892e-622f297e32be"). InnerVolumeSpecName "kube-api-access-pvt5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:45:38.814601 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.814564 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvt5n\" (UniqueName: \"kubernetes.io/projected/bb284e7d-7925-45d6-892e-622f297e32be-kube-api-access-pvt5n\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:38.896207 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.896134 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4be240-84fd4f496c-7z9qv"] Oct 30 14:45:38.918169 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.918134 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4be240-84fd4f496c-7z9qv"] Oct 30 14:45:38.959878 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.959829 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4be240-58b8ddbdcb-n8p5d"] Oct 30 14:45:38.970865 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:38.970776 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4be240-58b8ddbdcb-n8p5d"] Oct 30 14:45:39.660041 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:39.659995 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:45:40.403048 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:40.403015 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e1d6dd2-aa38-431e-aacc-7ba23fd2310d" path="/var/lib/kubelet/pods/9e1d6dd2-aa38-431e-aacc-7ba23fd2310d/volumes" Oct 30 14:45:40.403473 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:40.403454 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb284e7d-7925-45d6-892e-622f297e32be" path="/var/lib/kubelet/pods/bb284e7d-7925-45d6-892e-622f297e32be/volumes" Oct 30 14:45:40.403888 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:40.403875 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2d18448-7965-46d9-bc11-dd64afe3bad7" path="/var/lib/kubelet/pods/e2d18448-7965-46d9-bc11-dd64afe3bad7/volumes" Oct 30 14:45:40.462450 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:40.462422 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:45:40.554374 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:40.554341 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-90b406-548d97d98d-tjsf4"] Oct 30 14:45:40.587571 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:40.587543 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-90b406-98b94bb7-4wb24"] Oct 30 14:45:40.732489 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:40.732415 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:45:40.876552 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:40.876518 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:45:41.054183 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.054091 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:45:41.218109 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.218072 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:45:41.366574 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.366529 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:45:41.657105 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.657031 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d05e21-85db5c75d6-kg8b9"] Oct 30 14:45:41.657721 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.657703 4982 cpu_manager.go:414] "containerMap: removing container" podUID="1a89d4da-0eef-472e-bf20-613e750dd598" containerName="cleanup" Oct 30 14:45:41.657796 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.657723 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a89d4da-0eef-472e-bf20-613e750dd598" containerName="cleanup" Oct 30 14:45:41.657796 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.657739 4982 cpu_manager.go:414] "containerMap: removing container" podUID="e2d18448-7965-46d9-bc11-dd64afe3bad7" containerName="payload-processing" Oct 30 14:45:41.657796 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.657744 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2d18448-7965-46d9-bc11-dd64afe3bad7" containerName="payload-processing" Oct 30 14:45:41.657796 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.657756 4982 cpu_manager.go:414] "containerMap: removing container" podUID="1ca2abf9-60f0-489a-99fa-70a5f4d44def" containerName="cleanup" Oct 30 14:45:41.657796 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.657761 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ca2abf9-60f0-489a-99fa-70a5f4d44def" containerName="cleanup" Oct 30 14:45:41.657796 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.657770 4982 cpu_manager.go:414] "containerMap: removing container" podUID="d524b057-6219-43cf-a11b-5048f36a8dfd" containerName="cleanup" Oct 30 14:45:41.657796 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.657775 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d524b057-6219-43cf-a11b-5048f36a8dfd" containerName="cleanup" Oct 30 14:45:41.657796 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.657788 4982 cpu_manager.go:414] "containerMap: removing container" podUID="4e292b6c-6d65-41cd-a634-0a33f9cd86a4" containerName="cleanup" Oct 30 14:45:41.657796 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.657793 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e292b6c-6d65-41cd-a634-0a33f9cd86a4" containerName="cleanup" Oct 30 14:45:41.658088 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.657800 4982 cpu_manager.go:414] "containerMap: removing container" podUID="620719c8-c4d6-4f56-a843-24137f57fec0" containerName="cleanup" Oct 30 14:45:41.658088 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.657805 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="620719c8-c4d6-4f56-a843-24137f57fec0" containerName="cleanup" Oct 30 14:45:41.658088 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.657810 4982 cpu_manager.go:414] "containerMap: removing container" podUID="1ca2abf9-60f0-489a-99fa-70a5f4d44def" containerName="cleanup" Oct 30 14:45:41.658088 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.657814 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ca2abf9-60f0-489a-99fa-70a5f4d44def" containerName="cleanup" Oct 30 14:45:41.658088 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.657831 4982 cpu_manager.go:414] "containerMap: removing container" podUID="d524b057-6219-43cf-a11b-5048f36a8dfd" containerName="cleanup" Oct 30 14:45:41.658088 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.657835 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d524b057-6219-43cf-a11b-5048f36a8dfd" containerName="cleanup" Oct 30 14:45:41.658088 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.657841 4982 cpu_manager.go:414] "containerMap: removing container" podUID="9e1d6dd2-aa38-431e-aacc-7ba23fd2310d" containerName="istio-proxy" Oct 30 14:45:41.658088 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.657846 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e1d6dd2-aa38-431e-aacc-7ba23fd2310d" containerName="istio-proxy" Oct 30 14:45:41.658088 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.657853 4982 cpu_manager.go:414] "containerMap: removing container" podUID="f95c53ac-ceef-47a4-abe7-aed7fba44238" containerName="cleanup" Oct 30 14:45:41.658088 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.657857 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f95c53ac-ceef-47a4-abe7-aed7fba44238" containerName="cleanup" Oct 30 14:45:41.658088 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.657865 4982 cpu_manager.go:414] "containerMap: removing container" podUID="4e292b6c-6d65-41cd-a634-0a33f9cd86a4" containerName="cleanup" Oct 30 14:45:41.658088 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.657869 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e292b6c-6d65-41cd-a634-0a33f9cd86a4" containerName="cleanup" Oct 30 14:45:41.658088 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.657881 4982 cpu_manager.go:414] "containerMap: removing container" podUID="bb284e7d-7925-45d6-892e-622f297e32be" containerName="payload-pre-processing" Oct 30 14:45:41.658088 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.657885 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb284e7d-7925-45d6-892e-622f297e32be" containerName="payload-pre-processing" Oct 30 14:45:41.658088 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.657893 4982 cpu_manager.go:414] "containerMap: removing container" podUID="1ca2abf9-60f0-489a-99fa-70a5f4d44def" containerName="cleanup" Oct 30 14:45:41.658088 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.657897 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ca2abf9-60f0-489a-99fa-70a5f4d44def" containerName="cleanup" Oct 30 14:45:41.658088 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.657904 4982 cpu_manager.go:414] "containerMap: removing container" podUID="62a23269-9415-4a1f-b459-c6bb4a06cbae" containerName="cleanup" Oct 30 14:45:41.658088 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.657908 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62a23269-9415-4a1f-b459-c6bb4a06cbae" containerName="cleanup" Oct 30 14:45:41.658088 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.657917 4982 cpu_manager.go:414] "containerMap: removing container" podUID="62a23269-9415-4a1f-b459-c6bb4a06cbae" containerName="cleanup" Oct 30 14:45:41.658088 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.657922 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62a23269-9415-4a1f-b459-c6bb4a06cbae" containerName="cleanup" Oct 30 14:45:41.658088 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.657930 4982 cpu_manager.go:414] "containerMap: removing container" podUID="4e292b6c-6d65-41cd-a634-0a33f9cd86a4" containerName="cleanup" Oct 30 14:45:41.658088 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.657935 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e292b6c-6d65-41cd-a634-0a33f9cd86a4" containerName="cleanup" Oct 30 14:45:41.658088 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.657942 4982 cpu_manager.go:414] "containerMap: removing container" podUID="1a89d4da-0eef-472e-bf20-613e750dd598" containerName="cleanup" Oct 30 14:45:41.658088 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.657945 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a89d4da-0eef-472e-bf20-613e750dd598" containerName="cleanup" Oct 30 14:45:41.658088 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.657955 4982 cpu_manager.go:414] "containerMap: removing container" podUID="620719c8-c4d6-4f56-a843-24137f57fec0" containerName="cleanup" Oct 30 14:45:41.658088 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.657959 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="620719c8-c4d6-4f56-a843-24137f57fec0" containerName="cleanup" Oct 30 14:45:41.658088 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.657967 4982 cpu_manager.go:414] "containerMap: removing container" podUID="1b30286e-1e19-4330-94e0-15decca18f76" containerName="istio-proxy" Oct 30 14:45:41.658088 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.657972 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b30286e-1e19-4330-94e0-15decca18f76" containerName="istio-proxy" Oct 30 14:45:41.658088 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.658073 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="df8648aa-12d5-4d90-88ad-e7a1a8333142" containerName="cleanup" Oct 30 14:45:41.658088 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.658082 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ca2abf9-60f0-489a-99fa-70a5f4d44def" containerName="cleanup" Oct 30 14:45:41.658088 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.658088 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="d524b057-6219-43cf-a11b-5048f36a8dfd" containerName="cleanup" Oct 30 14:45:41.658088 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.658096 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e292b6c-6d65-41cd-a634-0a33f9cd86a4" containerName="cleanup" Oct 30 14:45:41.658088 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.658102 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="620719c8-c4d6-4f56-a843-24137f57fec0" containerName="cleanup" Oct 30 14:45:41.659071 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.658109 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e1d6dd2-aa38-431e-aacc-7ba23fd2310d" containerName="istio-proxy" Oct 30 14:45:41.659071 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.658117 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e292b6c-6d65-41cd-a634-0a33f9cd86a4" containerName="cleanup" Oct 30 14:45:41.659071 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.658123 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb284e7d-7925-45d6-892e-622f297e32be" containerName="payload-pre-processing" Oct 30 14:45:41.659071 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.658129 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ca2abf9-60f0-489a-99fa-70a5f4d44def" containerName="cleanup" Oct 30 14:45:41.659071 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.658134 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="62a23269-9415-4a1f-b459-c6bb4a06cbae" containerName="cleanup" Oct 30 14:45:41.659071 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.658139 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="381a4942-5434-499a-9d21-6e07dce9926b" containerName="cleanup" Oct 30 14:45:41.659071 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.658145 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a89d4da-0eef-472e-bf20-613e750dd598" containerName="cleanup" Oct 30 14:45:41.659071 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.658150 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="62a23269-9415-4a1f-b459-c6bb4a06cbae" containerName="cleanup" Oct 30 14:45:41.659071 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.658156 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="62a23269-9415-4a1f-b459-c6bb4a06cbae" containerName="cleanup" Oct 30 14:45:41.659071 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.658163 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e292b6c-6d65-41cd-a634-0a33f9cd86a4" containerName="cleanup" Oct 30 14:45:41.659071 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.658167 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a89d4da-0eef-472e-bf20-613e750dd598" containerName="cleanup" Oct 30 14:45:41.659071 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.658173 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="620719c8-c4d6-4f56-a843-24137f57fec0" containerName="cleanup" Oct 30 14:45:41.659071 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.658178 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b30286e-1e19-4330-94e0-15decca18f76" containerName="istio-proxy" Oct 30 14:45:41.659071 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.658183 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="d524b057-6219-43cf-a11b-5048f36a8dfd" containerName="cleanup" Oct 30 14:45:41.659071 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.658188 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a89d4da-0eef-472e-bf20-613e750dd598" containerName="cleanup" Oct 30 14:45:41.659071 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.658194 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2d18448-7965-46d9-bc11-dd64afe3bad7" containerName="payload-processing" Oct 30 14:45:41.663247 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.663229 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d05e21-85db5c75d6-kg8b9" Oct 30 14:45:41.680508 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.680488 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-d05e21-dockercfg-lb2ng\"" Oct 30 14:45:41.683582 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.683561 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-d05e21\"" Oct 30 14:45:41.702720 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.702688 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d05e21-85db5c75d6-kg8b9"] Oct 30 14:45:41.777990 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.777963 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02808c74-feb9-4fe4-8fe0-9f040dd3e8a1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d05e21-85db5c75d6-kg8b9\" (UID: \"02808c74-feb9-4fe4-8fe0-9f040dd3e8a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d05e21-85db5c75d6-kg8b9" Oct 30 14:45:41.778307 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.778081 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b9m4\" (UniqueName: \"kubernetes.io/projected/02808c74-feb9-4fe4-8fe0-9f040dd3e8a1-kube-api-access-6b9m4\") pod \"payload-processing-e2e-ipp-b-d05e21-85db5c75d6-kg8b9\" (UID: \"02808c74-feb9-4fe4-8fe0-9f040dd3e8a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d05e21-85db5c75d6-kg8b9" Oct 30 14:45:41.879009 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.878974 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02808c74-feb9-4fe4-8fe0-9f040dd3e8a1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d05e21-85db5c75d6-kg8b9\" (UID: \"02808c74-feb9-4fe4-8fe0-9f040dd3e8a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d05e21-85db5c75d6-kg8b9" Oct 30 14:45:41.879146 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.879062 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b9m4\" (UniqueName: \"kubernetes.io/projected/02808c74-feb9-4fe4-8fe0-9f040dd3e8a1-kube-api-access-6b9m4\") pod \"payload-processing-e2e-ipp-b-d05e21-85db5c75d6-kg8b9\" (UID: \"02808c74-feb9-4fe4-8fe0-9f040dd3e8a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d05e21-85db5c75d6-kg8b9" Oct 30 14:45:41.879686 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.879662 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02808c74-feb9-4fe4-8fe0-9f040dd3e8a1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d05e21-85db5c75d6-kg8b9\" (UID: \"02808c74-feb9-4fe4-8fe0-9f040dd3e8a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d05e21-85db5c75d6-kg8b9" Oct 30 14:45:41.909986 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.909925 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b9m4\" (UniqueName: \"kubernetes.io/projected/02808c74-feb9-4fe4-8fe0-9f040dd3e8a1-kube-api-access-6b9m4\") pod \"payload-processing-e2e-ipp-b-d05e21-85db5c75d6-kg8b9\" (UID: \"02808c74-feb9-4fe4-8fe0-9f040dd3e8a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d05e21-85db5c75d6-kg8b9" Oct 30 14:45:41.976415 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:41.976372 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d05e21-85db5c75d6-kg8b9" Oct 30 14:45:42.105918 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:42.105883 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-1d6347-7778f9d655-545gn" podUID="d15be33a-a221-4357-8b19-609ba138d557" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:45:42.292326 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:42.292285 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d05e21-85db5c75d6-kg8b9"] Oct 30 14:45:42.292914 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:45:42.292889 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02808c74_feb9_4fe4_8fe0_9f040dd3e8a1.slice/crio-d22669199c714624e37124797cc10d78e5e4295ab166ad1116ac24134b07c67e WatchSource:0}: Error finding container d22669199c714624e37124797cc10d78e5e4295ab166ad1116ac24134b07c67e: Status 404 returned error can't find the container with id d22669199c714624e37124797cc10d78e5e4295ab166ad1116ac24134b07c67e Oct 30 14:45:42.493727 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:42.493693 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:45:42.623056 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:42.623007 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d05e21-85db5c75d6-kg8b9" event={"ID":"02808c74-feb9-4fe4-8fe0-9f040dd3e8a1","Type":"ContainerStarted","Data":"f7712b3365b33a5b4e5c83f11dc549cafc56ef63c1c5f653719a5785624c4412"} Oct 30 14:45:42.623225 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:42.623072 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d05e21-85db5c75d6-kg8b9" event={"ID":"02808c74-feb9-4fe4-8fe0-9f040dd3e8a1","Type":"ContainerStarted","Data":"d22669199c714624e37124797cc10d78e5e4295ab166ad1116ac24134b07c67e"} Oct 30 14:45:42.623225 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:42.623101 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d05e21-85db5c75d6-kg8b9" Oct 30 14:45:42.669096 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:42.669039 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-d05e21-85db5c75d6-kg8b9" podStartSLOduration=1.6690254869999999 podStartE2EDuration="1.669025487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:45:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:45:42.667699013 +0000 UTC m=+3412.411833054" watchObservedRunningTime="2026-10-30 14:45:42.669025487 +0000 UTC m=+3412.413159527" Oct 30 14:45:44.666979 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:44.666940 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:45:45.272108 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:45.272072 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-1d6347-7778f9d655-545gn" Oct 30 14:45:45.323736 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:45.323703 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d15be33a-a221-4357-8b19-609ba138d557-kube-api-access-l4dtf\" (UniqueName: \"kubernetes.io/projected/d15be33a-a221-4357-8b19-609ba138d557-kube-api-access-l4dtf\") pod \"d15be33a-a221-4357-8b19-609ba138d557\" (UID: \"d15be33a-a221-4357-8b19-609ba138d557\") " Oct 30 14:45:45.323892 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:45.323822 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d15be33a-a221-4357-8b19-609ba138d557-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d15be33a-a221-4357-8b19-609ba138d557-plugins-config-volume\") pod \"d15be33a-a221-4357-8b19-609ba138d557\" (UID: \"d15be33a-a221-4357-8b19-609ba138d557\") " Oct 30 14:45:45.324208 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:45.324182 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d15be33a-a221-4357-8b19-609ba138d557-plugins-config-volume" pod "d15be33a-a221-4357-8b19-609ba138d557" (UID: "d15be33a-a221-4357-8b19-609ba138d557"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:45:45.324504 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:45.324485 4982 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d15be33a-a221-4357-8b19-609ba138d557-plugins-config-volume\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:45.326128 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:45.326101 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d15be33a-a221-4357-8b19-609ba138d557-kube-api-access-l4dtf" pod "d15be33a-a221-4357-8b19-609ba138d557" (UID: "d15be33a-a221-4357-8b19-609ba138d557"). InnerVolumeSpecName "kube-api-access-l4dtf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:45:45.426106 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:45.426067 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4dtf\" (UniqueName: \"kubernetes.io/projected/d15be33a-a221-4357-8b19-609ba138d557-kube-api-access-l4dtf\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:45.555483 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:45.555414 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-90b406-548d97d98d-tjsf4" podUID="1ac28c86-bcac-404f-93e5-20f03557e069" containerName="payload-processing" containerID="cri-o://6aa48cff6d0123a9a75ab6acca95c6bc5664e8833da3678de8713e3c25f8b74b" gracePeriod=25 Oct 30 14:45:45.588374 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:45.588341 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90b406-98b94bb7-4wb24" podUID="57dd3cfb-792c-49c3-90c6-c58c34f9b5c4" containerName="payload-pre-processing" containerID="cri-o://e02a140d414d4758240fc1675340cc50d573ce84a52c7ef50f90ddd6f2d0545b" gracePeriod=25 Oct 30 14:45:45.648615 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:45.648584 4982 generic.go:356] "Generic (PLEG): container finished" podID="d15be33a-a221-4357-8b19-609ba138d557" containerID="5a2145c7f3ea47c29161cb564a34d4be342b22ef2cc1abf97fa0c96662f1a6e1" exitCode=0 Oct 30 14:45:45.648742 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:45.648626 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-1d6347-7778f9d655-545gn" event={"ID":"d15be33a-a221-4357-8b19-609ba138d557","Type":"ContainerDied","Data":"5a2145c7f3ea47c29161cb564a34d4be342b22ef2cc1abf97fa0c96662f1a6e1"} Oct 30 14:45:45.648742 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:45.648677 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-1d6347-7778f9d655-545gn" Oct 30 14:45:45.648742 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:45.648715 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-1d6347-7778f9d655-545gn" event={"ID":"d15be33a-a221-4357-8b19-609ba138d557","Type":"ContainerDied","Data":"70fb63ebf4e31d8baf60d24b8aa0a4f5617f7888d08236b308538184528210ef"} Oct 30 14:45:45.648862 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:45.648745 4982 scope.go:122] "RemoveContainer" containerID="5a2145c7f3ea47c29161cb564a34d4be342b22ef2cc1abf97fa0c96662f1a6e1" Oct 30 14:45:45.703584 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:45.703558 4982 scope.go:122] "RemoveContainer" containerID="5a2145c7f3ea47c29161cb564a34d4be342b22ef2cc1abf97fa0c96662f1a6e1" Oct 30 14:45:45.704010 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:45:45.703980 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a2145c7f3ea47c29161cb564a34d4be342b22ef2cc1abf97fa0c96662f1a6e1\": container with ID starting with 5a2145c7f3ea47c29161cb564a34d4be342b22ef2cc1abf97fa0c96662f1a6e1 not found: ID does not exist" containerID="5a2145c7f3ea47c29161cb564a34d4be342b22ef2cc1abf97fa0c96662f1a6e1" Oct 30 14:45:45.704079 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:45.704025 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a2145c7f3ea47c29161cb564a34d4be342b22ef2cc1abf97fa0c96662f1a6e1"} err="failed to get container status \"5a2145c7f3ea47c29161cb564a34d4be342b22ef2cc1abf97fa0c96662f1a6e1\": rpc error: code = NotFound desc = could not find container \"5a2145c7f3ea47c29161cb564a34d4be342b22ef2cc1abf97fa0c96662f1a6e1\": container with ID starting with 5a2145c7f3ea47c29161cb564a34d4be342b22ef2cc1abf97fa0c96662f1a6e1 not found: ID does not exist" Oct 30 14:45:45.730757 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:45.729042 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-1d6347-7778f9d655-545gn"] Oct 30 14:45:45.732724 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:45.732611 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-1d6347-7778f9d655-545gn"] Oct 30 14:45:45.884333 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:45.884306 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-90b406-548d97d98d-tjsf4" Oct 30 14:45:45.910865 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:45.910839 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90b406-98b94bb7-4wb24" Oct 30 14:45:46.033883 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:46.033849 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57dd3cfb-792c-49c3-90c6-c58c34f9b5c4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57dd3cfb-792c-49c3-90c6-c58c34f9b5c4-plugins-config-volume\") pod \"57dd3cfb-792c-49c3-90c6-c58c34f9b5c4\" (UID: \"57dd3cfb-792c-49c3-90c6-c58c34f9b5c4\") " Oct 30 14:45:46.034058 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:46.033972 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ac28c86-bcac-404f-93e5-20f03557e069-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ac28c86-bcac-404f-93e5-20f03557e069-plugins-config-volume\") pod \"1ac28c86-bcac-404f-93e5-20f03557e069\" (UID: \"1ac28c86-bcac-404f-93e5-20f03557e069\") " Oct 30 14:45:46.034058 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:46.034028 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ac28c86-bcac-404f-93e5-20f03557e069-kube-api-access-zkg9s\" (UniqueName: \"kubernetes.io/projected/1ac28c86-bcac-404f-93e5-20f03557e069-kube-api-access-zkg9s\") pod \"1ac28c86-bcac-404f-93e5-20f03557e069\" (UID: \"1ac28c86-bcac-404f-93e5-20f03557e069\") " Oct 30 14:45:46.034218 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:46.034194 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57dd3cfb-792c-49c3-90c6-c58c34f9b5c4-plugins-config-volume" pod "57dd3cfb-792c-49c3-90c6-c58c34f9b5c4" (UID: "57dd3cfb-792c-49c3-90c6-c58c34f9b5c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:45:46.034289 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:46.034201 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57dd3cfb-792c-49c3-90c6-c58c34f9b5c4-kube-api-access-6h5md\" (UniqueName: \"kubernetes.io/projected/57dd3cfb-792c-49c3-90c6-c58c34f9b5c4-kube-api-access-6h5md\") pod \"57dd3cfb-792c-49c3-90c6-c58c34f9b5c4\" (UID: \"57dd3cfb-792c-49c3-90c6-c58c34f9b5c4\") " Oct 30 14:45:46.034354 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:46.034289 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ac28c86-bcac-404f-93e5-20f03557e069-plugins-config-volume" pod "1ac28c86-bcac-404f-93e5-20f03557e069" (UID: "1ac28c86-bcac-404f-93e5-20f03557e069"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:45:46.034704 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:46.034688 4982 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57dd3cfb-792c-49c3-90c6-c58c34f9b5c4-plugins-config-volume\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:46.034786 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:46.034706 4982 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ac28c86-bcac-404f-93e5-20f03557e069-plugins-config-volume\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:46.036280 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:46.036250 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ac28c86-bcac-404f-93e5-20f03557e069-kube-api-access-zkg9s" pod "1ac28c86-bcac-404f-93e5-20f03557e069" (UID: "1ac28c86-bcac-404f-93e5-20f03557e069"). InnerVolumeSpecName "kube-api-access-zkg9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:45:46.036376 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:46.036310 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57dd3cfb-792c-49c3-90c6-c58c34f9b5c4-kube-api-access-6h5md" pod "57dd3cfb-792c-49c3-90c6-c58c34f9b5c4" (UID: "57dd3cfb-792c-49c3-90c6-c58c34f9b5c4"). InnerVolumeSpecName "kube-api-access-6h5md". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:45:46.136209 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:46.136139 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h5md\" (UniqueName: \"kubernetes.io/projected/57dd3cfb-792c-49c3-90c6-c58c34f9b5c4-kube-api-access-6h5md\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:46.136209 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:46.136172 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkg9s\" (UniqueName: \"kubernetes.io/projected/1ac28c86-bcac-404f-93e5-20f03557e069-kube-api-access-zkg9s\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:45:46.400853 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:46.400782 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d15be33a-a221-4357-8b19-609ba138d557" path="/var/lib/kubelet/pods/d15be33a-a221-4357-8b19-609ba138d557/volumes" Oct 30 14:45:46.606821 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:46.606783 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:45:46.657821 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:46.657731 4982 generic.go:356] "Generic (PLEG): container finished" podID="1ac28c86-bcac-404f-93e5-20f03557e069" containerID="6aa48cff6d0123a9a75ab6acca95c6bc5664e8833da3678de8713e3c25f8b74b" exitCode=0 Oct 30 14:45:46.657821 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:46.657774 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-90b406-548d97d98d-tjsf4" Oct 30 14:45:46.658067 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:46.657840 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-90b406-548d97d98d-tjsf4" event={"ID":"1ac28c86-bcac-404f-93e5-20f03557e069","Type":"ContainerDied","Data":"6aa48cff6d0123a9a75ab6acca95c6bc5664e8833da3678de8713e3c25f8b74b"} Oct 30 14:45:46.658067 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:46.657893 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-90b406-548d97d98d-tjsf4" event={"ID":"1ac28c86-bcac-404f-93e5-20f03557e069","Type":"ContainerDied","Data":"430c1347b31e1590bd6d68a06d5a23a104c5d8de5050b583d54dd77bab245b86"} Oct 30 14:45:46.658067 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:46.657923 4982 scope.go:122] "RemoveContainer" containerID="6aa48cff6d0123a9a75ab6acca95c6bc5664e8833da3678de8713e3c25f8b74b" Oct 30 14:45:46.659329 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:46.659312 4982 generic.go:356] "Generic (PLEG): container finished" podID="57dd3cfb-792c-49c3-90c6-c58c34f9b5c4" containerID="e02a140d414d4758240fc1675340cc50d573ce84a52c7ef50f90ddd6f2d0545b" exitCode=0 Oct 30 14:45:46.659454 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:46.659357 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90b406-98b94bb7-4wb24" Oct 30 14:45:46.659454 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:46.659408 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90b406-98b94bb7-4wb24" event={"ID":"57dd3cfb-792c-49c3-90c6-c58c34f9b5c4","Type":"ContainerDied","Data":"e02a140d414d4758240fc1675340cc50d573ce84a52c7ef50f90ddd6f2d0545b"} Oct 30 14:45:46.659576 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:46.659457 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90b406-98b94bb7-4wb24" event={"ID":"57dd3cfb-792c-49c3-90c6-c58c34f9b5c4","Type":"ContainerDied","Data":"ad2047303164895dd1a4e5c34451a737505da973eabbb54ae95792ee47de2369"} Oct 30 14:45:46.669109 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:46.669090 4982 scope.go:122] "RemoveContainer" containerID="6aa48cff6d0123a9a75ab6acca95c6bc5664e8833da3678de8713e3c25f8b74b" Oct 30 14:45:46.669397 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:45:46.669374 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6aa48cff6d0123a9a75ab6acca95c6bc5664e8833da3678de8713e3c25f8b74b\": container with ID starting with 6aa48cff6d0123a9a75ab6acca95c6bc5664e8833da3678de8713e3c25f8b74b not found: ID does not exist" containerID="6aa48cff6d0123a9a75ab6acca95c6bc5664e8833da3678de8713e3c25f8b74b" Oct 30 14:45:46.669445 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:46.669406 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6aa48cff6d0123a9a75ab6acca95c6bc5664e8833da3678de8713e3c25f8b74b"} err="failed to get container status \"6aa48cff6d0123a9a75ab6acca95c6bc5664e8833da3678de8713e3c25f8b74b\": rpc error: code = NotFound desc = could not find container \"6aa48cff6d0123a9a75ab6acca95c6bc5664e8833da3678de8713e3c25f8b74b\": container with ID starting with 6aa48cff6d0123a9a75ab6acca95c6bc5664e8833da3678de8713e3c25f8b74b not found: ID does not exist" Oct 30 14:45:46.669445 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:46.669421 4982 scope.go:122] "RemoveContainer" containerID="e02a140d414d4758240fc1675340cc50d573ce84a52c7ef50f90ddd6f2d0545b" Oct 30 14:45:46.682327 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:46.682306 4982 scope.go:122] "RemoveContainer" containerID="e02a140d414d4758240fc1675340cc50d573ce84a52c7ef50f90ddd6f2d0545b" Oct 30 14:45:46.683431 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:45:46.683407 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e02a140d414d4758240fc1675340cc50d573ce84a52c7ef50f90ddd6f2d0545b\": container with ID starting with e02a140d414d4758240fc1675340cc50d573ce84a52c7ef50f90ddd6f2d0545b not found: ID does not exist" containerID="e02a140d414d4758240fc1675340cc50d573ce84a52c7ef50f90ddd6f2d0545b" Oct 30 14:45:46.683561 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:46.683433 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e02a140d414d4758240fc1675340cc50d573ce84a52c7ef50f90ddd6f2d0545b"} err="failed to get container status \"e02a140d414d4758240fc1675340cc50d573ce84a52c7ef50f90ddd6f2d0545b\": rpc error: code = NotFound desc = could not find container \"e02a140d414d4758240fc1675340cc50d573ce84a52c7ef50f90ddd6f2d0545b\": container with ID starting with e02a140d414d4758240fc1675340cc50d573ce84a52c7ef50f90ddd6f2d0545b not found: ID does not exist" Oct 30 14:45:46.717522 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:46.717494 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-90b406-98b94bb7-4wb24"] Oct 30 14:45:46.728787 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:46.728762 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-90b406-98b94bb7-4wb24"] Oct 30 14:45:46.767067 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:46.767040 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-90b406-548d97d98d-tjsf4"] Oct 30 14:45:46.773430 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:46.773404 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-90b406-548d97d98d-tjsf4"] Oct 30 14:45:47.448432 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:47.448401 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:45:47.634203 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:47.634162 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:45:47.782272 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:47.782189 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:45:48.403595 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:48.403554 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ac28c86-bcac-404f-93e5-20f03557e069" path="/var/lib/kubelet/pods/1ac28c86-bcac-404f-93e5-20f03557e069/volumes" Oct 30 14:45:48.404104 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:48.404081 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57dd3cfb-792c-49c3-90c6-c58c34f9b5c4" path="/var/lib/kubelet/pods/57dd3cfb-792c-49c3-90c6-c58c34f9b5c4/volumes" Oct 30 14:45:48.515566 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:48.515538 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:45:48.700579 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:48.700488 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:45:50.355290 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:50.355251 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:45:52.316331 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:52.316281 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:45:52.480841 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:52.480809 4982 scope.go:122] "RemoveContainer" containerID="5e958dd07e62c7240dd42891d79a3ec7c3f5dd019ff06c72311442a0bf9fbeb3" Oct 30 14:45:52.492531 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:52.492499 4982 scope.go:122] "RemoveContainer" containerID="762f1200fbcebeb18a5affddc0234cae8baaefedcb4b1dc5ef71bfa543b5cd25" Oct 30 14:45:52.503244 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:52.503218 4982 scope.go:122] "RemoveContainer" containerID="6d21eb8adac7bad5d0d7babbab2c00a7366aed244a2dd3fa658c571a32768d2b" Oct 30 14:45:52.513145 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:52.513121 4982 scope.go:122] "RemoveContainer" containerID="694e5a20aa97cc087183432e4154630dd6de03600ff35f771b06d868e7da05b0" Oct 30 14:45:52.522490 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:52.522457 4982 scope.go:122] "RemoveContainer" containerID="1e387171fe5e42298d004876a1876c337581db9adbc0a959018a19c7417cea89" Oct 30 14:45:52.532571 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:52.532552 4982 scope.go:122] "RemoveContainer" containerID="429fa549b9c353100f0eebee444097c33fed8ae919d8e01ffa62c49e1009d113" Oct 30 14:45:53.632592 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:53.632558 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d05e21-85db5c75d6-kg8b9" Oct 30 14:45:54.183920 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:54.181099 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:45:56.462912 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:56.462852 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:45:57.018746 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:57.018709 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:45:57.160502 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:57.160469 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:45:57.306615 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:57.306530 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:45:57.663929 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:45:57.663840 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:45:58.015994 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:45:58.015902 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:46:00.033034 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:00.032986 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:02.842106 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:02.842068 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:04.572564 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:04.572532 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:04.804112 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:04.804030 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:05.081246 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:05.081202 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:05.249024 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:05.248981 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:05.403129 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:05.403032 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:05.507895 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:05.507861 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2"] Oct 30 14:46:05.508353 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:05.508319 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2" podUID="30019687-440e-44fd-8fd8-0d2902927ca0" containerName="istio-proxy" containerID="cri-o://84fd2193c786eeb72126d7ad2ce8da8b2ca3e283eb48540c6bee55e1d4380633" gracePeriod=30 Oct 30 14:46:05.560615 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:05.560586 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:05.721840 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:05.721635 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:05.955470 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:05.955436 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:06.106160 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:06.106119 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:07.359610 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:07.359575 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:07.979500 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:07.979468 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:08.138383 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:08.138350 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:08.295703 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:08.295600 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:08.657742 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:08.657701 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:10.801797 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:10.801741 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2" Oct 30 14:46:10.880012 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:10.879979 4982 generic.go:356] "Generic (PLEG): container finished" podID="30019687-440e-44fd-8fd8-0d2902927ca0" containerID="84fd2193c786eeb72126d7ad2ce8da8b2ca3e283eb48540c6bee55e1d4380633" exitCode=0 Oct 30 14:46:10.880161 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:10.880037 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2" Oct 30 14:46:10.880161 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:10.880068 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2" event={"ID":"30019687-440e-44fd-8fd8-0d2902927ca0","Type":"ContainerDied","Data":"84fd2193c786eeb72126d7ad2ce8da8b2ca3e283eb48540c6bee55e1d4380633"} Oct 30 14:46:10.880161 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:10.880125 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2" event={"ID":"30019687-440e-44fd-8fd8-0d2902927ca0","Type":"ContainerDied","Data":"213953ad184cc54276d522de79b3ab600a266b45e281138ce27ced41ecf31f96"} Oct 30 14:46:10.880161 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:10.880150 4982 scope.go:122] "RemoveContainer" containerID="84fd2193c786eeb72126d7ad2ce8da8b2ca3e283eb48540c6bee55e1d4380633" Oct 30 14:46:10.893014 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:10.892951 4982 scope.go:122] "RemoveContainer" containerID="883719cef92123e240e8eaca0d0210e852634519c879e191779b6e08cd6a87f7" Oct 30 14:46:10.905003 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:10.904976 4982 scope.go:122] "RemoveContainer" containerID="84fd2193c786eeb72126d7ad2ce8da8b2ca3e283eb48540c6bee55e1d4380633" Oct 30 14:46:10.905293 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:46:10.905273 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84fd2193c786eeb72126d7ad2ce8da8b2ca3e283eb48540c6bee55e1d4380633\": container with ID starting with 84fd2193c786eeb72126d7ad2ce8da8b2ca3e283eb48540c6bee55e1d4380633 not found: ID does not exist" containerID="84fd2193c786eeb72126d7ad2ce8da8b2ca3e283eb48540c6bee55e1d4380633" Oct 30 14:46:10.905395 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:10.905302 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84fd2193c786eeb72126d7ad2ce8da8b2ca3e283eb48540c6bee55e1d4380633"} err="failed to get container status \"84fd2193c786eeb72126d7ad2ce8da8b2ca3e283eb48540c6bee55e1d4380633\": rpc error: code = NotFound desc = could not find container \"84fd2193c786eeb72126d7ad2ce8da8b2ca3e283eb48540c6bee55e1d4380633\": container with ID starting with 84fd2193c786eeb72126d7ad2ce8da8b2ca3e283eb48540c6bee55e1d4380633 not found: ID does not exist" Oct 30 14:46:10.905395 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:10.905319 4982 scope.go:122] "RemoveContainer" containerID="883719cef92123e240e8eaca0d0210e852634519c879e191779b6e08cd6a87f7" Oct 30 14:46:10.905598 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:10.905578 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30019687-440e-44fd-8fd8-0d2902927ca0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30019687-440e-44fd-8fd8-0d2902927ca0-istio-envoy\") pod \"30019687-440e-44fd-8fd8-0d2902927ca0\" (UID: \"30019687-440e-44fd-8fd8-0d2902927ca0\") " Oct 30 14:46:10.905693 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:46:10.905601 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"883719cef92123e240e8eaca0d0210e852634519c879e191779b6e08cd6a87f7\": container with ID starting with 883719cef92123e240e8eaca0d0210e852634519c879e191779b6e08cd6a87f7 not found: ID does not exist" containerID="883719cef92123e240e8eaca0d0210e852634519c879e191779b6e08cd6a87f7" Oct 30 14:46:10.905693 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:10.905619 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"883719cef92123e240e8eaca0d0210e852634519c879e191779b6e08cd6a87f7"} err="failed to get container status \"883719cef92123e240e8eaca0d0210e852634519c879e191779b6e08cd6a87f7\": rpc error: code = NotFound desc = could not find container \"883719cef92123e240e8eaca0d0210e852634519c879e191779b6e08cd6a87f7\": container with ID starting with 883719cef92123e240e8eaca0d0210e852634519c879e191779b6e08cd6a87f7 not found: ID does not exist" Oct 30 14:46:10.905693 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:10.905651 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30019687-440e-44fd-8fd8-0d2902927ca0-kube-api-access-wt2nm\" (UniqueName: \"kubernetes.io/projected/30019687-440e-44fd-8fd8-0d2902927ca0-kube-api-access-wt2nm\") pod \"30019687-440e-44fd-8fd8-0d2902927ca0\" (UID: \"30019687-440e-44fd-8fd8-0d2902927ca0\") " Oct 30 14:46:10.905693 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:10.905680 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30019687-440e-44fd-8fd8-0d2902927ca0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30019687-440e-44fd-8fd8-0d2902927ca0-istio-data\") pod \"30019687-440e-44fd-8fd8-0d2902927ca0\" (UID: \"30019687-440e-44fd-8fd8-0d2902927ca0\") " Oct 30 14:46:10.905936 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:10.905698 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30019687-440e-44fd-8fd8-0d2902927ca0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30019687-440e-44fd-8fd8-0d2902927ca0-istiod-ca-cert\") pod \"30019687-440e-44fd-8fd8-0d2902927ca0\" (UID: \"30019687-440e-44fd-8fd8-0d2902927ca0\") " Oct 30 14:46:10.905936 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:10.905775 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/30019687-440e-44fd-8fd8-0d2902927ca0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30019687-440e-44fd-8fd8-0d2902927ca0-istio-podinfo\") pod \"30019687-440e-44fd-8fd8-0d2902927ca0\" (UID: \"30019687-440e-44fd-8fd8-0d2902927ca0\") " Oct 30 14:46:10.905936 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:10.905823 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30019687-440e-44fd-8fd8-0d2902927ca0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30019687-440e-44fd-8fd8-0d2902927ca0-workload-certs\") pod \"30019687-440e-44fd-8fd8-0d2902927ca0\" (UID: \"30019687-440e-44fd-8fd8-0d2902927ca0\") " Oct 30 14:46:10.905936 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:10.905859 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30019687-440e-44fd-8fd8-0d2902927ca0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30019687-440e-44fd-8fd8-0d2902927ca0-workload-socket\") pod \"30019687-440e-44fd-8fd8-0d2902927ca0\" (UID: \"30019687-440e-44fd-8fd8-0d2902927ca0\") " Oct 30 14:46:10.905936 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:10.905886 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30019687-440e-44fd-8fd8-0d2902927ca0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30019687-440e-44fd-8fd8-0d2902927ca0-credential-socket\") pod \"30019687-440e-44fd-8fd8-0d2902927ca0\" (UID: \"30019687-440e-44fd-8fd8-0d2902927ca0\") " Oct 30 14:46:10.906205 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:10.905988 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30019687-440e-44fd-8fd8-0d2902927ca0-istio-token\" (UniqueName: \"kubernetes.io/projected/30019687-440e-44fd-8fd8-0d2902927ca0-istio-token\") pod \"30019687-440e-44fd-8fd8-0d2902927ca0\" (UID: \"30019687-440e-44fd-8fd8-0d2902927ca0\") " Oct 30 14:46:10.906205 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:10.906013 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30019687-440e-44fd-8fd8-0d2902927ca0-istio-data" pod "30019687-440e-44fd-8fd8-0d2902927ca0" (UID: "30019687-440e-44fd-8fd8-0d2902927ca0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:46:10.906205 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:10.906138 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30019687-440e-44fd-8fd8-0d2902927ca0-istiod-ca-cert" pod "30019687-440e-44fd-8fd8-0d2902927ca0" (UID: "30019687-440e-44fd-8fd8-0d2902927ca0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:46:10.906537 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:10.906510 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30019687-440e-44fd-8fd8-0d2902927ca0-workload-socket" pod "30019687-440e-44fd-8fd8-0d2902927ca0" (UID: "30019687-440e-44fd-8fd8-0d2902927ca0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:46:10.906685 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:10.906540 4982 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30019687-440e-44fd-8fd8-0d2902927ca0-istio-data\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:46:10.906685 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:10.906559 4982 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30019687-440e-44fd-8fd8-0d2902927ca0-istiod-ca-cert\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:46:10.906685 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:10.906576 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30019687-440e-44fd-8fd8-0d2902927ca0-workload-certs" pod "30019687-440e-44fd-8fd8-0d2902927ca0" (UID: "30019687-440e-44fd-8fd8-0d2902927ca0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:46:10.906890 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:10.906699 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30019687-440e-44fd-8fd8-0d2902927ca0-credential-socket" pod "30019687-440e-44fd-8fd8-0d2902927ca0" (UID: "30019687-440e-44fd-8fd8-0d2902927ca0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:46:10.907955 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:10.907931 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30019687-440e-44fd-8fd8-0d2902927ca0-kube-api-access-wt2nm" pod "30019687-440e-44fd-8fd8-0d2902927ca0" (UID: "30019687-440e-44fd-8fd8-0d2902927ca0"). InnerVolumeSpecName "kube-api-access-wt2nm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:46:10.911066 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:10.911040 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30019687-440e-44fd-8fd8-0d2902927ca0-istio-envoy" pod "30019687-440e-44fd-8fd8-0d2902927ca0" (UID: "30019687-440e-44fd-8fd8-0d2902927ca0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:46:10.911066 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:10.911040 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30019687-440e-44fd-8fd8-0d2902927ca0-istio-podinfo" pod "30019687-440e-44fd-8fd8-0d2902927ca0" (UID: "30019687-440e-44fd-8fd8-0d2902927ca0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:46:10.911195 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:10.911064 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30019687-440e-44fd-8fd8-0d2902927ca0-istio-token" pod "30019687-440e-44fd-8fd8-0d2902927ca0" (UID: "30019687-440e-44fd-8fd8-0d2902927ca0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:46:11.007926 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:11.007901 4982 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30019687-440e-44fd-8fd8-0d2902927ca0-istio-envoy\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:46:11.007926 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:11.007926 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wt2nm\" (UniqueName: \"kubernetes.io/projected/30019687-440e-44fd-8fd8-0d2902927ca0-kube-api-access-wt2nm\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:46:11.008082 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:11.007937 4982 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30019687-440e-44fd-8fd8-0d2902927ca0-istio-podinfo\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:46:11.008082 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:11.007946 4982 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30019687-440e-44fd-8fd8-0d2902927ca0-workload-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:46:11.008082 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:11.007954 4982 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30019687-440e-44fd-8fd8-0d2902927ca0-workload-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:46:11.008082 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:11.007963 4982 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30019687-440e-44fd-8fd8-0d2902927ca0-credential-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:46:11.008082 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:11.007972 4982 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30019687-440e-44fd-8fd8-0d2902927ca0-istio-token\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:46:11.075198 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:11.075167 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd"] Oct 30 14:46:11.150803 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:11.150772 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:11.252098 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:11.251970 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2"] Oct 30 14:46:11.254215 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:11.254170 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-1d6347-openshift-default-59d5cc8d8f-cjmm2"] Oct 30 14:46:11.315707 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:11.315669 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:12.405121 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:12.405082 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30019687-440e-44fd-8fd8-0d2902927ca0" path="/var/lib/kubelet/pods/30019687-440e-44fd-8fd8-0d2902927ca0/volumes" Oct 30 14:46:12.949038 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:12.948998 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:13.324186 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:13.324156 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:13.479867 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:13.479835 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:13.594974 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:13.594863 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5"] Oct 30 14:46:13.595698 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:13.595515 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5" podUID="7cbc37dd-65f1-40f7-8b5a-0ab581c99f86" containerName="istio-proxy" containerID="cri-o://f7ce5a0b43bb323ea8cf13a6ff78e3d2bc40c02006661b9b0dee4a64fd62d478" gracePeriod=30 Oct 30 14:46:13.656680 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:13.656652 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:15.040955 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:15.040914 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:16.298871 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:16.298831 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:16.463656 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:16.463603 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:16.615090 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:16.615046 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:16.906985 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:16.906881 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:17.070221 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:17.070178 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:17.269872 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:17.269779 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:18.903514 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:18.903437 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5" Oct 30 14:46:18.939679 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:18.939619 4982 generic.go:356] "Generic (PLEG): container finished" podID="7cbc37dd-65f1-40f7-8b5a-0ab581c99f86" containerID="f7ce5a0b43bb323ea8cf13a6ff78e3d2bc40c02006661b9b0dee4a64fd62d478" exitCode=0 Oct 30 14:46:18.939842 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:18.939674 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5" event={"ID":"7cbc37dd-65f1-40f7-8b5a-0ab581c99f86","Type":"ContainerDied","Data":"f7ce5a0b43bb323ea8cf13a6ff78e3d2bc40c02006661b9b0dee4a64fd62d478"} Oct 30 14:46:18.939842 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:18.939745 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5" event={"ID":"7cbc37dd-65f1-40f7-8b5a-0ab581c99f86","Type":"ContainerDied","Data":"4a21489fa65866af64c4afd46abd7699d4f221e0c27dd04931c47039bc5e581c"} Oct 30 14:46:18.939842 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:18.939686 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5" Oct 30 14:46:18.939842 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:18.939781 4982 scope.go:122] "RemoveContainer" containerID="f7ce5a0b43bb323ea8cf13a6ff78e3d2bc40c02006661b9b0dee4a64fd62d478" Oct 30 14:46:18.949872 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:18.949854 4982 scope.go:122] "RemoveContainer" containerID="42d6a9828625667b4d322e348b24ce5bae9dd7647f0ad389a0c84dcc9275d6ff" Oct 30 14:46:18.959171 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:18.959153 4982 scope.go:122] "RemoveContainer" containerID="f7ce5a0b43bb323ea8cf13a6ff78e3d2bc40c02006661b9b0dee4a64fd62d478" Oct 30 14:46:18.959429 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:46:18.959408 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7ce5a0b43bb323ea8cf13a6ff78e3d2bc40c02006661b9b0dee4a64fd62d478\": container with ID starting with f7ce5a0b43bb323ea8cf13a6ff78e3d2bc40c02006661b9b0dee4a64fd62d478 not found: ID does not exist" containerID="f7ce5a0b43bb323ea8cf13a6ff78e3d2bc40c02006661b9b0dee4a64fd62d478" Oct 30 14:46:18.959468 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:18.959444 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7ce5a0b43bb323ea8cf13a6ff78e3d2bc40c02006661b9b0dee4a64fd62d478"} err="failed to get container status \"f7ce5a0b43bb323ea8cf13a6ff78e3d2bc40c02006661b9b0dee4a64fd62d478\": rpc error: code = NotFound desc = could not find container \"f7ce5a0b43bb323ea8cf13a6ff78e3d2bc40c02006661b9b0dee4a64fd62d478\": container with ID starting with f7ce5a0b43bb323ea8cf13a6ff78e3d2bc40c02006661b9b0dee4a64fd62d478 not found: ID does not exist" Oct 30 14:46:18.959468 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:18.959465 4982 scope.go:122] "RemoveContainer" containerID="42d6a9828625667b4d322e348b24ce5bae9dd7647f0ad389a0c84dcc9275d6ff" Oct 30 14:46:18.959778 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:46:18.959758 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42d6a9828625667b4d322e348b24ce5bae9dd7647f0ad389a0c84dcc9275d6ff\": container with ID starting with 42d6a9828625667b4d322e348b24ce5bae9dd7647f0ad389a0c84dcc9275d6ff not found: ID does not exist" containerID="42d6a9828625667b4d322e348b24ce5bae9dd7647f0ad389a0c84dcc9275d6ff" Oct 30 14:46:18.959832 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:18.959783 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42d6a9828625667b4d322e348b24ce5bae9dd7647f0ad389a0c84dcc9275d6ff"} err="failed to get container status \"42d6a9828625667b4d322e348b24ce5bae9dd7647f0ad389a0c84dcc9275d6ff\": rpc error: code = NotFound desc = could not find container \"42d6a9828625667b4d322e348b24ce5bae9dd7647f0ad389a0c84dcc9275d6ff\": container with ID starting with 42d6a9828625667b4d322e348b24ce5bae9dd7647f0ad389a0c84dcc9275d6ff not found: ID does not exist" Oct 30 14:46:19.009678 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:19.009648 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cbc37dd-65f1-40f7-8b5a-0ab581c99f86-istio-token\" (UniqueName: \"kubernetes.io/projected/7cbc37dd-65f1-40f7-8b5a-0ab581c99f86-istio-token\") pod \"7cbc37dd-65f1-40f7-8b5a-0ab581c99f86\" (UID: \"7cbc37dd-65f1-40f7-8b5a-0ab581c99f86\") " Oct 30 14:46:19.009966 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:19.009944 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cbc37dd-65f1-40f7-8b5a-0ab581c99f86-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cbc37dd-65f1-40f7-8b5a-0ab581c99f86-workload-certs\") pod \"7cbc37dd-65f1-40f7-8b5a-0ab581c99f86\" (UID: \"7cbc37dd-65f1-40f7-8b5a-0ab581c99f86\") " Oct 30 14:46:19.010106 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:19.010085 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cbc37dd-65f1-40f7-8b5a-0ab581c99f86-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cbc37dd-65f1-40f7-8b5a-0ab581c99f86-workload-socket\") pod \"7cbc37dd-65f1-40f7-8b5a-0ab581c99f86\" (UID: \"7cbc37dd-65f1-40f7-8b5a-0ab581c99f86\") " Oct 30 14:46:19.010244 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:19.010224 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cbc37dd-65f1-40f7-8b5a-0ab581c99f86-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cbc37dd-65f1-40f7-8b5a-0ab581c99f86-istiod-ca-cert\") pod \"7cbc37dd-65f1-40f7-8b5a-0ab581c99f86\" (UID: \"7cbc37dd-65f1-40f7-8b5a-0ab581c99f86\") " Oct 30 14:46:19.010389 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:19.010372 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cbc37dd-65f1-40f7-8b5a-0ab581c99f86-kube-api-access-cgb42\" (UniqueName: \"kubernetes.io/projected/7cbc37dd-65f1-40f7-8b5a-0ab581c99f86-kube-api-access-cgb42\") pod \"7cbc37dd-65f1-40f7-8b5a-0ab581c99f86\" (UID: \"7cbc37dd-65f1-40f7-8b5a-0ab581c99f86\") " Oct 30 14:46:19.010585 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:19.010255 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cbc37dd-65f1-40f7-8b5a-0ab581c99f86-workload-certs" pod "7cbc37dd-65f1-40f7-8b5a-0ab581c99f86" (UID: "7cbc37dd-65f1-40f7-8b5a-0ab581c99f86"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:46:19.010689 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:19.010390 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cbc37dd-65f1-40f7-8b5a-0ab581c99f86-workload-socket" pod "7cbc37dd-65f1-40f7-8b5a-0ab581c99f86" (UID: "7cbc37dd-65f1-40f7-8b5a-0ab581c99f86"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:46:19.010689 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:19.010557 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cbc37dd-65f1-40f7-8b5a-0ab581c99f86-istiod-ca-cert" pod "7cbc37dd-65f1-40f7-8b5a-0ab581c99f86" (UID: "7cbc37dd-65f1-40f7-8b5a-0ab581c99f86"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:46:19.010689 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:19.010563 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cbc37dd-65f1-40f7-8b5a-0ab581c99f86-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cbc37dd-65f1-40f7-8b5a-0ab581c99f86-credential-socket\") pod \"7cbc37dd-65f1-40f7-8b5a-0ab581c99f86\" (UID: \"7cbc37dd-65f1-40f7-8b5a-0ab581c99f86\") " Oct 30 14:46:19.010877 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:19.010684 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7cbc37dd-65f1-40f7-8b5a-0ab581c99f86-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cbc37dd-65f1-40f7-8b5a-0ab581c99f86-istio-podinfo\") pod \"7cbc37dd-65f1-40f7-8b5a-0ab581c99f86\" (UID: \"7cbc37dd-65f1-40f7-8b5a-0ab581c99f86\") " Oct 30 14:46:19.010877 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:19.010722 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cbc37dd-65f1-40f7-8b5a-0ab581c99f86-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cbc37dd-65f1-40f7-8b5a-0ab581c99f86-istio-data\") pod \"7cbc37dd-65f1-40f7-8b5a-0ab581c99f86\" (UID: \"7cbc37dd-65f1-40f7-8b5a-0ab581c99f86\") " Oct 30 14:46:19.010877 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:19.010774 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cbc37dd-65f1-40f7-8b5a-0ab581c99f86-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cbc37dd-65f1-40f7-8b5a-0ab581c99f86-istio-envoy\") pod \"7cbc37dd-65f1-40f7-8b5a-0ab581c99f86\" (UID: \"7cbc37dd-65f1-40f7-8b5a-0ab581c99f86\") " Oct 30 14:46:19.011227 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:19.011207 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cbc37dd-65f1-40f7-8b5a-0ab581c99f86-credential-socket" pod "7cbc37dd-65f1-40f7-8b5a-0ab581c99f86" (UID: "7cbc37dd-65f1-40f7-8b5a-0ab581c99f86"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:46:19.011339 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:19.011221 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cbc37dd-65f1-40f7-8b5a-0ab581c99f86-istio-data" pod "7cbc37dd-65f1-40f7-8b5a-0ab581c99f86" (UID: "7cbc37dd-65f1-40f7-8b5a-0ab581c99f86"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:46:19.011699 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:19.011671 4982 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cbc37dd-65f1-40f7-8b5a-0ab581c99f86-workload-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:46:19.011823 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:19.011809 4982 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cbc37dd-65f1-40f7-8b5a-0ab581c99f86-workload-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:46:19.011924 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:19.011912 4982 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cbc37dd-65f1-40f7-8b5a-0ab581c99f86-istiod-ca-cert\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:46:19.012013 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:19.012001 4982 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cbc37dd-65f1-40f7-8b5a-0ab581c99f86-credential-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:46:19.012107 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:19.012093 4982 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cbc37dd-65f1-40f7-8b5a-0ab581c99f86-istio-data\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:46:19.014904 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:19.014871 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cbc37dd-65f1-40f7-8b5a-0ab581c99f86-istio-envoy" pod "7cbc37dd-65f1-40f7-8b5a-0ab581c99f86" (UID: "7cbc37dd-65f1-40f7-8b5a-0ab581c99f86"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:46:19.014995 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:19.014919 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cbc37dd-65f1-40f7-8b5a-0ab581c99f86-kube-api-access-cgb42" pod "7cbc37dd-65f1-40f7-8b5a-0ab581c99f86" (UID: "7cbc37dd-65f1-40f7-8b5a-0ab581c99f86"). InnerVolumeSpecName "kube-api-access-cgb42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:46:19.015297 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:19.015252 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7cbc37dd-65f1-40f7-8b5a-0ab581c99f86-istio-podinfo" pod "7cbc37dd-65f1-40f7-8b5a-0ab581c99f86" (UID: "7cbc37dd-65f1-40f7-8b5a-0ab581c99f86"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:46:19.015419 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:19.015398 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cbc37dd-65f1-40f7-8b5a-0ab581c99f86-istio-token" pod "7cbc37dd-65f1-40f7-8b5a-0ab581c99f86" (UID: "7cbc37dd-65f1-40f7-8b5a-0ab581c99f86"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:46:19.113772 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:19.113744 4982 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cbc37dd-65f1-40f7-8b5a-0ab581c99f86-istio-envoy\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:46:19.113772 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:19.113768 4982 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cbc37dd-65f1-40f7-8b5a-0ab581c99f86-istio-token\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:46:19.113912 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:19.113783 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgb42\" (UniqueName: \"kubernetes.io/projected/7cbc37dd-65f1-40f7-8b5a-0ab581c99f86-kube-api-access-cgb42\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:46:19.113912 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:19.113796 4982 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cbc37dd-65f1-40f7-8b5a-0ab581c99f86-istio-podinfo\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:46:19.292989 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:19.292961 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5"] Oct 30 14:46:19.304406 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:19.304376 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-a1854b-openshift-default-66ff8f6c6f-z8ff5"] Oct 30 14:46:19.489636 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:19.489604 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:20.297461 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:20.297424 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bcc710-7478845cd-x7lwp"] Oct 30 14:46:20.330279 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:20.330242 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bcc710-76656769cf-dmk7s"] Oct 30 14:46:20.409960 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:20.409929 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cbc37dd-65f1-40f7-8b5a-0ab581c99f86" path="/var/lib/kubelet/pods/7cbc37dd-65f1-40f7-8b5a-0ab581c99f86/volumes" Oct 30 14:46:20.479675 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:20.479630 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:20.644326 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:20.644282 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:20.808776 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:20.808747 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:22.272087 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:22.272050 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:24.224736 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:24.224695 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:25.298019 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:25.297961 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-bcc710-7478845cd-x7lwp" podUID="a422b119-172d-4748-bfad-7cae7a82021c" containerName="payload-processing" containerID="cri-o://c98fbff5cb97829a22fced2550d54a050dc0daeebef6103028ed6a8a353eb513" gracePeriod=25 Oct 30 14:46:25.331242 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:25.331208 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcc710-76656769cf-dmk7s" podUID="9139ba02-42d7-4ab6-895e-4bcf265f0449" containerName="payload-pre-processing" containerID="cri-o://869eed9b05c780139f0d307d217b8539acc9388092a1b1a26ad565e00712bc59" gracePeriod=25 Oct 30 14:46:25.657339 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:25.657318 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-bcc710-7478845cd-x7lwp" Oct 30 14:46:25.660596 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:25.660574 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcc710-76656769cf-dmk7s" Oct 30 14:46:25.801538 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:25.801486 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a422b119-172d-4748-bfad-7cae7a82021c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a422b119-172d-4748-bfad-7cae7a82021c-plugins-config-volume\") pod \"a422b119-172d-4748-bfad-7cae7a82021c\" (UID: \"a422b119-172d-4748-bfad-7cae7a82021c\") " Oct 30 14:46:25.801750 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:25.801560 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9139ba02-42d7-4ab6-895e-4bcf265f0449-kube-api-access-227v9\" (UniqueName: \"kubernetes.io/projected/9139ba02-42d7-4ab6-895e-4bcf265f0449-kube-api-access-227v9\") pod \"9139ba02-42d7-4ab6-895e-4bcf265f0449\" (UID: \"9139ba02-42d7-4ab6-895e-4bcf265f0449\") " Oct 30 14:46:25.801750 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:25.801674 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a422b119-172d-4748-bfad-7cae7a82021c-kube-api-access-b4j5x\" (UniqueName: \"kubernetes.io/projected/a422b119-172d-4748-bfad-7cae7a82021c-kube-api-access-b4j5x\") pod \"a422b119-172d-4748-bfad-7cae7a82021c\" (UID: \"a422b119-172d-4748-bfad-7cae7a82021c\") " Oct 30 14:46:25.801901 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:25.801788 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9139ba02-42d7-4ab6-895e-4bcf265f0449-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9139ba02-42d7-4ab6-895e-4bcf265f0449-plugins-config-volume\") pod \"9139ba02-42d7-4ab6-895e-4bcf265f0449\" (UID: \"9139ba02-42d7-4ab6-895e-4bcf265f0449\") " Oct 30 14:46:25.802061 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:25.802030 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a422b119-172d-4748-bfad-7cae7a82021c-plugins-config-volume" pod "a422b119-172d-4748-bfad-7cae7a82021c" (UID: "a422b119-172d-4748-bfad-7cae7a82021c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:46:25.802608 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:25.802580 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9139ba02-42d7-4ab6-895e-4bcf265f0449-plugins-config-volume" pod "9139ba02-42d7-4ab6-895e-4bcf265f0449" (UID: "9139ba02-42d7-4ab6-895e-4bcf265f0449"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:46:25.802762 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:25.802743 4982 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a422b119-172d-4748-bfad-7cae7a82021c-plugins-config-volume\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:46:25.804787 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:25.804762 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a422b119-172d-4748-bfad-7cae7a82021c-kube-api-access-b4j5x" pod "a422b119-172d-4748-bfad-7cae7a82021c" (UID: "a422b119-172d-4748-bfad-7cae7a82021c"). InnerVolumeSpecName "kube-api-access-b4j5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:46:25.805649 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:25.805621 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9139ba02-42d7-4ab6-895e-4bcf265f0449-kube-api-access-227v9" pod "9139ba02-42d7-4ab6-895e-4bcf265f0449" (UID: "9139ba02-42d7-4ab6-895e-4bcf265f0449"). InnerVolumeSpecName "kube-api-access-227v9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:46:25.904515 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:25.904455 4982 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9139ba02-42d7-4ab6-895e-4bcf265f0449-plugins-config-volume\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:46:25.904515 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:25.904485 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-227v9\" (UniqueName: \"kubernetes.io/projected/9139ba02-42d7-4ab6-895e-4bcf265f0449-kube-api-access-227v9\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:46:25.904515 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:25.904500 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4j5x\" (UniqueName: \"kubernetes.io/projected/a422b119-172d-4748-bfad-7cae7a82021c-kube-api-access-b4j5x\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:46:25.992568 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:25.992540 4982 generic.go:356] "Generic (PLEG): container finished" podID="a422b119-172d-4748-bfad-7cae7a82021c" containerID="c98fbff5cb97829a22fced2550d54a050dc0daeebef6103028ed6a8a353eb513" exitCode=0 Oct 30 14:46:25.992738 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:25.992613 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-bcc710-7478845cd-x7lwp" Oct 30 14:46:25.992738 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:25.992669 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-bcc710-7478845cd-x7lwp" event={"ID":"a422b119-172d-4748-bfad-7cae7a82021c","Type":"ContainerDied","Data":"c98fbff5cb97829a22fced2550d54a050dc0daeebef6103028ed6a8a353eb513"} Oct 30 14:46:25.992855 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:25.992740 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-bcc710-7478845cd-x7lwp" event={"ID":"a422b119-172d-4748-bfad-7cae7a82021c","Type":"ContainerDied","Data":"4051f393c06851c8277ee66d96a808662088f7e629920a6045a937e21cd30c5d"} Oct 30 14:46:25.992855 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:25.992770 4982 scope.go:122] "RemoveContainer" containerID="c98fbff5cb97829a22fced2550d54a050dc0daeebef6103028ed6a8a353eb513" Oct 30 14:46:26.008694 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:26.008403 4982 generic.go:356] "Generic (PLEG): container finished" podID="9139ba02-42d7-4ab6-895e-4bcf265f0449" containerID="869eed9b05c780139f0d307d217b8539acc9388092a1b1a26ad565e00712bc59" exitCode=0 Oct 30 14:46:26.008694 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:26.008525 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcc710-76656769cf-dmk7s" event={"ID":"9139ba02-42d7-4ab6-895e-4bcf265f0449","Type":"ContainerDied","Data":"869eed9b05c780139f0d307d217b8539acc9388092a1b1a26ad565e00712bc59"} Oct 30 14:46:26.008694 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:26.008608 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcc710-76656769cf-dmk7s" event={"ID":"9139ba02-42d7-4ab6-895e-4bcf265f0449","Type":"ContainerDied","Data":"e2fd2fc4d20159b2fd3ac1cb9c4be89fde9d8851db82ae31ddc1860872f167a3"} Oct 30 14:46:26.014579 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:26.009083 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcc710-76656769cf-dmk7s" Oct 30 14:46:26.023185 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:26.023064 4982 scope.go:122] "RemoveContainer" containerID="c98fbff5cb97829a22fced2550d54a050dc0daeebef6103028ed6a8a353eb513" Oct 30 14:46:26.023411 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:46:26.023386 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c98fbff5cb97829a22fced2550d54a050dc0daeebef6103028ed6a8a353eb513\": container with ID starting with c98fbff5cb97829a22fced2550d54a050dc0daeebef6103028ed6a8a353eb513 not found: ID does not exist" containerID="c98fbff5cb97829a22fced2550d54a050dc0daeebef6103028ed6a8a353eb513" Oct 30 14:46:26.023482 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:26.023421 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c98fbff5cb97829a22fced2550d54a050dc0daeebef6103028ed6a8a353eb513"} err="failed to get container status \"c98fbff5cb97829a22fced2550d54a050dc0daeebef6103028ed6a8a353eb513\": rpc error: code = NotFound desc = could not find container \"c98fbff5cb97829a22fced2550d54a050dc0daeebef6103028ed6a8a353eb513\": container with ID starting with c98fbff5cb97829a22fced2550d54a050dc0daeebef6103028ed6a8a353eb513 not found: ID does not exist" Oct 30 14:46:26.023482 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:26.023445 4982 scope.go:122] "RemoveContainer" containerID="869eed9b05c780139f0d307d217b8539acc9388092a1b1a26ad565e00712bc59" Oct 30 14:46:26.033802 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:26.033782 4982 scope.go:122] "RemoveContainer" containerID="869eed9b05c780139f0d307d217b8539acc9388092a1b1a26ad565e00712bc59" Oct 30 14:46:26.034049 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:46:26.034028 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"869eed9b05c780139f0d307d217b8539acc9388092a1b1a26ad565e00712bc59\": container with ID starting with 869eed9b05c780139f0d307d217b8539acc9388092a1b1a26ad565e00712bc59 not found: ID does not exist" containerID="869eed9b05c780139f0d307d217b8539acc9388092a1b1a26ad565e00712bc59" Oct 30 14:46:26.034126 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:26.034061 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"869eed9b05c780139f0d307d217b8539acc9388092a1b1a26ad565e00712bc59"} err="failed to get container status \"869eed9b05c780139f0d307d217b8539acc9388092a1b1a26ad565e00712bc59\": rpc error: code = NotFound desc = could not find container \"869eed9b05c780139f0d307d217b8539acc9388092a1b1a26ad565e00712bc59\": container with ID starting with 869eed9b05c780139f0d307d217b8539acc9388092a1b1a26ad565e00712bc59 not found: ID does not exist" Oct 30 14:46:26.051458 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:26.051426 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bcc710-7478845cd-x7lwp"] Oct 30 14:46:26.063004 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:26.062976 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:26.064527 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:26.064502 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bcc710-7478845cd-x7lwp"] Oct 30 14:46:26.096618 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:26.096582 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" podUID="15e7b7f9-2570-410a-a398-4c07c2e465ea" containerName="main" containerID="cri-o://170fae7c862522701e03c398e0da08f9e9ab01e2937569bbb2b821605438a838" gracePeriod=45 Oct 30 14:46:26.117748 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:26.117688 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bcc710-76656769cf-dmk7s"] Oct 30 14:46:26.122570 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:26.122540 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bcc710-76656769cf-dmk7s"] Oct 30 14:46:26.171456 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:26.171387 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" podUID="15e7b7f9-2570-410a-a398-4c07c2e465ea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": dial tcp 10.132.0.48:8000: connect: connection refused" Oct 30 14:46:26.373122 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:26.373099 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" Oct 30 14:46:26.403236 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:26.403200 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9139ba02-42d7-4ab6-895e-4bcf265f0449" path="/var/lib/kubelet/pods/9139ba02-42d7-4ab6-895e-4bcf265f0449/volumes" Oct 30 14:46:26.403828 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:26.403796 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a422b119-172d-4748-bfad-7cae7a82021c" path="/var/lib/kubelet/pods/a422b119-172d-4748-bfad-7cae7a82021c/volumes" Oct 30 14:46:26.517375 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:26.517296 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15e7b7f9-2570-410a-a398-4c07c2e465ea-home\" (UniqueName: \"kubernetes.io/empty-dir/15e7b7f9-2570-410a-a398-4c07c2e465ea-home\") pod \"15e7b7f9-2570-410a-a398-4c07c2e465ea\" (UID: \"15e7b7f9-2570-410a-a398-4c07c2e465ea\") " Oct 30 14:46:26.517540 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:26.517472 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15e7b7f9-2570-410a-a398-4c07c2e465ea-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15e7b7f9-2570-410a-a398-4c07c2e465ea-model-cache\") pod \"15e7b7f9-2570-410a-a398-4c07c2e465ea\" (UID: \"15e7b7f9-2570-410a-a398-4c07c2e465ea\") " Oct 30 14:46:26.517613 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:26.517538 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15e7b7f9-2570-410a-a398-4c07c2e465ea-home" pod "15e7b7f9-2570-410a-a398-4c07c2e465ea" (UID: "15e7b7f9-2570-410a-a398-4c07c2e465ea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:46:26.517613 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:26.517552 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15e7b7f9-2570-410a-a398-4c07c2e465ea-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15e7b7f9-2570-410a-a398-4c07c2e465ea-tmp-dir\") pod \"15e7b7f9-2570-410a-a398-4c07c2e465ea\" (UID: \"15e7b7f9-2570-410a-a398-4c07c2e465ea\") " Oct 30 14:46:26.517768 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:26.517613 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15e7b7f9-2570-410a-a398-4c07c2e465ea-tls-certs\" (UniqueName: \"kubernetes.io/secret/15e7b7f9-2570-410a-a398-4c07c2e465ea-tls-certs\") pod \"15e7b7f9-2570-410a-a398-4c07c2e465ea\" (UID: \"15e7b7f9-2570-410a-a398-4c07c2e465ea\") " Oct 30 14:46:26.517768 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:26.517730 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15e7b7f9-2570-410a-a398-4c07c2e465ea-dshm\" (UniqueName: \"kubernetes.io/empty-dir/15e7b7f9-2570-410a-a398-4c07c2e465ea-dshm\") pod \"15e7b7f9-2570-410a-a398-4c07c2e465ea\" (UID: \"15e7b7f9-2570-410a-a398-4c07c2e465ea\") " Oct 30 14:46:26.517882 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:26.517768 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15e7b7f9-2570-410a-a398-4c07c2e465ea-model-cache" pod "15e7b7f9-2570-410a-a398-4c07c2e465ea" (UID: "15e7b7f9-2570-410a-a398-4c07c2e465ea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:46:26.517882 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:26.517785 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15e7b7f9-2570-410a-a398-4c07c2e465ea-tmp-dir" pod "15e7b7f9-2570-410a-a398-4c07c2e465ea" (UID: "15e7b7f9-2570-410a-a398-4c07c2e465ea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:46:26.517882 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:26.517775 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15e7b7f9-2570-410a-a398-4c07c2e465ea-kube-api-access-nl8b2\" (UniqueName: \"kubernetes.io/projected/15e7b7f9-2570-410a-a398-4c07c2e465ea-kube-api-access-nl8b2\") pod \"15e7b7f9-2570-410a-a398-4c07c2e465ea\" (UID: \"15e7b7f9-2570-410a-a398-4c07c2e465ea\") " Oct 30 14:46:26.518694 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:26.518627 4982 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15e7b7f9-2570-410a-a398-4c07c2e465ea-model-cache\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:46:26.518833 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:26.518697 4982 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15e7b7f9-2570-410a-a398-4c07c2e465ea-tmp-dir\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:46:26.518833 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:26.518711 4982 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15e7b7f9-2570-410a-a398-4c07c2e465ea-home\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:46:26.520683 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:26.520370 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15e7b7f9-2570-410a-a398-4c07c2e465ea-dshm" pod "15e7b7f9-2570-410a-a398-4c07c2e465ea" (UID: "15e7b7f9-2570-410a-a398-4c07c2e465ea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:46:26.520683 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:26.520463 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15e7b7f9-2570-410a-a398-4c07c2e465ea-kube-api-access-nl8b2" pod "15e7b7f9-2570-410a-a398-4c07c2e465ea" (UID: "15e7b7f9-2570-410a-a398-4c07c2e465ea"). InnerVolumeSpecName "kube-api-access-nl8b2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:46:26.521175 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:26.521145 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15e7b7f9-2570-410a-a398-4c07c2e465ea-tls-certs" pod "15e7b7f9-2570-410a-a398-4c07c2e465ea" (UID: "15e7b7f9-2570-410a-a398-4c07c2e465ea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:46:26.620334 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:26.620287 4982 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15e7b7f9-2570-410a-a398-4c07c2e465ea-tls-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:46:26.620334 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:26.620334 4982 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15e7b7f9-2570-410a-a398-4c07c2e465ea-dshm\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:46:26.620334 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:26.620348 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nl8b2\" (UniqueName: \"kubernetes.io/projected/15e7b7f9-2570-410a-a398-4c07c2e465ea-kube-api-access-nl8b2\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:46:27.019799 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:27.019764 4982 generic.go:356] "Generic (PLEG): container finished" podID="15e7b7f9-2570-410a-a398-4c07c2e465ea" containerID="170fae7c862522701e03c398e0da08f9e9ab01e2937569bbb2b821605438a838" exitCode=0 Oct 30 14:46:27.019997 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:27.019816 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" Oct 30 14:46:27.019997 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:27.019847 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" event={"ID":"15e7b7f9-2570-410a-a398-4c07c2e465ea","Type":"ContainerDied","Data":"170fae7c862522701e03c398e0da08f9e9ab01e2937569bbb2b821605438a838"} Oct 30 14:46:27.019997 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:27.019899 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd" event={"ID":"15e7b7f9-2570-410a-a398-4c07c2e465ea","Type":"ContainerDied","Data":"9f60ca8e065d8af1e8dc1a4e9d2d4cfe2f1f1df4ba78c16e1539bc47c9a461fa"} Oct 30 14:46:27.019997 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:27.019922 4982 scope.go:122] "RemoveContainer" containerID="170fae7c862522701e03c398e0da08f9e9ab01e2937569bbb2b821605438a838" Oct 30 14:46:27.030798 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:27.030776 4982 scope.go:122] "RemoveContainer" containerID="170fae7c862522701e03c398e0da08f9e9ab01e2937569bbb2b821605438a838" Oct 30 14:46:27.031087 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:46:27.031066 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"170fae7c862522701e03c398e0da08f9e9ab01e2937569bbb2b821605438a838\": container with ID starting with 170fae7c862522701e03c398e0da08f9e9ab01e2937569bbb2b821605438a838 not found: ID does not exist" containerID="170fae7c862522701e03c398e0da08f9e9ab01e2937569bbb2b821605438a838" Oct 30 14:46:27.031174 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:27.031095 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"170fae7c862522701e03c398e0da08f9e9ab01e2937569bbb2b821605438a838"} err="failed to get container status \"170fae7c862522701e03c398e0da08f9e9ab01e2937569bbb2b821605438a838\": rpc error: code = NotFound desc = could not find container \"170fae7c862522701e03c398e0da08f9e9ab01e2937569bbb2b821605438a838\": container with ID starting with 170fae7c862522701e03c398e0da08f9e9ab01e2937569bbb2b821605438a838 not found: ID does not exist" Oct 30 14:46:27.077335 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:27.077304 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd"] Oct 30 14:46:27.091563 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:27.091539 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-a7bb0c/simulator-w5-a7bb0c-kserve-6b48f65659-p68vd"] Oct 30 14:46:27.579851 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:27.579818 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4be240-78755bcc95-r6pkn"] Oct 30 14:46:27.580338 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:27.580013 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4be240-78755bcc95-r6pkn" podUID="a8130ecc-2ff2-416b-99fb-d0f7ae285432" containerName="maas-api" containerID="cri-o://9bbb36009a89977191aa34637c4836f65d29c8271eda4acc47e3d2f6da8ccfe7" gracePeriod=30 Oct 30 14:46:27.684656 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:27.684612 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:27.951570 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:27.951542 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4be240-78755bcc95-r6pkn" Oct 30 14:46:28.034661 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:28.032897 4982 generic.go:356] "Generic (PLEG): container finished" podID="a8130ecc-2ff2-416b-99fb-d0f7ae285432" containerID="9bbb36009a89977191aa34637c4836f65d29c8271eda4acc47e3d2f6da8ccfe7" exitCode=0 Oct 30 14:46:28.034661 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:28.033092 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4be240-78755bcc95-r6pkn" event={"ID":"a8130ecc-2ff2-416b-99fb-d0f7ae285432","Type":"ContainerDied","Data":"9bbb36009a89977191aa34637c4836f65d29c8271eda4acc47e3d2f6da8ccfe7"} Oct 30 14:46:28.034661 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:28.033167 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4be240-78755bcc95-r6pkn" event={"ID":"a8130ecc-2ff2-416b-99fb-d0f7ae285432","Type":"ContainerDied","Data":"856145b507e5d401b6647a246254df8d4ca9f457247c62c55dd58cbd1cb252f1"} Oct 30 14:46:28.034661 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:28.033219 4982 scope.go:122] "RemoveContainer" containerID="9bbb36009a89977191aa34637c4836f65d29c8271eda4acc47e3d2f6da8ccfe7" Oct 30 14:46:28.034661 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:46:28.033469 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:46:28.036524 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:28.036507 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4be240-78755bcc95-r6pkn" Oct 30 14:46:28.081477 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:28.081436 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8130ecc-2ff2-416b-99fb-d0f7ae285432-kube-api-access-9g52k\" (UniqueName: \"kubernetes.io/projected/a8130ecc-2ff2-416b-99fb-d0f7ae285432-kube-api-access-9g52k\") pod \"a8130ecc-2ff2-416b-99fb-d0f7ae285432\" (UID: \"a8130ecc-2ff2-416b-99fb-d0f7ae285432\") " Oct 30 14:46:28.082451 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:28.082423 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8130ecc-2ff2-416b-99fb-d0f7ae285432-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8130ecc-2ff2-416b-99fb-d0f7ae285432-maas-api-tls\") pod \"a8130ecc-2ff2-416b-99fb-d0f7ae285432\" (UID: \"a8130ecc-2ff2-416b-99fb-d0f7ae285432\") " Oct 30 14:46:28.084180 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:28.084157 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8130ecc-2ff2-416b-99fb-d0f7ae285432-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8130ecc-2ff2-416b-99fb-d0f7ae285432-openshift-service-ca\") pod \"a8130ecc-2ff2-416b-99fb-d0f7ae285432\" (UID: \"a8130ecc-2ff2-416b-99fb-d0f7ae285432\") " Oct 30 14:46:28.084746 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:28.084723 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8130ecc-2ff2-416b-99fb-d0f7ae285432-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8130ecc-2ff2-416b-99fb-d0f7ae285432-metrics-certs\") pod \"a8130ecc-2ff2-416b-99fb-d0f7ae285432\" (UID: \"a8130ecc-2ff2-416b-99fb-d0f7ae285432\") " Oct 30 14:46:28.085733 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:28.084589 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8130ecc-2ff2-416b-99fb-d0f7ae285432-openshift-service-ca" pod "a8130ecc-2ff2-416b-99fb-d0f7ae285432" (UID: "a8130ecc-2ff2-416b-99fb-d0f7ae285432"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:46:28.086469 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:28.086445 4982 scope.go:122] "RemoveContainer" containerID="9bbb36009a89977191aa34637c4836f65d29c8271eda4acc47e3d2f6da8ccfe7" Oct 30 14:46:28.094671 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:46:28.087887 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bbb36009a89977191aa34637c4836f65d29c8271eda4acc47e3d2f6da8ccfe7\": container with ID starting with 9bbb36009a89977191aa34637c4836f65d29c8271eda4acc47e3d2f6da8ccfe7 not found: ID does not exist" containerID="9bbb36009a89977191aa34637c4836f65d29c8271eda4acc47e3d2f6da8ccfe7" Oct 30 14:46:28.094671 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:28.087926 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bbb36009a89977191aa34637c4836f65d29c8271eda4acc47e3d2f6da8ccfe7"} err="failed to get container status \"9bbb36009a89977191aa34637c4836f65d29c8271eda4acc47e3d2f6da8ccfe7\": rpc error: code = NotFound desc = could not find container \"9bbb36009a89977191aa34637c4836f65d29c8271eda4acc47e3d2f6da8ccfe7\": container with ID starting with 9bbb36009a89977191aa34637c4836f65d29c8271eda4acc47e3d2f6da8ccfe7 not found: ID does not exist" Oct 30 14:46:28.101814 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:28.101772 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8130ecc-2ff2-416b-99fb-d0f7ae285432-metrics-certs" pod "a8130ecc-2ff2-416b-99fb-d0f7ae285432" (UID: "a8130ecc-2ff2-416b-99fb-d0f7ae285432"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:46:28.110903 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:28.103068 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8130ecc-2ff2-416b-99fb-d0f7ae285432-kube-api-access-9g52k" pod "a8130ecc-2ff2-416b-99fb-d0f7ae285432" (UID: "a8130ecc-2ff2-416b-99fb-d0f7ae285432"). InnerVolumeSpecName "kube-api-access-9g52k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:46:28.110903 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:28.105775 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8130ecc-2ff2-416b-99fb-d0f7ae285432-maas-api-tls" pod "a8130ecc-2ff2-416b-99fb-d0f7ae285432" (UID: "a8130ecc-2ff2-416b-99fb-d0f7ae285432"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:46:28.186990 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:28.186909 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g52k\" (UniqueName: \"kubernetes.io/projected/a8130ecc-2ff2-416b-99fb-d0f7ae285432-kube-api-access-9g52k\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:46:28.186990 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:28.186941 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8130ecc-2ff2-416b-99fb-d0f7ae285432-maas-api-tls\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:46:28.186990 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:28.186955 4982 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8130ecc-2ff2-416b-99fb-d0f7ae285432-openshift-service-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:46:28.186990 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:28.186967 4982 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8130ecc-2ff2-416b-99fb-d0f7ae285432-metrics-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:46:28.261892 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:28.261848 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:28.395668 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:28.394539 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4be240-78755bcc95-r6pkn"] Oct 30 14:46:28.406008 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:28.405933 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:28.412332 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:28.412294 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15e7b7f9-2570-410a-a398-4c07c2e465ea" path="/var/lib/kubelet/pods/15e7b7f9-2570-410a-a398-4c07c2e465ea/volumes" Oct 30 14:46:28.412771 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:28.412751 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4be240-78755bcc95-r6pkn"] Oct 30 14:46:28.902437 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:28.902358 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:30.414886 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:30.414852 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8130ecc-2ff2-416b-99fb-d0f7ae285432" path="/var/lib/kubelet/pods/a8130ecc-2ff2-416b-99fb-d0f7ae285432/volumes" Oct 30 14:46:30.517944 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:30.517911 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:31.893203 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:31.893154 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:32.072701 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:32.072623 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:32.281715 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:32.280629 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-90b406-5dd68cb667-gfd5c"] Oct 30 14:46:32.281715 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:32.281574 4982 cpu_manager.go:414] "containerMap: removing container" podUID="57dd3cfb-792c-49c3-90c6-c58c34f9b5c4" containerName="payload-pre-processing" Oct 30 14:46:32.281715 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:32.281595 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57dd3cfb-792c-49c3-90c6-c58c34f9b5c4" containerName="payload-pre-processing" Oct 30 14:46:32.281715 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:32.281623 4982 cpu_manager.go:414] "containerMap: removing container" podUID="62a23269-9415-4a1f-b459-c6bb4a06cbae" containerName="cleanup" Oct 30 14:46:32.281715 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:32.281631 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62a23269-9415-4a1f-b459-c6bb4a06cbae" containerName="cleanup" Oct 30 14:46:32.281715 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:32.281662 4982 cpu_manager.go:414] "containerMap: removing container" podUID="d15be33a-a221-4357-8b19-609ba138d557" containerName="payload-pre-processing" Oct 30 14:46:32.281715 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:32.281671 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d15be33a-a221-4357-8b19-609ba138d557" containerName="payload-pre-processing" Oct 30 14:46:32.281715 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:32.281689 4982 cpu_manager.go:414] "containerMap: removing container" podUID="15e7b7f9-2570-410a-a398-4c07c2e465ea" containerName="main" Oct 30 14:46:32.281715 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:32.281697 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15e7b7f9-2570-410a-a398-4c07c2e465ea" containerName="main" Oct 30 14:46:32.281715 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:32.281707 4982 cpu_manager.go:414] "containerMap: removing container" podUID="7cbc37dd-65f1-40f7-8b5a-0ab581c99f86" containerName="istio-proxy" Oct 30 14:46:32.281715 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:32.281715 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cbc37dd-65f1-40f7-8b5a-0ab581c99f86" containerName="istio-proxy" Oct 30 14:46:32.282434 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:32.281728 4982 cpu_manager.go:414] "containerMap: removing container" podUID="620719c8-c4d6-4f56-a843-24137f57fec0" containerName="cleanup" Oct 30 14:46:32.282434 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:32.281735 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="620719c8-c4d6-4f56-a843-24137f57fec0" containerName="cleanup" Oct 30 14:46:32.282434 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:32.281751 4982 cpu_manager.go:414] "containerMap: removing container" podUID="a422b119-172d-4748-bfad-7cae7a82021c" containerName="payload-processing" Oct 30 14:46:32.282434 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:32.281759 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a422b119-172d-4748-bfad-7cae7a82021c" containerName="payload-processing" Oct 30 14:46:32.282434 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:32.281770 4982 cpu_manager.go:414] "containerMap: removing container" podUID="a8130ecc-2ff2-416b-99fb-d0f7ae285432" containerName="maas-api" Oct 30 14:46:32.282434 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:32.281777 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8130ecc-2ff2-416b-99fb-d0f7ae285432" containerName="maas-api" Oct 30 14:46:32.282434 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:32.281792 4982 cpu_manager.go:414] "containerMap: removing container" podUID="1a89d4da-0eef-472e-bf20-613e750dd598" containerName="cleanup" Oct 30 14:46:32.282434 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:32.281799 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a89d4da-0eef-472e-bf20-613e750dd598" containerName="cleanup" Oct 30 14:46:32.282434 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:32.281816 4982 cpu_manager.go:414] "containerMap: removing container" podUID="1ac28c86-bcac-404f-93e5-20f03557e069" containerName="payload-processing" Oct 30 14:46:32.282434 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:32.281824 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ac28c86-bcac-404f-93e5-20f03557e069" containerName="payload-processing" Oct 30 14:46:32.282434 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:32.281850 4982 cpu_manager.go:414] "containerMap: removing container" podUID="d524b057-6219-43cf-a11b-5048f36a8dfd" containerName="cleanup" Oct 30 14:46:32.282434 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:32.281858 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d524b057-6219-43cf-a11b-5048f36a8dfd" containerName="cleanup" Oct 30 14:46:32.282434 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:32.281874 4982 cpu_manager.go:414] "containerMap: removing container" podUID="30019687-440e-44fd-8fd8-0d2902927ca0" containerName="istio-proxy" Oct 30 14:46:32.282434 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:32.281881 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30019687-440e-44fd-8fd8-0d2902927ca0" containerName="istio-proxy" Oct 30 14:46:32.282434 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:32.281902 4982 cpu_manager.go:414] "containerMap: removing container" podUID="9139ba02-42d7-4ab6-895e-4bcf265f0449" containerName="payload-pre-processing" Oct 30 14:46:32.282434 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:32.281911 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9139ba02-42d7-4ab6-895e-4bcf265f0449" containerName="payload-pre-processing" Oct 30 14:46:32.282434 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:32.281923 4982 cpu_manager.go:414] "containerMap: removing container" podUID="30019687-440e-44fd-8fd8-0d2902927ca0" containerName="istio-proxy" Oct 30 14:46:32.282434 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:32.281931 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30019687-440e-44fd-8fd8-0d2902927ca0" containerName="istio-proxy" Oct 30 14:46:32.282434 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:32.282108 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ac28c86-bcac-404f-93e5-20f03557e069" containerName="payload-processing" Oct 30 14:46:32.282434 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:32.282123 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ca2abf9-60f0-489a-99fa-70a5f4d44def" containerName="cleanup" Oct 30 14:46:32.282434 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:32.282137 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="d524b057-6219-43cf-a11b-5048f36a8dfd" containerName="cleanup" Oct 30 14:46:32.282434 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:32.282146 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cbc37dd-65f1-40f7-8b5a-0ab581c99f86" containerName="istio-proxy" Oct 30 14:46:32.282434 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:32.282159 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="30019687-440e-44fd-8fd8-0d2902927ca0" containerName="istio-proxy" Oct 30 14:46:32.282434 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:32.282171 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="9139ba02-42d7-4ab6-895e-4bcf265f0449" containerName="payload-pre-processing" Oct 30 14:46:32.282434 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:32.282182 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="30019687-440e-44fd-8fd8-0d2902927ca0" containerName="istio-proxy" Oct 30 14:46:32.282434 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:32.282196 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="57dd3cfb-792c-49c3-90c6-c58c34f9b5c4" containerName="payload-pre-processing" Oct 30 14:46:32.282434 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:32.282207 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="d15be33a-a221-4357-8b19-609ba138d557" containerName="payload-pre-processing" Oct 30 14:46:32.282434 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:32.282219 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="15e7b7f9-2570-410a-a398-4c07c2e465ea" containerName="main" Oct 30 14:46:32.282434 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:32.282229 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="620719c8-c4d6-4f56-a843-24137f57fec0" containerName="cleanup" Oct 30 14:46:32.282434 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:32.282240 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="a422b119-172d-4748-bfad-7cae7a82021c" containerName="payload-processing" Oct 30 14:46:32.282434 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:32.282253 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8130ecc-2ff2-416b-99fb-d0f7ae285432" containerName="maas-api" Oct 30 14:46:32.306664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:32.303401 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90b406-5dd68cb667-gfd5c" Oct 30 14:46:32.345353 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:32.323547 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-90b406\"" Oct 30 14:46:32.345353 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:32.336215 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-90b406-dockercfg-dn85k\"" Oct 30 14:46:32.345353 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:32.337802 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-90b406-5dd68cb667-gfd5c"] Oct 30 14:46:32.346798 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:32.346607 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5424ec16-5768-428c-b826-fc0df029900b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-90b406-5dd68cb667-gfd5c\" (UID: \"5424ec16-5768-428c-b826-fc0df029900b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90b406-5dd68cb667-gfd5c" Oct 30 14:46:32.346798 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:32.346716 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv9hn\" (UniqueName: \"kubernetes.io/projected/5424ec16-5768-428c-b826-fc0df029900b-kube-api-access-cv9hn\") pod \"payload-pre-processing-e2e-ipp-b-90b406-5dd68cb667-gfd5c\" (UID: \"5424ec16-5768-428c-b826-fc0df029900b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90b406-5dd68cb667-gfd5c" Oct 30 14:46:32.448959 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:32.448918 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5424ec16-5768-428c-b826-fc0df029900b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-90b406-5dd68cb667-gfd5c\" (UID: \"5424ec16-5768-428c-b826-fc0df029900b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90b406-5dd68cb667-gfd5c" Oct 30 14:46:32.449148 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:32.449020 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv9hn\" (UniqueName: \"kubernetes.io/projected/5424ec16-5768-428c-b826-fc0df029900b-kube-api-access-cv9hn\") pod \"payload-pre-processing-e2e-ipp-b-90b406-5dd68cb667-gfd5c\" (UID: \"5424ec16-5768-428c-b826-fc0df029900b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90b406-5dd68cb667-gfd5c" Oct 30 14:46:32.449619 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:32.449597 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5424ec16-5768-428c-b826-fc0df029900b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-90b406-5dd68cb667-gfd5c\" (UID: \"5424ec16-5768-428c-b826-fc0df029900b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90b406-5dd68cb667-gfd5c" Oct 30 14:46:32.452833 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:32.452808 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-90b406-5f96f96db6-dwhwj"] Oct 30 14:46:32.453451 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:32.453435 4982 cpu_manager.go:414] "containerMap: removing container" podUID="7cbc37dd-65f1-40f7-8b5a-0ab581c99f86" containerName="istio-proxy" Oct 30 14:46:32.453505 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:32.453453 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cbc37dd-65f1-40f7-8b5a-0ab581c99f86" containerName="istio-proxy" Oct 30 14:46:32.453558 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:32.453549 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cbc37dd-65f1-40f7-8b5a-0ab581c99f86" containerName="istio-proxy" Oct 30 14:46:32.457000 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:32.456985 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-90b406-5f96f96db6-dwhwj" Oct 30 14:46:32.474889 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:32.474864 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv9hn\" (UniqueName: \"kubernetes.io/projected/5424ec16-5768-428c-b826-fc0df029900b-kube-api-access-cv9hn\") pod \"payload-pre-processing-e2e-ipp-b-90b406-5dd68cb667-gfd5c\" (UID: \"5424ec16-5768-428c-b826-fc0df029900b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90b406-5dd68cb667-gfd5c" Oct 30 14:46:32.489146 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:32.482914 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-90b406-5f96f96db6-dwhwj"] Oct 30 14:46:32.550520 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:32.550439 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffeee019-eb71-4340-87a4-8451e5c670c6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-90b406-5f96f96db6-dwhwj\" (UID: \"ffeee019-eb71-4340-87a4-8451e5c670c6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-90b406-5f96f96db6-dwhwj" Oct 30 14:46:32.550678 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:32.550576 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xv79\" (UniqueName: \"kubernetes.io/projected/ffeee019-eb71-4340-87a4-8451e5c670c6-kube-api-access-6xv79\") pod \"payload-processing-e2e-ipp-b-90b406-5f96f96db6-dwhwj\" (UID: \"ffeee019-eb71-4340-87a4-8451e5c670c6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-90b406-5f96f96db6-dwhwj" Oct 30 14:46:32.652238 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:32.652206 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffeee019-eb71-4340-87a4-8451e5c670c6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-90b406-5f96f96db6-dwhwj\" (UID: \"ffeee019-eb71-4340-87a4-8451e5c670c6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-90b406-5f96f96db6-dwhwj" Oct 30 14:46:32.652399 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:32.652327 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xv79\" (UniqueName: \"kubernetes.io/projected/ffeee019-eb71-4340-87a4-8451e5c670c6-kube-api-access-6xv79\") pod \"payload-processing-e2e-ipp-b-90b406-5f96f96db6-dwhwj\" (UID: \"ffeee019-eb71-4340-87a4-8451e5c670c6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-90b406-5f96f96db6-dwhwj" Oct 30 14:46:32.652920 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:32.652898 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffeee019-eb71-4340-87a4-8451e5c670c6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-90b406-5f96f96db6-dwhwj\" (UID: \"ffeee019-eb71-4340-87a4-8451e5c670c6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-90b406-5f96f96db6-dwhwj" Oct 30 14:46:32.661966 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:32.661941 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90b406-5dd68cb667-gfd5c" Oct 30 14:46:32.677984 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:32.677955 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xv79\" (UniqueName: \"kubernetes.io/projected/ffeee019-eb71-4340-87a4-8451e5c670c6-kube-api-access-6xv79\") pod \"payload-processing-e2e-ipp-b-90b406-5f96f96db6-dwhwj\" (UID: \"ffeee019-eb71-4340-87a4-8451e5c670c6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-90b406-5f96f96db6-dwhwj" Oct 30 14:46:32.770277 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:32.770240 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-90b406-5f96f96db6-dwhwj" Oct 30 14:46:32.968919 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:32.968891 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-90b406-5dd68cb667-gfd5c"] Oct 30 14:46:32.971433 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:46:32.971406 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5424ec16_5768_428c_b826_fc0df029900b.slice/crio-f33252183b943df4d42b8757e0090dc337a74939abf6b33c0a42245988845479 WatchSource:0}: Error finding container f33252183b943df4d42b8757e0090dc337a74939abf6b33c0a42245988845479: Status 404 returned error can't find the container with id f33252183b943df4d42b8757e0090dc337a74939abf6b33c0a42245988845479 Oct 30 14:46:33.072443 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:33.072412 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90b406-5dd68cb667-gfd5c" event={"ID":"5424ec16-5768-428c-b826-fc0df029900b","Type":"ContainerStarted","Data":"57eb031ffd8d278aaaec427e96ad9bff53d542f7fcaaba8e7a22cd6415fe8110"} Oct 30 14:46:33.072598 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:33.072465 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90b406-5dd68cb667-gfd5c" event={"ID":"5424ec16-5768-428c-b826-fc0df029900b","Type":"ContainerStarted","Data":"f33252183b943df4d42b8757e0090dc337a74939abf6b33c0a42245988845479"} Oct 30 14:46:33.072598 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:33.072519 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90b406-5dd68cb667-gfd5c" Oct 30 14:46:33.090686 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:33.090657 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-90b406-5f96f96db6-dwhwj"] Oct 30 14:46:33.091874 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:46:33.091847 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffeee019_eb71_4340_87a4_8451e5c670c6.slice/crio-7c20e8c5736bddbdc743f42ff2c540485a2d69d359ec9bf5047d79219cd858e1 WatchSource:0}: Error finding container 7c20e8c5736bddbdc743f42ff2c540485a2d69d359ec9bf5047d79219cd858e1: Status 404 returned error can't find the container with id 7c20e8c5736bddbdc743f42ff2c540485a2d69d359ec9bf5047d79219cd858e1 Oct 30 14:46:33.137660 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:33.137596 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90b406-5dd68cb667-gfd5c" podStartSLOduration=1.137582718 podStartE2EDuration="1.137582718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:46:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:46:33.137227008 +0000 UTC m=+3462.881361049" watchObservedRunningTime="2026-10-30 14:46:33.137582718 +0000 UTC m=+3462.881716758" Oct 30 14:46:33.328944 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:33.328903 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:34.079948 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:34.079913 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-90b406-5f96f96db6-dwhwj" event={"ID":"ffeee019-eb71-4340-87a4-8451e5c670c6","Type":"ContainerStarted","Data":"d44884feb547f6306b257870176c264162b7ec58824d761a9163c843f0a55f88"} Oct 30 14:46:34.080364 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:34.079978 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-90b406-5f96f96db6-dwhwj" event={"ID":"ffeee019-eb71-4340-87a4-8451e5c670c6","Type":"ContainerStarted","Data":"7c20e8c5736bddbdc743f42ff2c540485a2d69d359ec9bf5047d79219cd858e1"} Oct 30 14:46:34.080364 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:34.080094 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-90b406-5f96f96db6-dwhwj" Oct 30 14:46:34.126403 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:34.126351 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-90b406-5f96f96db6-dwhwj" podStartSLOduration=2.126337253 podStartE2EDuration="2.126337253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:46:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:46:34.124051674 +0000 UTC m=+3463.868185715" watchObservedRunningTime="2026-10-30 14:46:34.126337253 +0000 UTC m=+3463.870471293" Oct 30 14:46:35.108213 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:35.108167 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:35.822285 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:35.822246 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:35.958658 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:35.958599 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:36.174476 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:36.174362 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:37.228051 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:37.227893 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:39.842318 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:39.842276 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:40.073808 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:40.073769 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:41.722391 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:41.722333 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:43.690962 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:43.690930 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:44.081773 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:44.081744 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90b406-5dd68cb667-gfd5c" Oct 30 14:46:45.088825 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:45.088795 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-90b406-5f96f96db6-dwhwj" Oct 30 14:46:45.550510 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:45.550435 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:47.074629 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:47.074595 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-396d70-86964fb4d6-zmxfm"] Oct 30 14:46:47.122231 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:47.122194 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-396d70-56874f4975-w4bw5"] Oct 30 14:46:47.144183 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:47.144139 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:47.346097 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:47.346007 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:47.515985 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:47.515950 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:47.664999 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:47.664904 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:47.857101 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:47.857067 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:48.129560 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:48.129495 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:48.605115 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:48.605078 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:48.774104 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:48.774063 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:49.246142 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:49.246105 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:50.666982 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:50.666940 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-33c209-56ff9dc7df-lr8ft"] Oct 30 14:46:50.667529 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:50.667226 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33c209-56ff9dc7df-lr8ft" podUID="e6c9e188-d466-4ad8-b72c-f89717ff9f47" containerName="maas-api" containerID="cri-o://952d091925135db74f36260a933e533634d7613a5d59e50705d5dae69c9330c9" gracePeriod=30 Oct 30 14:46:50.830957 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:50.830912 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:50.934135 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:50.934046 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-33c209-5b7b659959-97sdt"] Oct 30 14:46:50.948027 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:50.947988 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c209-5b7b659959-97sdt" podUID="b4f5d962-a1e9-4f43-85c7-e4984032b6da" containerName="payload-pre-processing" containerID="cri-o://60c09878a089e13239e42a1668f9b98145aff1d02970eb8d8a4aa0fc54ebdc4a" gracePeriod=30 Oct 30 14:46:51.004777 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:51.004225 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:51.053949 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:51.053916 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33c209-56ff9dc7df-lr8ft" Oct 30 14:46:51.153306 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:51.153273 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:51.180326 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:51.180283 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6c9e188-d466-4ad8-b72c-f89717ff9f47-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6c9e188-d466-4ad8-b72c-f89717ff9f47-maas-api-tls\") pod \"e6c9e188-d466-4ad8-b72c-f89717ff9f47\" (UID: \"e6c9e188-d466-4ad8-b72c-f89717ff9f47\") " Oct 30 14:46:51.180466 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:51.180355 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6c9e188-d466-4ad8-b72c-f89717ff9f47-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6c9e188-d466-4ad8-b72c-f89717ff9f47-openshift-service-ca\") pod \"e6c9e188-d466-4ad8-b72c-f89717ff9f47\" (UID: \"e6c9e188-d466-4ad8-b72c-f89717ff9f47\") " Oct 30 14:46:51.180466 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:51.180431 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6c9e188-d466-4ad8-b72c-f89717ff9f47-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6c9e188-d466-4ad8-b72c-f89717ff9f47-metrics-certs\") pod \"e6c9e188-d466-4ad8-b72c-f89717ff9f47\" (UID: \"e6c9e188-d466-4ad8-b72c-f89717ff9f47\") " Oct 30 14:46:51.180593 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:51.180500 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6c9e188-d466-4ad8-b72c-f89717ff9f47-kube-api-access-klp2j\" (UniqueName: \"kubernetes.io/projected/e6c9e188-d466-4ad8-b72c-f89717ff9f47-kube-api-access-klp2j\") pod \"e6c9e188-d466-4ad8-b72c-f89717ff9f47\" (UID: \"e6c9e188-d466-4ad8-b72c-f89717ff9f47\") " Oct 30 14:46:51.181897 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:51.181866 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6c9e188-d466-4ad8-b72c-f89717ff9f47-openshift-service-ca" pod "e6c9e188-d466-4ad8-b72c-f89717ff9f47" (UID: "e6c9e188-d466-4ad8-b72c-f89717ff9f47"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:46:51.190242 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:51.190177 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6c9e188-d466-4ad8-b72c-f89717ff9f47-kube-api-access-klp2j" pod "e6c9e188-d466-4ad8-b72c-f89717ff9f47" (UID: "e6c9e188-d466-4ad8-b72c-f89717ff9f47"). InnerVolumeSpecName "kube-api-access-klp2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:46:51.198210 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:51.198180 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6c9e188-d466-4ad8-b72c-f89717ff9f47-metrics-certs" pod "e6c9e188-d466-4ad8-b72c-f89717ff9f47" (UID: "e6c9e188-d466-4ad8-b72c-f89717ff9f47"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:46:51.198332 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:51.198204 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6c9e188-d466-4ad8-b72c-f89717ff9f47-maas-api-tls" pod "e6c9e188-d466-4ad8-b72c-f89717ff9f47" (UID: "e6c9e188-d466-4ad8-b72c-f89717ff9f47"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:46:51.236410 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:51.236378 4982 generic.go:356] "Generic (PLEG): container finished" podID="b4f5d962-a1e9-4f43-85c7-e4984032b6da" containerID="60c09878a089e13239e42a1668f9b98145aff1d02970eb8d8a4aa0fc54ebdc4a" exitCode=0 Oct 30 14:46:51.236552 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:51.236457 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c209-5b7b659959-97sdt" event={"ID":"b4f5d962-a1e9-4f43-85c7-e4984032b6da","Type":"ContainerDied","Data":"60c09878a089e13239e42a1668f9b98145aff1d02970eb8d8a4aa0fc54ebdc4a"} Oct 30 14:46:51.237936 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:51.237915 4982 generic.go:356] "Generic (PLEG): container finished" podID="e6c9e188-d466-4ad8-b72c-f89717ff9f47" containerID="952d091925135db74f36260a933e533634d7613a5d59e50705d5dae69c9330c9" exitCode=0 Oct 30 14:46:51.238046 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:51.237956 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33c209-56ff9dc7df-lr8ft" event={"ID":"e6c9e188-d466-4ad8-b72c-f89717ff9f47","Type":"ContainerDied","Data":"952d091925135db74f36260a933e533634d7613a5d59e50705d5dae69c9330c9"} Oct 30 14:46:51.238046 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:51.237974 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33c209-56ff9dc7df-lr8ft" Oct 30 14:46:51.238046 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:51.237998 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33c209-56ff9dc7df-lr8ft" event={"ID":"e6c9e188-d466-4ad8-b72c-f89717ff9f47","Type":"ContainerDied","Data":"011f26e0fd514ef4f92bc8beaf4569075b562ada99227f1c203b65419b3cffa9"} Oct 30 14:46:51.238046 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:51.238020 4982 scope.go:122] "RemoveContainer" containerID="952d091925135db74f36260a933e533634d7613a5d59e50705d5dae69c9330c9" Oct 30 14:46:51.249603 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:51.249582 4982 scope.go:122] "RemoveContainer" containerID="952d091925135db74f36260a933e533634d7613a5d59e50705d5dae69c9330c9" Oct 30 14:46:51.250028 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:46:51.249930 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"952d091925135db74f36260a933e533634d7613a5d59e50705d5dae69c9330c9\": container with ID starting with 952d091925135db74f36260a933e533634d7613a5d59e50705d5dae69c9330c9 not found: ID does not exist" containerID="952d091925135db74f36260a933e533634d7613a5d59e50705d5dae69c9330c9" Oct 30 14:46:51.250028 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:51.249969 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"952d091925135db74f36260a933e533634d7613a5d59e50705d5dae69c9330c9"} err="failed to get container status \"952d091925135db74f36260a933e533634d7613a5d59e50705d5dae69c9330c9\": rpc error: code = NotFound desc = could not find container \"952d091925135db74f36260a933e533634d7613a5d59e50705d5dae69c9330c9\": container with ID starting with 952d091925135db74f36260a933e533634d7613a5d59e50705d5dae69c9330c9 not found: ID does not exist" Oct 30 14:46:51.297291 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:51.297171 4982 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6c9e188-d466-4ad8-b72c-f89717ff9f47-metrics-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:46:51.297291 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:51.297205 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klp2j\" (UniqueName: \"kubernetes.io/projected/e6c9e188-d466-4ad8-b72c-f89717ff9f47-kube-api-access-klp2j\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:46:51.297291 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:51.297224 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6c9e188-d466-4ad8-b72c-f89717ff9f47-maas-api-tls\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:46:51.297291 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:51.297239 4982 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6c9e188-d466-4ad8-b72c-f89717ff9f47-openshift-service-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:46:51.341694 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:51.341342 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-33c209-56ff9dc7df-lr8ft"] Oct 30 14:46:51.347774 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:51.347745 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-33c209-56ff9dc7df-lr8ft"] Oct 30 14:46:51.364661 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:51.358062 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c209-5b7b659959-97sdt" Oct 30 14:46:51.499409 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:51.499330 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4f5d962-a1e9-4f43-85c7-e4984032b6da-kube-api-access-srknv\" (UniqueName: \"kubernetes.io/projected/b4f5d962-a1e9-4f43-85c7-e4984032b6da-kube-api-access-srknv\") pod \"b4f5d962-a1e9-4f43-85c7-e4984032b6da\" (UID: \"b4f5d962-a1e9-4f43-85c7-e4984032b6da\") " Oct 30 14:46:51.499591 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:51.499433 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4f5d962-a1e9-4f43-85c7-e4984032b6da-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4f5d962-a1e9-4f43-85c7-e4984032b6da-plugins-config-volume\") pod \"b4f5d962-a1e9-4f43-85c7-e4984032b6da\" (UID: \"b4f5d962-a1e9-4f43-85c7-e4984032b6da\") " Oct 30 14:46:51.499817 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:51.499791 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4f5d962-a1e9-4f43-85c7-e4984032b6da-plugins-config-volume" pod "b4f5d962-a1e9-4f43-85c7-e4984032b6da" (UID: "b4f5d962-a1e9-4f43-85c7-e4984032b6da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:46:51.499985 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:51.499964 4982 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4f5d962-a1e9-4f43-85c7-e4984032b6da-plugins-config-volume\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:46:51.501505 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:51.501482 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4f5d962-a1e9-4f43-85c7-e4984032b6da-kube-api-access-srknv" pod "b4f5d962-a1e9-4f43-85c7-e4984032b6da" (UID: "b4f5d962-a1e9-4f43-85c7-e4984032b6da"). InnerVolumeSpecName "kube-api-access-srknv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:46:51.580061 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:51.580037 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:51.601392 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:51.601357 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srknv\" (UniqueName: \"kubernetes.io/projected/b4f5d962-a1e9-4f43-85c7-e4984032b6da-kube-api-access-srknv\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:46:51.731117 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:51.731071 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:51.885593 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:51.885553 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:52.087662 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:52.078699 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-396d70-86964fb4d6-zmxfm" podUID="973f4a7b-21e1-4e4d-8190-56032de67388" containerName="payload-processing" containerID="cri-o://dc8275981ecb1e7a37a746490d9d950946536e02c7dd9cfc2a25f60eeaec2c34" gracePeriod=25 Oct 30 14:46:52.123482 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:52.123441 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-396d70-56874f4975-w4bw5" podUID="b8343dab-b9ab-4453-bfa3-aada32b3ebf0" containerName="payload-pre-processing" containerID="cri-o://4b21cc04f93430c749d2d445ef0195bdb38c0db93fb5477ab5c175a4086ebfa9" gracePeriod=25 Oct 30 14:46:52.247519 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:52.247442 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c209-5b7b659959-97sdt" event={"ID":"b4f5d962-a1e9-4f43-85c7-e4984032b6da","Type":"ContainerDied","Data":"0213b8126cb6e0a6f42ebe68bf3405e39d4702fb31351ec667cdc199c4bee834"} Oct 30 14:46:52.247747 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:52.247449 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c209-5b7b659959-97sdt" Oct 30 14:46:52.247747 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:52.247536 4982 scope.go:122] "RemoveContainer" containerID="60c09878a089e13239e42a1668f9b98145aff1d02970eb8d8a4aa0fc54ebdc4a" Oct 30 14:46:52.256336 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:52.256288 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:52.314872 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:52.314842 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-33c209-5b7b659959-97sdt"] Oct 30 14:46:52.327822 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:52.327798 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-33c209-5b7b659959-97sdt"] Oct 30 14:46:52.400584 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:52.400552 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4f5d962-a1e9-4f43-85c7-e4984032b6da" path="/var/lib/kubelet/pods/b4f5d962-a1e9-4f43-85c7-e4984032b6da/volumes" Oct 30 14:46:52.401123 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:52.401100 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6c9e188-d466-4ad8-b72c-f89717ff9f47" path="/var/lib/kubelet/pods/e6c9e188-d466-4ad8-b72c-f89717ff9f47/volumes" Oct 30 14:46:52.818520 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:52.818494 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:53.890516 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:53.890475 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:56.050400 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:56.050362 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:57.150358 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:57.150320 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:57.292057 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:57.292022 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:57.467561 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:57.467475 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj"] Oct 30 14:46:57.467856 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:57.467811 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj" podUID="2de9f651-0083-42cd-89b2-6d28d35bb449" containerName="istio-proxy" containerID="cri-o://e2e8377d208c59d979d3c59cf2d0733e9dfd509fa7f957e13d571980796e509a" gracePeriod=30 Oct 30 14:46:57.541261 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:57.541231 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:58.013504 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:46:58.013462 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:46:58.629339 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:58.629302 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:46:59.074386 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:59.074354 4982 patch_prober.go:29] interesting pod/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 14:46:59.074545 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:46:59.074407 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj" podUID="2de9f651-0083-42cd-89b2-6d28d35bb449" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 14:47:00.096488 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:00.096438 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:00.438946 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:00.438850 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:00.499529 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:00.499482 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-396d70-86964fb4d6-zmxfm" podUID="973f4a7b-21e1-4e4d-8190-56032de67388" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:47:00.499697 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:00.499574 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-396d70-56874f4975-w4bw5" podUID="b8343dab-b9ab-4453-bfa3-aada32b3ebf0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:47:00.634125 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:00.634000 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:00.788765 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:00.788683 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:01.034313 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:01.034278 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:02.745793 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:02.745734 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj" Oct 30 14:47:02.762036 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:02.762010 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2de9f651-0083-42cd-89b2-6d28d35bb449-kube-api-access-r97zz\" (UniqueName: \"kubernetes.io/projected/2de9f651-0083-42cd-89b2-6d28d35bb449-kube-api-access-r97zz\") pod \"2de9f651-0083-42cd-89b2-6d28d35bb449\" (UID: \"2de9f651-0083-42cd-89b2-6d28d35bb449\") " Oct 30 14:47:02.762145 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:02.762044 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2de9f651-0083-42cd-89b2-6d28d35bb449-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2de9f651-0083-42cd-89b2-6d28d35bb449-istio-data\") pod \"2de9f651-0083-42cd-89b2-6d28d35bb449\" (UID: \"2de9f651-0083-42cd-89b2-6d28d35bb449\") " Oct 30 14:47:02.762191 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:02.762168 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2de9f651-0083-42cd-89b2-6d28d35bb449-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2de9f651-0083-42cd-89b2-6d28d35bb449-workload-certs\") pod \"2de9f651-0083-42cd-89b2-6d28d35bb449\" (UID: \"2de9f651-0083-42cd-89b2-6d28d35bb449\") " Oct 30 14:47:02.762231 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:02.762190 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2de9f651-0083-42cd-89b2-6d28d35bb449-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2de9f651-0083-42cd-89b2-6d28d35bb449-istio-envoy\") pod \"2de9f651-0083-42cd-89b2-6d28d35bb449\" (UID: \"2de9f651-0083-42cd-89b2-6d28d35bb449\") " Oct 30 14:47:02.762286 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:02.762244 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2de9f651-0083-42cd-89b2-6d28d35bb449-istio-token\" (UniqueName: \"kubernetes.io/projected/2de9f651-0083-42cd-89b2-6d28d35bb449-istio-token\") pod \"2de9f651-0083-42cd-89b2-6d28d35bb449\" (UID: \"2de9f651-0083-42cd-89b2-6d28d35bb449\") " Oct 30 14:47:02.762354 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:02.762336 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2de9f651-0083-42cd-89b2-6d28d35bb449-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2de9f651-0083-42cd-89b2-6d28d35bb449-istiod-ca-cert\") pod \"2de9f651-0083-42cd-89b2-6d28d35bb449\" (UID: \"2de9f651-0083-42cd-89b2-6d28d35bb449\") " Oct 30 14:47:02.762391 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:02.762380 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2de9f651-0083-42cd-89b2-6d28d35bb449-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2de9f651-0083-42cd-89b2-6d28d35bb449-credential-socket\") pod \"2de9f651-0083-42cd-89b2-6d28d35bb449\" (UID: \"2de9f651-0083-42cd-89b2-6d28d35bb449\") " Oct 30 14:47:02.762444 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:02.762365 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2de9f651-0083-42cd-89b2-6d28d35bb449-istio-data" pod "2de9f651-0083-42cd-89b2-6d28d35bb449" (UID: "2de9f651-0083-42cd-89b2-6d28d35bb449"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:47:02.762444 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:02.762406 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2de9f651-0083-42cd-89b2-6d28d35bb449-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2de9f651-0083-42cd-89b2-6d28d35bb449-workload-socket\") pod \"2de9f651-0083-42cd-89b2-6d28d35bb449\" (UID: \"2de9f651-0083-42cd-89b2-6d28d35bb449\") " Oct 30 14:47:02.762549 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:02.762439 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2de9f651-0083-42cd-89b2-6d28d35bb449-workload-certs" pod "2de9f651-0083-42cd-89b2-6d28d35bb449" (UID: "2de9f651-0083-42cd-89b2-6d28d35bb449"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:47:02.762549 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:02.762458 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2de9f651-0083-42cd-89b2-6d28d35bb449-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2de9f651-0083-42cd-89b2-6d28d35bb449-istio-podinfo\") pod \"2de9f651-0083-42cd-89b2-6d28d35bb449\" (UID: \"2de9f651-0083-42cd-89b2-6d28d35bb449\") " Oct 30 14:47:02.762990 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:02.762666 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2de9f651-0083-42cd-89b2-6d28d35bb449-workload-socket" pod "2de9f651-0083-42cd-89b2-6d28d35bb449" (UID: "2de9f651-0083-42cd-89b2-6d28d35bb449"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:47:02.762990 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:02.762713 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2de9f651-0083-42cd-89b2-6d28d35bb449-credential-socket" pod "2de9f651-0083-42cd-89b2-6d28d35bb449" (UID: "2de9f651-0083-42cd-89b2-6d28d35bb449"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:47:02.762990 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:02.762935 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2de9f651-0083-42cd-89b2-6d28d35bb449-istiod-ca-cert" pod "2de9f651-0083-42cd-89b2-6d28d35bb449" (UID: "2de9f651-0083-42cd-89b2-6d28d35bb449"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:47:02.763230 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:02.763169 4982 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2de9f651-0083-42cd-89b2-6d28d35bb449-istiod-ca-cert\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:47:02.763230 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:02.763192 4982 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2de9f651-0083-42cd-89b2-6d28d35bb449-credential-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:47:02.763230 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:02.763208 4982 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2de9f651-0083-42cd-89b2-6d28d35bb449-workload-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:47:02.763230 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:02.763222 4982 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2de9f651-0083-42cd-89b2-6d28d35bb449-istio-data\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:47:02.763398 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:02.763235 4982 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2de9f651-0083-42cd-89b2-6d28d35bb449-workload-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:47:02.764248 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:02.764228 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2de9f651-0083-42cd-89b2-6d28d35bb449-istio-envoy" pod "2de9f651-0083-42cd-89b2-6d28d35bb449" (UID: "2de9f651-0083-42cd-89b2-6d28d35bb449"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:47:02.764429 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:02.764409 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2de9f651-0083-42cd-89b2-6d28d35bb449-istio-podinfo" pod "2de9f651-0083-42cd-89b2-6d28d35bb449" (UID: "2de9f651-0083-42cd-89b2-6d28d35bb449"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:47:02.764665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:02.764634 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2de9f651-0083-42cd-89b2-6d28d35bb449-kube-api-access-r97zz" pod "2de9f651-0083-42cd-89b2-6d28d35bb449" (UID: "2de9f651-0083-42cd-89b2-6d28d35bb449"). InnerVolumeSpecName "kube-api-access-r97zz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:47:02.764721 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:02.764691 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2de9f651-0083-42cd-89b2-6d28d35bb449-istio-token" pod "2de9f651-0083-42cd-89b2-6d28d35bb449" (UID: "2de9f651-0083-42cd-89b2-6d28d35bb449"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:47:02.864253 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:02.864219 4982 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2de9f651-0083-42cd-89b2-6d28d35bb449-istio-token\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:47:02.864253 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:02.864252 4982 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2de9f651-0083-42cd-89b2-6d28d35bb449-istio-podinfo\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:47:02.864481 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:02.864265 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r97zz\" (UniqueName: \"kubernetes.io/projected/2de9f651-0083-42cd-89b2-6d28d35bb449-kube-api-access-r97zz\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:47:02.864481 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:02.864279 4982 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2de9f651-0083-42cd-89b2-6d28d35bb449-istio-envoy\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:47:03.269523 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:03.269485 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:03.332238 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:03.332203 4982 generic.go:356] "Generic (PLEG): container finished" podID="2de9f651-0083-42cd-89b2-6d28d35bb449" containerID="e2e8377d208c59d979d3c59cf2d0733e9dfd509fa7f957e13d571980796e509a" exitCode=0 Oct 30 14:47:03.332423 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:03.332261 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj" event={"ID":"2de9f651-0083-42cd-89b2-6d28d35bb449","Type":"ContainerDied","Data":"e2e8377d208c59d979d3c59cf2d0733e9dfd509fa7f957e13d571980796e509a"} Oct 30 14:47:03.332423 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:03.332268 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj" Oct 30 14:47:03.332423 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:03.332362 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj" event={"ID":"2de9f651-0083-42cd-89b2-6d28d35bb449","Type":"ContainerDied","Data":"40bb0d63f3a67212de14391f8e86eeab7c060397645bac2dcba7a5acd570a0a1"} Oct 30 14:47:03.332423 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:03.332394 4982 scope.go:122] "RemoveContainer" containerID="e2e8377d208c59d979d3c59cf2d0733e9dfd509fa7f957e13d571980796e509a" Oct 30 14:47:03.343243 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:03.343224 4982 scope.go:122] "RemoveContainer" containerID="e5559289b82187bb9c7a4148b85b8a8b2bbe15ed976c90af906c8283cf10f3c4" Oct 30 14:47:03.352572 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:03.352549 4982 scope.go:122] "RemoveContainer" containerID="e2e8377d208c59d979d3c59cf2d0733e9dfd509fa7f957e13d571980796e509a" Oct 30 14:47:03.352863 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:47:03.352842 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2e8377d208c59d979d3c59cf2d0733e9dfd509fa7f957e13d571980796e509a\": container with ID starting with e2e8377d208c59d979d3c59cf2d0733e9dfd509fa7f957e13d571980796e509a not found: ID does not exist" containerID="e2e8377d208c59d979d3c59cf2d0733e9dfd509fa7f957e13d571980796e509a" Oct 30 14:47:03.352914 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:03.352873 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2e8377d208c59d979d3c59cf2d0733e9dfd509fa7f957e13d571980796e509a"} err="failed to get container status \"e2e8377d208c59d979d3c59cf2d0733e9dfd509fa7f957e13d571980796e509a\": rpc error: code = NotFound desc = could not find container \"e2e8377d208c59d979d3c59cf2d0733e9dfd509fa7f957e13d571980796e509a\": container with ID starting with e2e8377d208c59d979d3c59cf2d0733e9dfd509fa7f957e13d571980796e509a not found: ID does not exist" Oct 30 14:47:03.352914 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:03.352890 4982 scope.go:122] "RemoveContainer" containerID="e5559289b82187bb9c7a4148b85b8a8b2bbe15ed976c90af906c8283cf10f3c4" Oct 30 14:47:03.353119 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:47:03.353103 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5559289b82187bb9c7a4148b85b8a8b2bbe15ed976c90af906c8283cf10f3c4\": container with ID starting with e5559289b82187bb9c7a4148b85b8a8b2bbe15ed976c90af906c8283cf10f3c4 not found: ID does not exist" containerID="e5559289b82187bb9c7a4148b85b8a8b2bbe15ed976c90af906c8283cf10f3c4" Oct 30 14:47:03.353169 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:03.353124 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5559289b82187bb9c7a4148b85b8a8b2bbe15ed976c90af906c8283cf10f3c4"} err="failed to get container status \"e5559289b82187bb9c7a4148b85b8a8b2bbe15ed976c90af906c8283cf10f3c4\": rpc error: code = NotFound desc = could not find container \"e5559289b82187bb9c7a4148b85b8a8b2bbe15ed976c90af906c8283cf10f3c4\": container with ID starting with e5559289b82187bb9c7a4148b85b8a8b2bbe15ed976c90af906c8283cf10f3c4 not found: ID does not exist" Oct 30 14:47:03.401203 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:03.401172 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj"] Oct 30 14:47:03.411108 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:03.411079 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-gxhnj"] Oct 30 14:47:04.401869 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:04.401839 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2de9f651-0083-42cd-89b2-6d28d35bb449" path="/var/lib/kubelet/pods/2de9f651-0083-42cd-89b2-6d28d35bb449/volumes" Oct 30 14:47:04.788491 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:04.788385 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:04.976166 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:04.976133 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:06.283048 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:06.283008 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:08.564383 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:08.564320 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:08.725782 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:08.725741 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:08.883620 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:08.883534 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:10.499508 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:10.499469 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-396d70-86964fb4d6-zmxfm" podUID="973f4a7b-21e1-4e4d-8190-56032de67388" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:47:10.499983 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:10.499955 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-396d70-56874f4975-w4bw5" podUID="b8343dab-b9ab-4453-bfa3-aada32b3ebf0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:47:11.471395 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:11.471348 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:12.411179 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:12.411151 4982 generic.go:356] "Generic (PLEG): container finished" podID="973f4a7b-21e1-4e4d-8190-56032de67388" containerID="dc8275981ecb1e7a37a746490d9d950946536e02c7dd9cfc2a25f60eeaec2c34" exitCode=0 Oct 30 14:47:12.411721 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:12.411229 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-396d70-86964fb4d6-zmxfm" event={"ID":"973f4a7b-21e1-4e4d-8190-56032de67388","Type":"ContainerDied","Data":"dc8275981ecb1e7a37a746490d9d950946536e02c7dd9cfc2a25f60eeaec2c34"} Oct 30 14:47:12.411721 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:12.411290 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-396d70-86964fb4d6-zmxfm" event={"ID":"973f4a7b-21e1-4e4d-8190-56032de67388","Type":"ContainerDied","Data":"87d68f3808213c7f29f98fca14e0ce0ade36c84b6b3ee2ee5d362802823a186c"} Oct 30 14:47:12.411721 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:12.411319 4982 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87d68f3808213c7f29f98fca14e0ce0ade36c84b6b3ee2ee5d362802823a186c" Oct 30 14:47:12.413791 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:12.413458 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-396d70-86964fb4d6-zmxfm" Oct 30 14:47:12.413791 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:12.413480 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-396d70-56874f4975-w4bw5" event={"ID":"b8343dab-b9ab-4453-bfa3-aada32b3ebf0","Type":"ContainerDied","Data":"4b21cc04f93430c749d2d445ef0195bdb38c0db93fb5477ab5c175a4086ebfa9"} Oct 30 14:47:12.413791 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:12.413464 4982 generic.go:356] "Generic (PLEG): container finished" podID="b8343dab-b9ab-4453-bfa3-aada32b3ebf0" containerID="4b21cc04f93430c749d2d445ef0195bdb38c0db93fb5477ab5c175a4086ebfa9" exitCode=0 Oct 30 14:47:12.459350 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:12.459328 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-396d70-56874f4975-w4bw5" Oct 30 14:47:12.506043 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:12.506012 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/973f4a7b-21e1-4e4d-8190-56032de67388-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/973f4a7b-21e1-4e4d-8190-56032de67388-plugins-config-volume\") pod \"973f4a7b-21e1-4e4d-8190-56032de67388\" (UID: \"973f4a7b-21e1-4e4d-8190-56032de67388\") " Oct 30 14:47:12.506176 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:12.506137 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/973f4a7b-21e1-4e4d-8190-56032de67388-kube-api-access-q2f4s\" (UniqueName: \"kubernetes.io/projected/973f4a7b-21e1-4e4d-8190-56032de67388-kube-api-access-q2f4s\") pod \"973f4a7b-21e1-4e4d-8190-56032de67388\" (UID: \"973f4a7b-21e1-4e4d-8190-56032de67388\") " Oct 30 14:47:12.506905 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:12.506874 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/973f4a7b-21e1-4e4d-8190-56032de67388-plugins-config-volume" pod "973f4a7b-21e1-4e4d-8190-56032de67388" (UID: "973f4a7b-21e1-4e4d-8190-56032de67388"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:47:12.512809 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:12.512783 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/973f4a7b-21e1-4e4d-8190-56032de67388-kube-api-access-q2f4s" pod "973f4a7b-21e1-4e4d-8190-56032de67388" (UID: "973f4a7b-21e1-4e4d-8190-56032de67388"). InnerVolumeSpecName "kube-api-access-q2f4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:47:12.608088 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:12.608061 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8343dab-b9ab-4453-bfa3-aada32b3ebf0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8343dab-b9ab-4453-bfa3-aada32b3ebf0-plugins-config-volume\") pod \"b8343dab-b9ab-4453-bfa3-aada32b3ebf0\" (UID: \"b8343dab-b9ab-4453-bfa3-aada32b3ebf0\") " Oct 30 14:47:12.608088 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:12.608093 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8343dab-b9ab-4453-bfa3-aada32b3ebf0-kube-api-access-ctlps\" (UniqueName: \"kubernetes.io/projected/b8343dab-b9ab-4453-bfa3-aada32b3ebf0-kube-api-access-ctlps\") pod \"b8343dab-b9ab-4453-bfa3-aada32b3ebf0\" (UID: \"b8343dab-b9ab-4453-bfa3-aada32b3ebf0\") " Oct 30 14:47:12.608422 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:12.608407 4982 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/973f4a7b-21e1-4e4d-8190-56032de67388-plugins-config-volume\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:47:12.608495 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:12.608425 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2f4s\" (UniqueName: \"kubernetes.io/projected/973f4a7b-21e1-4e4d-8190-56032de67388-kube-api-access-q2f4s\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:47:12.608495 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:12.608473 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8343dab-b9ab-4453-bfa3-aada32b3ebf0-plugins-config-volume" pod "b8343dab-b9ab-4453-bfa3-aada32b3ebf0" (UID: "b8343dab-b9ab-4453-bfa3-aada32b3ebf0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:47:12.610331 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:12.610277 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8343dab-b9ab-4453-bfa3-aada32b3ebf0-kube-api-access-ctlps" pod "b8343dab-b9ab-4453-bfa3-aada32b3ebf0" (UID: "b8343dab-b9ab-4453-bfa3-aada32b3ebf0"). InnerVolumeSpecName "kube-api-access-ctlps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:47:12.709027 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:12.709001 4982 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8343dab-b9ab-4453-bfa3-aada32b3ebf0-plugins-config-volume\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:47:12.709027 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:12.709024 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctlps\" (UniqueName: \"kubernetes.io/projected/b8343dab-b9ab-4453-bfa3-aada32b3ebf0-kube-api-access-ctlps\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:47:13.384136 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:13.384047 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:13.423484 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:13.423452 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-396d70-56874f4975-w4bw5" Oct 30 14:47:13.423973 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:13.423479 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-396d70-56874f4975-w4bw5" event={"ID":"b8343dab-b9ab-4453-bfa3-aada32b3ebf0","Type":"ContainerDied","Data":"15f3ae1d2e300de3536033c249997622b888c9042a723f516a43364f5d25065b"} Oct 30 14:47:13.423973 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:13.423558 4982 scope.go:122] "RemoveContainer" containerID="4b21cc04f93430c749d2d445ef0195bdb38c0db93fb5477ab5c175a4086ebfa9" Oct 30 14:47:13.423973 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:13.423795 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-396d70-86964fb4d6-zmxfm" Oct 30 14:47:13.527234 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:13.527204 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-396d70-56874f4975-w4bw5"] Oct 30 14:47:13.541751 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:13.541725 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-396d70-56874f4975-w4bw5"] Oct 30 14:47:13.591892 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:13.591863 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-396d70-86964fb4d6-zmxfm"] Oct 30 14:47:13.598037 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:13.598010 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-396d70-86964fb4d6-zmxfm"] Oct 30 14:47:14.401843 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:14.401809 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="973f4a7b-21e1-4e4d-8190-56032de67388" path="/var/lib/kubelet/pods/973f4a7b-21e1-4e4d-8190-56032de67388/volumes" Oct 30 14:47:14.402219 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:14.402206 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8343dab-b9ab-4453-bfa3-aada32b3ebf0" path="/var/lib/kubelet/pods/b8343dab-b9ab-4453-bfa3-aada32b3ebf0/volumes" Oct 30 14:47:15.389132 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:15.389100 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:17.462945 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:17.462914 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:19.702524 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:19.702492 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:22.279582 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:22.279530 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:24.327723 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:24.327693 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:26.413393 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:26.413345 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:28.014931 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:47:28.014892 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:47:29.146558 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:29.146513 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:31.251489 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:31.251239 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:33.544161 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:33.544113 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:35.761118 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:35.761083 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:37.488306 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:37.488257 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:38.068775 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:38.068728 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:38.233665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:38.233602 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:38.411896 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:38.411809 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:38.926694 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:38.926629 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:40.788899 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:40.788868 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0e8b9-748c768747-vlzmg"] Oct 30 14:47:40.789436 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:40.789117 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0e8b9-748c768747-vlzmg" podUID="738cda0d-b92c-4e95-8b44-bb296e7b68dd" containerName="maas-api" containerID="cri-o://9489a52983ba107be28f0dc06b54946ba7e56cbd208a478420f0a094f5d0c2d2" gracePeriod=30 Oct 30 14:47:40.906475 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:40.906444 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:41.115727 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:41.115697 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:41.127140 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:41.127116 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0e8b9-748c768747-vlzmg" Oct 30 14:47:41.235951 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:41.235899 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/738cda0d-b92c-4e95-8b44-bb296e7b68dd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/738cda0d-b92c-4e95-8b44-bb296e7b68dd-openshift-service-ca\") pod \"738cda0d-b92c-4e95-8b44-bb296e7b68dd\" (UID: \"738cda0d-b92c-4e95-8b44-bb296e7b68dd\") " Oct 30 14:47:41.235951 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:41.235945 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/738cda0d-b92c-4e95-8b44-bb296e7b68dd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/738cda0d-b92c-4e95-8b44-bb296e7b68dd-metrics-certs\") pod \"738cda0d-b92c-4e95-8b44-bb296e7b68dd\" (UID: \"738cda0d-b92c-4e95-8b44-bb296e7b68dd\") " Oct 30 14:47:41.236187 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:41.235983 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/738cda0d-b92c-4e95-8b44-bb296e7b68dd-kube-api-access-gkp8s\" (UniqueName: \"kubernetes.io/projected/738cda0d-b92c-4e95-8b44-bb296e7b68dd-kube-api-access-gkp8s\") pod \"738cda0d-b92c-4e95-8b44-bb296e7b68dd\" (UID: \"738cda0d-b92c-4e95-8b44-bb296e7b68dd\") " Oct 30 14:47:41.236187 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:41.236045 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/738cda0d-b92c-4e95-8b44-bb296e7b68dd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/738cda0d-b92c-4e95-8b44-bb296e7b68dd-maas-api-tls\") pod \"738cda0d-b92c-4e95-8b44-bb296e7b68dd\" (UID: \"738cda0d-b92c-4e95-8b44-bb296e7b68dd\") " Oct 30 14:47:41.236410 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:41.236378 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/738cda0d-b92c-4e95-8b44-bb296e7b68dd-openshift-service-ca" pod "738cda0d-b92c-4e95-8b44-bb296e7b68dd" (UID: "738cda0d-b92c-4e95-8b44-bb296e7b68dd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:47:41.236742 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:41.236720 4982 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/738cda0d-b92c-4e95-8b44-bb296e7b68dd-openshift-service-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:47:41.238185 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:41.238163 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/738cda0d-b92c-4e95-8b44-bb296e7b68dd-metrics-certs" pod "738cda0d-b92c-4e95-8b44-bb296e7b68dd" (UID: "738cda0d-b92c-4e95-8b44-bb296e7b68dd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:47:41.238665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:41.238626 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/738cda0d-b92c-4e95-8b44-bb296e7b68dd-maas-api-tls" pod "738cda0d-b92c-4e95-8b44-bb296e7b68dd" (UID: "738cda0d-b92c-4e95-8b44-bb296e7b68dd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:47:41.238766 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:41.238684 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/738cda0d-b92c-4e95-8b44-bb296e7b68dd-kube-api-access-gkp8s" pod "738cda0d-b92c-4e95-8b44-bb296e7b68dd" (UID: "738cda0d-b92c-4e95-8b44-bb296e7b68dd"). InnerVolumeSpecName "kube-api-access-gkp8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:47:41.273244 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:41.273215 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:41.337467 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:41.337444 4982 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/738cda0d-b92c-4e95-8b44-bb296e7b68dd-metrics-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:47:41.337560 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:41.337469 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkp8s\" (UniqueName: \"kubernetes.io/projected/738cda0d-b92c-4e95-8b44-bb296e7b68dd-kube-api-access-gkp8s\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:47:41.337560 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:41.337479 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/738cda0d-b92c-4e95-8b44-bb296e7b68dd-maas-api-tls\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:47:41.559095 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:41.559064 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:41.635736 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:41.635697 4982 generic.go:356] "Generic (PLEG): container finished" podID="738cda0d-b92c-4e95-8b44-bb296e7b68dd" containerID="9489a52983ba107be28f0dc06b54946ba7e56cbd208a478420f0a094f5d0c2d2" exitCode=0 Oct 30 14:47:41.635923 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:41.635765 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0e8b9-748c768747-vlzmg" Oct 30 14:47:41.635923 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:41.635779 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0e8b9-748c768747-vlzmg" event={"ID":"738cda0d-b92c-4e95-8b44-bb296e7b68dd","Type":"ContainerDied","Data":"9489a52983ba107be28f0dc06b54946ba7e56cbd208a478420f0a094f5d0c2d2"} Oct 30 14:47:41.635923 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:41.635843 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0e8b9-748c768747-vlzmg" event={"ID":"738cda0d-b92c-4e95-8b44-bb296e7b68dd","Type":"ContainerDied","Data":"2b944bc3ef15e6b44b895dea611e9ac811e513fd51018777134aabe71a55249e"} Oct 30 14:47:41.635923 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:41.635874 4982 scope.go:122] "RemoveContainer" containerID="9489a52983ba107be28f0dc06b54946ba7e56cbd208a478420f0a094f5d0c2d2" Oct 30 14:47:41.650520 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:41.650504 4982 scope.go:122] "RemoveContainer" containerID="9489a52983ba107be28f0dc06b54946ba7e56cbd208a478420f0a094f5d0c2d2" Oct 30 14:47:41.650835 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:47:41.650817 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9489a52983ba107be28f0dc06b54946ba7e56cbd208a478420f0a094f5d0c2d2\": container with ID starting with 9489a52983ba107be28f0dc06b54946ba7e56cbd208a478420f0a094f5d0c2d2 not found: ID does not exist" containerID="9489a52983ba107be28f0dc06b54946ba7e56cbd208a478420f0a094f5d0c2d2" Oct 30 14:47:41.650883 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:41.650843 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9489a52983ba107be28f0dc06b54946ba7e56cbd208a478420f0a094f5d0c2d2"} err="failed to get container status \"9489a52983ba107be28f0dc06b54946ba7e56cbd208a478420f0a094f5d0c2d2\": rpc error: code = NotFound desc = could not find container \"9489a52983ba107be28f0dc06b54946ba7e56cbd208a478420f0a094f5d0c2d2\": container with ID starting with 9489a52983ba107be28f0dc06b54946ba7e56cbd208a478420f0a094f5d0c2d2 not found: ID does not exist" Oct 30 14:47:41.705454 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:41.705414 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0e8b9-748c768747-vlzmg"] Oct 30 14:47:41.708998 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:41.708967 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0e8b9-748c768747-vlzmg"] Oct 30 14:47:41.721520 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:41.721493 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:42.400096 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:42.400063 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="738cda0d-b92c-4e95-8b44-bb296e7b68dd" path="/var/lib/kubelet/pods/738cda0d-b92c-4e95-8b44-bb296e7b68dd/volumes" Oct 30 14:47:43.429664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:43.429603 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:43.844341 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:43.844284 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6176c3-645458f68c-q8b98"] Oct 30 14:47:43.844913 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:43.844878 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6176c3-645458f68c-q8b98" podUID="3bd7ef80-e959-407e-8b8e-145600cdbc39" containerName="maas-api" containerID="cri-o://bde418952e69bd7dd22846190fa299be2c90428bf8f281150e477f42d04f8a5e" gracePeriod=30 Oct 30 14:47:43.910944 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:43.910910 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:44.066631 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:44.066555 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:44.165191 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:44.165163 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6176c3-645458f68c-q8b98" Oct 30 14:47:44.219028 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:44.218998 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:44.285112 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:44.285080 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bd7ef80-e959-407e-8b8e-145600cdbc39-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bd7ef80-e959-407e-8b8e-145600cdbc39-openshift-service-ca\") pod \"3bd7ef80-e959-407e-8b8e-145600cdbc39\" (UID: \"3bd7ef80-e959-407e-8b8e-145600cdbc39\") " Oct 30 14:47:44.285284 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:44.285148 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bd7ef80-e959-407e-8b8e-145600cdbc39-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bd7ef80-e959-407e-8b8e-145600cdbc39-maas-api-tls\") pod \"3bd7ef80-e959-407e-8b8e-145600cdbc39\" (UID: \"3bd7ef80-e959-407e-8b8e-145600cdbc39\") " Oct 30 14:47:44.285284 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:44.285255 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bd7ef80-e959-407e-8b8e-145600cdbc39-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bd7ef80-e959-407e-8b8e-145600cdbc39-metrics-certs\") pod \"3bd7ef80-e959-407e-8b8e-145600cdbc39\" (UID: \"3bd7ef80-e959-407e-8b8e-145600cdbc39\") " Oct 30 14:47:44.285410 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:44.285335 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bd7ef80-e959-407e-8b8e-145600cdbc39-kube-api-access-gm9mz\" (UniqueName: \"kubernetes.io/projected/3bd7ef80-e959-407e-8b8e-145600cdbc39-kube-api-access-gm9mz\") pod \"3bd7ef80-e959-407e-8b8e-145600cdbc39\" (UID: \"3bd7ef80-e959-407e-8b8e-145600cdbc39\") " Oct 30 14:47:44.285479 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:44.285449 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bd7ef80-e959-407e-8b8e-145600cdbc39-openshift-service-ca" pod "3bd7ef80-e959-407e-8b8e-145600cdbc39" (UID: "3bd7ef80-e959-407e-8b8e-145600cdbc39"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:47:44.285872 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:44.285842 4982 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bd7ef80-e959-407e-8b8e-145600cdbc39-openshift-service-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:47:44.287500 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:44.287475 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bd7ef80-e959-407e-8b8e-145600cdbc39-maas-api-tls" pod "3bd7ef80-e959-407e-8b8e-145600cdbc39" (UID: "3bd7ef80-e959-407e-8b8e-145600cdbc39"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:47:44.287619 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:44.287502 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bd7ef80-e959-407e-8b8e-145600cdbc39-metrics-certs" pod "3bd7ef80-e959-407e-8b8e-145600cdbc39" (UID: "3bd7ef80-e959-407e-8b8e-145600cdbc39"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:47:44.287957 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:44.287936 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bd7ef80-e959-407e-8b8e-145600cdbc39-kube-api-access-gm9mz" pod "3bd7ef80-e959-407e-8b8e-145600cdbc39" (UID: "3bd7ef80-e959-407e-8b8e-145600cdbc39"). InnerVolumeSpecName "kube-api-access-gm9mz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:47:44.394719 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:44.394373 4982 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bd7ef80-e959-407e-8b8e-145600cdbc39-metrics-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:47:44.394719 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:44.394408 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gm9mz\" (UniqueName: \"kubernetes.io/projected/3bd7ef80-e959-407e-8b8e-145600cdbc39-kube-api-access-gm9mz\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:47:44.394719 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:44.394447 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bd7ef80-e959-407e-8b8e-145600cdbc39-maas-api-tls\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:47:44.667094 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:44.667019 4982 generic.go:356] "Generic (PLEG): container finished" podID="3bd7ef80-e959-407e-8b8e-145600cdbc39" containerID="bde418952e69bd7dd22846190fa299be2c90428bf8f281150e477f42d04f8a5e" exitCode=0 Oct 30 14:47:44.667094 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:44.667063 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6176c3-645458f68c-q8b98" Oct 30 14:47:44.667608 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:44.667108 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6176c3-645458f68c-q8b98" event={"ID":"3bd7ef80-e959-407e-8b8e-145600cdbc39","Type":"ContainerDied","Data":"bde418952e69bd7dd22846190fa299be2c90428bf8f281150e477f42d04f8a5e"} Oct 30 14:47:44.667608 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:44.667161 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6176c3-645458f68c-q8b98" event={"ID":"3bd7ef80-e959-407e-8b8e-145600cdbc39","Type":"ContainerDied","Data":"ccbe4ffe794161be7977a33fd6b9b3fbbaa32293cc6f26eca0318073a2a2d83a"} Oct 30 14:47:44.667608 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:44.667188 4982 scope.go:122] "RemoveContainer" containerID="bde418952e69bd7dd22846190fa299be2c90428bf8f281150e477f42d04f8a5e" Oct 30 14:47:44.676896 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:44.676877 4982 scope.go:122] "RemoveContainer" containerID="bde418952e69bd7dd22846190fa299be2c90428bf8f281150e477f42d04f8a5e" Oct 30 14:47:44.677159 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:47:44.677140 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bde418952e69bd7dd22846190fa299be2c90428bf8f281150e477f42d04f8a5e\": container with ID starting with bde418952e69bd7dd22846190fa299be2c90428bf8f281150e477f42d04f8a5e not found: ID does not exist" containerID="bde418952e69bd7dd22846190fa299be2c90428bf8f281150e477f42d04f8a5e" Oct 30 14:47:44.677229 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:44.677166 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bde418952e69bd7dd22846190fa299be2c90428bf8f281150e477f42d04f8a5e"} err="failed to get container status \"bde418952e69bd7dd22846190fa299be2c90428bf8f281150e477f42d04f8a5e\": rpc error: code = NotFound desc = could not find container \"bde418952e69bd7dd22846190fa299be2c90428bf8f281150e477f42d04f8a5e\": container with ID starting with bde418952e69bd7dd22846190fa299be2c90428bf8f281150e477f42d04f8a5e not found: ID does not exist" Oct 30 14:47:44.723040 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:44.722997 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:44.725792 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:44.725769 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6176c3-645458f68c-q8b98"] Oct 30 14:47:44.738958 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:44.738935 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6176c3-645458f68c-q8b98"] Oct 30 14:47:44.867769 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:44.867739 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:45.026809 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:45.026731 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:45.422806 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:45.422766 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:46.400907 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:46.400870 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bd7ef80-e959-407e-8b8e-145600cdbc39" path="/var/lib/kubelet/pods/3bd7ef80-e959-407e-8b8e-145600cdbc39/volumes" Oct 30 14:47:46.872241 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:46.872209 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:47.028702 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:47.028667 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:47.593417 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:47.593374 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:49.272960 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:49.272917 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:52.121301 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:52.121249 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:52.295401 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:52.295366 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:52.753538 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:52.753506 4982 scope.go:122] "RemoveContainer" containerID="dc8275981ecb1e7a37a746490d9d950946536e02c7dd9cfc2a25f60eeaec2c34" Oct 30 14:47:53.072506 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:53.072472 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:53.241954 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:53.241919 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:54.023607 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:54.023576 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:54.192352 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:54.192317 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:54.434013 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:54.433979 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:54.672148 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:54.672081 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:54.903882 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:54.903845 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90b406-6jzrm"] Oct 30 14:47:54.904700 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:54.904678 4982 cpu_manager.go:414] "containerMap: removing container" podUID="738cda0d-b92c-4e95-8b44-bb296e7b68dd" containerName="maas-api" Oct 30 14:47:54.904775 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:54.904702 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="738cda0d-b92c-4e95-8b44-bb296e7b68dd" containerName="maas-api" Oct 30 14:47:54.904775 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:54.904720 4982 cpu_manager.go:414] "containerMap: removing container" podUID="2de9f651-0083-42cd-89b2-6d28d35bb449" containerName="istio-proxy" Oct 30 14:47:54.904775 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:54.904728 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2de9f651-0083-42cd-89b2-6d28d35bb449" containerName="istio-proxy" Oct 30 14:47:54.904775 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:54.904742 4982 cpu_manager.go:414] "containerMap: removing container" podUID="2de9f651-0083-42cd-89b2-6d28d35bb449" containerName="istio-proxy" Oct 30 14:47:54.904775 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:54.904750 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2de9f651-0083-42cd-89b2-6d28d35bb449" containerName="istio-proxy" Oct 30 14:47:54.904775 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:54.904760 4982 cpu_manager.go:414] "containerMap: removing container" podUID="973f4a7b-21e1-4e4d-8190-56032de67388" containerName="payload-processing" Oct 30 14:47:54.904775 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:54.904768 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="973f4a7b-21e1-4e4d-8190-56032de67388" containerName="payload-processing" Oct 30 14:47:54.905057 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:54.904782 4982 cpu_manager.go:414] "containerMap: removing container" podUID="b8343dab-b9ab-4453-bfa3-aada32b3ebf0" containerName="payload-pre-processing" Oct 30 14:47:54.905057 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:54.904788 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8343dab-b9ab-4453-bfa3-aada32b3ebf0" containerName="payload-pre-processing" Oct 30 14:47:54.905057 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:54.904818 4982 cpu_manager.go:414] "containerMap: removing container" podUID="e6c9e188-d466-4ad8-b72c-f89717ff9f47" containerName="maas-api" Oct 30 14:47:54.905057 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:54.904826 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6c9e188-d466-4ad8-b72c-f89717ff9f47" containerName="maas-api" Oct 30 14:47:54.905057 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:54.904843 4982 cpu_manager.go:414] "containerMap: removing container" podUID="b4f5d962-a1e9-4f43-85c7-e4984032b6da" containerName="payload-pre-processing" Oct 30 14:47:54.905057 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:54.904851 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4f5d962-a1e9-4f43-85c7-e4984032b6da" containerName="payload-pre-processing" Oct 30 14:47:54.905057 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:54.904870 4982 cpu_manager.go:414] "containerMap: removing container" podUID="3bd7ef80-e959-407e-8b8e-145600cdbc39" containerName="maas-api" Oct 30 14:47:54.905057 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:54.904879 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bd7ef80-e959-407e-8b8e-145600cdbc39" containerName="maas-api" Oct 30 14:47:54.905057 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:54.905029 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4f5d962-a1e9-4f43-85c7-e4984032b6da" containerName="payload-pre-processing" Oct 30 14:47:54.905057 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:54.905042 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bd7ef80-e959-407e-8b8e-145600cdbc39" containerName="maas-api" Oct 30 14:47:54.905057 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:54.905059 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="738cda0d-b92c-4e95-8b44-bb296e7b68dd" containerName="maas-api" Oct 30 14:47:54.905468 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:54.905068 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="2de9f651-0083-42cd-89b2-6d28d35bb449" containerName="istio-proxy" Oct 30 14:47:54.905468 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:54.905075 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="2de9f651-0083-42cd-89b2-6d28d35bb449" containerName="istio-proxy" Oct 30 14:47:54.905468 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:54.905082 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="973f4a7b-21e1-4e4d-8190-56032de67388" containerName="payload-processing" Oct 30 14:47:54.905468 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:54.905091 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8343dab-b9ab-4453-bfa3-aada32b3ebf0" containerName="payload-pre-processing" Oct 30 14:47:54.905468 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:54.905103 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6c9e188-d466-4ad8-b72c-f89717ff9f47" containerName="maas-api" Oct 30 14:47:54.910618 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:54.910595 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90b406-6jzrm" Oct 30 14:47:54.931685 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:54.931661 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90b406-6jzrm"] Oct 30 14:47:54.933518 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:54.933487 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncxfn\" (UniqueName: \"kubernetes.io/projected/12350227-cab2-45e5-aadc-f65f4e48bca6-kube-api-access-ncxfn\") pod \"maas-api-revoke-keys-e2e-ipp-b-90b406-6jzrm\" (UID: \"12350227-cab2-45e5-aadc-f65f4e48bca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90b406-6jzrm" Oct 30 14:47:54.933662 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:54.933600 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12350227-cab2-45e5-aadc-f65f4e48bca6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-90b406-6jzrm\" (UID: \"12350227-cab2-45e5-aadc-f65f4e48bca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90b406-6jzrm" Oct 30 14:47:54.933736 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:54.933709 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12350227-cab2-45e5-aadc-f65f4e48bca6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-90b406-6jzrm\" (UID: \"12350227-cab2-45e5-aadc-f65f4e48bca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90b406-6jzrm" Oct 30 14:47:54.941391 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:54.941369 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:47:54.941515 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:54.941438 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:47:55.034433 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:55.034404 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12350227-cab2-45e5-aadc-f65f4e48bca6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-90b406-6jzrm\" (UID: \"12350227-cab2-45e5-aadc-f65f4e48bca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90b406-6jzrm" Oct 30 14:47:55.034548 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:55.034453 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12350227-cab2-45e5-aadc-f65f4e48bca6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-90b406-6jzrm\" (UID: \"12350227-cab2-45e5-aadc-f65f4e48bca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90b406-6jzrm" Oct 30 14:47:55.034652 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:55.034573 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncxfn\" (UniqueName: \"kubernetes.io/projected/12350227-cab2-45e5-aadc-f65f4e48bca6-kube-api-access-ncxfn\") pod \"maas-api-revoke-keys-e2e-ipp-b-90b406-6jzrm\" (UID: \"12350227-cab2-45e5-aadc-f65f4e48bca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90b406-6jzrm" Oct 30 14:47:55.035076 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:55.035054 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12350227-cab2-45e5-aadc-f65f4e48bca6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-90b406-6jzrm\" (UID: \"12350227-cab2-45e5-aadc-f65f4e48bca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90b406-6jzrm" Oct 30 14:47:55.036927 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:55.036901 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12350227-cab2-45e5-aadc-f65f4e48bca6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-90b406-6jzrm\" (UID: \"12350227-cab2-45e5-aadc-f65f4e48bca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90b406-6jzrm" Oct 30 14:47:55.060191 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:55.060162 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncxfn\" (UniqueName: \"kubernetes.io/projected/12350227-cab2-45e5-aadc-f65f4e48bca6-kube-api-access-ncxfn\") pod \"maas-api-revoke-keys-e2e-ipp-b-90b406-6jzrm\" (UID: \"12350227-cab2-45e5-aadc-f65f4e48bca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90b406-6jzrm" Oct 30 14:47:55.226273 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:55.226212 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90b406-6jzrm" Oct 30 14:47:55.532331 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:55.532293 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90b406-6jzrm"] Oct 30 14:47:55.533370 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:47:55.533342 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12350227_cab2_45e5_aadc_f65f4e48bca6.slice/crio-9608bccc73109b6c753ea7cf732e61f61ca4002b25addd4222cbe657b17da6dc WatchSource:0}: Error finding container 9608bccc73109b6c753ea7cf732e61f61ca4002b25addd4222cbe657b17da6dc: Status 404 returned error can't find the container with id 9608bccc73109b6c753ea7cf732e61f61ca4002b25addd4222cbe657b17da6dc Oct 30 14:47:55.744852 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:55.744824 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90b406-6jzrm" event={"ID":"12350227-cab2-45e5-aadc-f65f4e48bca6","Type":"ContainerStarted","Data":"9608bccc73109b6c753ea7cf732e61f61ca4002b25addd4222cbe657b17da6dc"} Oct 30 14:47:56.752787 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:56.752710 4982 generic.go:356] "Generic (PLEG): container finished" podID="12350227-cab2-45e5-aadc-f65f4e48bca6" containerID="86d268d59d78d8923362c622e428d7fac3c9c0a5caf1eab2db638c732cda1880" exitCode=0 Oct 30 14:47:56.753172 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:56.752799 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90b406-6jzrm" event={"ID":"12350227-cab2-45e5-aadc-f65f4e48bca6","Type":"ContainerDied","Data":"86d268d59d78d8923362c622e428d7fac3c9c0a5caf1eab2db638c732cda1880"} Oct 30 14:47:57.174355 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:57.174316 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4"] Oct 30 14:47:57.174613 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:57.174586 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4" podUID="de285ba3-daf9-4154-98b0-95b642020499" containerName="istio-proxy" containerID="cri-o://72bbd3796327d2a8db0fa250d6d3f36b2d26da2f5b2e56bc84458ad2137dae23" gracePeriod=30 Oct 30 14:47:57.183198 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:57.183147 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:57.193396 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:57.193364 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-f0e8b9/ipp-route-f0e8b9-kserve-7f4f866ccc-pz7w2"] Oct 30 14:47:57.334032 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:57.333995 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:57.906395 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:57.906368 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90b406-6jzrm" Oct 30 14:47:57.994324 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:57.994289 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12350227-cab2-45e5-aadc-f65f4e48bca6-kube-api-access-ncxfn\" (UniqueName: \"kubernetes.io/projected/12350227-cab2-45e5-aadc-f65f4e48bca6-kube-api-access-ncxfn\") pod \"12350227-cab2-45e5-aadc-f65f4e48bca6\" (UID: \"12350227-cab2-45e5-aadc-f65f4e48bca6\") " Oct 30 14:47:57.994509 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:57.994342 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12350227-cab2-45e5-aadc-f65f4e48bca6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12350227-cab2-45e5-aadc-f65f4e48bca6-maas-api-openshift-ca\") pod \"12350227-cab2-45e5-aadc-f65f4e48bca6\" (UID: \"12350227-cab2-45e5-aadc-f65f4e48bca6\") " Oct 30 14:47:57.994509 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:57.994441 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12350227-cab2-45e5-aadc-f65f4e48bca6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12350227-cab2-45e5-aadc-f65f4e48bca6-maas-api-xks-ca\") pod \"12350227-cab2-45e5-aadc-f65f4e48bca6\" (UID: \"12350227-cab2-45e5-aadc-f65f4e48bca6\") " Oct 30 14:47:57.994794 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:57.994761 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12350227-cab2-45e5-aadc-f65f4e48bca6-maas-api-openshift-ca" pod "12350227-cab2-45e5-aadc-f65f4e48bca6" (UID: "12350227-cab2-45e5-aadc-f65f4e48bca6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:47:57.995009 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:57.994992 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12350227-cab2-45e5-aadc-f65f4e48bca6-maas-api-openshift-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:47:57.996903 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:57.996875 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12350227-cab2-45e5-aadc-f65f4e48bca6-maas-api-xks-ca" pod "12350227-cab2-45e5-aadc-f65f4e48bca6" (UID: "12350227-cab2-45e5-aadc-f65f4e48bca6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:47:57.997026 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:57.996955 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12350227-cab2-45e5-aadc-f65f4e48bca6-kube-api-access-ncxfn" pod "12350227-cab2-45e5-aadc-f65f4e48bca6" (UID: "12350227-cab2-45e5-aadc-f65f4e48bca6"). InnerVolumeSpecName "kube-api-access-ncxfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:47:58.017915 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:47:58.017844 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:47:58.096045 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:58.096017 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncxfn\" (UniqueName: \"kubernetes.io/projected/12350227-cab2-45e5-aadc-f65f4e48bca6-kube-api-access-ncxfn\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:47:58.096236 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:58.096052 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12350227-cab2-45e5-aadc-f65f4e48bca6-maas-api-xks-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:47:58.770527 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:58.770455 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90b406-6jzrm" Oct 30 14:47:58.770690 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:58.770461 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90b406-6jzrm" event={"ID":"12350227-cab2-45e5-aadc-f65f4e48bca6","Type":"ContainerDied","Data":"9608bccc73109b6c753ea7cf732e61f61ca4002b25addd4222cbe657b17da6dc"} Oct 30 14:47:58.770690 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:58.770570 4982 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9608bccc73109b6c753ea7cf732e61f61ca4002b25addd4222cbe657b17da6dc" Oct 30 14:47:59.143052 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:47:59.143019 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:00.971315 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:00.971286 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:01.704972 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:01.704936 4982 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 14:48:01.705140 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:01.704999 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4" podUID="de285ba3-daf9-4154-98b0-95b642020499" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 14:48:02.798849 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:02.798816 4982 generic.go:356] "Generic (PLEG): container finished" podID="de285ba3-daf9-4154-98b0-95b642020499" containerID="72bbd3796327d2a8db0fa250d6d3f36b2d26da2f5b2e56bc84458ad2137dae23" exitCode=0 Oct 30 14:48:02.799219 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:02.798889 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4" event={"ID":"de285ba3-daf9-4154-98b0-95b642020499","Type":"ContainerDied","Data":"72bbd3796327d2a8db0fa250d6d3f36b2d26da2f5b2e56bc84458ad2137dae23"} Oct 30 14:48:02.860700 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:02.860675 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4" Oct 30 14:48:02.950181 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:02.950093 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de285ba3-daf9-4154-98b0-95b642020499-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de285ba3-daf9-4154-98b0-95b642020499-credential-socket\") pod \"de285ba3-daf9-4154-98b0-95b642020499\" (UID: \"de285ba3-daf9-4154-98b0-95b642020499\") " Oct 30 14:48:02.950357 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:02.950189 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de285ba3-daf9-4154-98b0-95b642020499-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de285ba3-daf9-4154-98b0-95b642020499-istiod-ca-cert\") pod \"de285ba3-daf9-4154-98b0-95b642020499\" (UID: \"de285ba3-daf9-4154-98b0-95b642020499\") " Oct 30 14:48:02.950357 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:02.950224 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de285ba3-daf9-4154-98b0-95b642020499-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de285ba3-daf9-4154-98b0-95b642020499-workload-certs\") pod \"de285ba3-daf9-4154-98b0-95b642020499\" (UID: \"de285ba3-daf9-4154-98b0-95b642020499\") " Oct 30 14:48:02.950357 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:02.950298 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de285ba3-daf9-4154-98b0-95b642020499-istio-token\" (UniqueName: \"kubernetes.io/projected/de285ba3-daf9-4154-98b0-95b642020499-istio-token\") pod \"de285ba3-daf9-4154-98b0-95b642020499\" (UID: \"de285ba3-daf9-4154-98b0-95b642020499\") " Oct 30 14:48:02.950357 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:02.950351 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/de285ba3-daf9-4154-98b0-95b642020499-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de285ba3-daf9-4154-98b0-95b642020499-istio-podinfo\") pod \"de285ba3-daf9-4154-98b0-95b642020499\" (UID: \"de285ba3-daf9-4154-98b0-95b642020499\") " Oct 30 14:48:02.950617 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:02.950374 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de285ba3-daf9-4154-98b0-95b642020499-credential-socket" pod "de285ba3-daf9-4154-98b0-95b642020499" (UID: "de285ba3-daf9-4154-98b0-95b642020499"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:48:02.950617 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:02.950396 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de285ba3-daf9-4154-98b0-95b642020499-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de285ba3-daf9-4154-98b0-95b642020499-workload-socket\") pod \"de285ba3-daf9-4154-98b0-95b642020499\" (UID: \"de285ba3-daf9-4154-98b0-95b642020499\") " Oct 30 14:48:02.950617 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:02.950434 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de285ba3-daf9-4154-98b0-95b642020499-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de285ba3-daf9-4154-98b0-95b642020499-istio-data\") pod \"de285ba3-daf9-4154-98b0-95b642020499\" (UID: \"de285ba3-daf9-4154-98b0-95b642020499\") " Oct 30 14:48:02.950617 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:02.950540 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de285ba3-daf9-4154-98b0-95b642020499-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de285ba3-daf9-4154-98b0-95b642020499-istio-envoy\") pod \"de285ba3-daf9-4154-98b0-95b642020499\" (UID: \"de285ba3-daf9-4154-98b0-95b642020499\") " Oct 30 14:48:02.950940 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:02.950631 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de285ba3-daf9-4154-98b0-95b642020499-kube-api-access-zkxzb\" (UniqueName: \"kubernetes.io/projected/de285ba3-daf9-4154-98b0-95b642020499-kube-api-access-zkxzb\") pod \"de285ba3-daf9-4154-98b0-95b642020499\" (UID: \"de285ba3-daf9-4154-98b0-95b642020499\") " Oct 30 14:48:02.950940 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:02.950890 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de285ba3-daf9-4154-98b0-95b642020499-workload-socket" pod "de285ba3-daf9-4154-98b0-95b642020499" (UID: "de285ba3-daf9-4154-98b0-95b642020499"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:48:02.951139 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:02.951109 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de285ba3-daf9-4154-98b0-95b642020499-istio-data" pod "de285ba3-daf9-4154-98b0-95b642020499" (UID: "de285ba3-daf9-4154-98b0-95b642020499"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:48:02.951208 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:02.951157 4982 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de285ba3-daf9-4154-98b0-95b642020499-credential-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:02.951208 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:02.951174 4982 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de285ba3-daf9-4154-98b0-95b642020499-workload-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:02.951332 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:02.951199 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de285ba3-daf9-4154-98b0-95b642020499-istiod-ca-cert" pod "de285ba3-daf9-4154-98b0-95b642020499" (UID: "de285ba3-daf9-4154-98b0-95b642020499"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:48:02.951389 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:02.951359 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de285ba3-daf9-4154-98b0-95b642020499-workload-certs" pod "de285ba3-daf9-4154-98b0-95b642020499" (UID: "de285ba3-daf9-4154-98b0-95b642020499"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:48:02.955490 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:02.955440 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de285ba3-daf9-4154-98b0-95b642020499-istio-podinfo" pod "de285ba3-daf9-4154-98b0-95b642020499" (UID: "de285ba3-daf9-4154-98b0-95b642020499"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:48:02.955800 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:02.955764 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de285ba3-daf9-4154-98b0-95b642020499-istio-envoy" pod "de285ba3-daf9-4154-98b0-95b642020499" (UID: "de285ba3-daf9-4154-98b0-95b642020499"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:48:02.956097 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:02.956073 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de285ba3-daf9-4154-98b0-95b642020499-istio-token" pod "de285ba3-daf9-4154-98b0-95b642020499" (UID: "de285ba3-daf9-4154-98b0-95b642020499"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:48:02.956198 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:02.956109 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de285ba3-daf9-4154-98b0-95b642020499-kube-api-access-zkxzb" pod "de285ba3-daf9-4154-98b0-95b642020499" (UID: "de285ba3-daf9-4154-98b0-95b642020499"). InnerVolumeSpecName "kube-api-access-zkxzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:48:02.990530 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:02.990506 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:03.052544 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:03.052512 4982 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de285ba3-daf9-4154-98b0-95b642020499-istiod-ca-cert\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:03.052544 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:03.052545 4982 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de285ba3-daf9-4154-98b0-95b642020499-workload-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:03.052748 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:03.052559 4982 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de285ba3-daf9-4154-98b0-95b642020499-istio-token\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:03.052748 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:03.052572 4982 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de285ba3-daf9-4154-98b0-95b642020499-istio-podinfo\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:03.052748 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:03.052584 4982 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de285ba3-daf9-4154-98b0-95b642020499-istio-data\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:03.052748 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:03.052595 4982 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de285ba3-daf9-4154-98b0-95b642020499-istio-envoy\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:03.052748 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:03.052607 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkxzb\" (UniqueName: \"kubernetes.io/projected/de285ba3-daf9-4154-98b0-95b642020499-kube-api-access-zkxzb\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:03.409362 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:03.409322 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:03.807055 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:03.807025 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4" Oct 30 14:48:03.807556 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:03.807058 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4" event={"ID":"de285ba3-daf9-4154-98b0-95b642020499","Type":"ContainerDied","Data":"c75d4e0aa30d8ca859e34c3a2aa1676dc6a35a9a2e5e5f5d8d6a27e12c086ad1"} Oct 30 14:48:03.807556 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:03.807126 4982 scope.go:122] "RemoveContainer" containerID="72bbd3796327d2a8db0fa250d6d3f36b2d26da2f5b2e56bc84458ad2137dae23" Oct 30 14:48:03.884947 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:03.884915 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4"] Oct 30 14:48:03.897914 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:03.897884 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-828n4"] Oct 30 14:48:03.965838 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:03.965812 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:04.044333 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:04.044291 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-da78c4-87kkb"] Oct 30 14:48:04.044919 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:04.044902 4982 cpu_manager.go:414] "containerMap: removing container" podUID="de285ba3-daf9-4154-98b0-95b642020499" containerName="istio-proxy" Oct 30 14:48:04.044977 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:04.044923 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de285ba3-daf9-4154-98b0-95b642020499" containerName="istio-proxy" Oct 30 14:48:04.044977 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:04.044941 4982 cpu_manager.go:414] "containerMap: removing container" podUID="12350227-cab2-45e5-aadc-f65f4e48bca6" containerName="revoke-keys" Oct 30 14:48:04.044977 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:04.044948 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12350227-cab2-45e5-aadc-f65f4e48bca6" containerName="revoke-keys" Oct 30 14:48:04.045078 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:04.045031 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="12350227-cab2-45e5-aadc-f65f4e48bca6" containerName="revoke-keys" Oct 30 14:48:04.045078 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:04.045048 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="de285ba3-daf9-4154-98b0-95b642020499" containerName="istio-proxy" Oct 30 14:48:04.049609 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:04.049592 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-da78c4-87kkb" Oct 30 14:48:04.070217 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:04.070159 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:48:04.070429 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:04.070410 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:48:04.077424 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:04.077401 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-da78c4-87kkb"] Oct 30 14:48:04.169049 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:04.169023 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaecf278-2542-4af3-b82e-233e83c24e47-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-da78c4-87kkb\" (UID: \"aaecf278-2542-4af3-b82e-233e83c24e47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-da78c4-87kkb" Oct 30 14:48:04.169188 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:04.169153 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svrgf\" (UniqueName: \"kubernetes.io/projected/aaecf278-2542-4af3-b82e-233e83c24e47-kube-api-access-svrgf\") pod \"maas-api-revoke-keys-e2e-worker-w2-da78c4-87kkb\" (UID: \"aaecf278-2542-4af3-b82e-233e83c24e47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-da78c4-87kkb" Oct 30 14:48:04.169320 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:04.169292 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaecf278-2542-4af3-b82e-233e83c24e47-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-da78c4-87kkb\" (UID: \"aaecf278-2542-4af3-b82e-233e83c24e47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-da78c4-87kkb" Oct 30 14:48:04.270796 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:04.270765 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaecf278-2542-4af3-b82e-233e83c24e47-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-da78c4-87kkb\" (UID: \"aaecf278-2542-4af3-b82e-233e83c24e47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-da78c4-87kkb" Oct 30 14:48:04.270943 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:04.270827 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaecf278-2542-4af3-b82e-233e83c24e47-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-da78c4-87kkb\" (UID: \"aaecf278-2542-4af3-b82e-233e83c24e47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-da78c4-87kkb" Oct 30 14:48:04.270943 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:04.270913 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svrgf\" (UniqueName: \"kubernetes.io/projected/aaecf278-2542-4af3-b82e-233e83c24e47-kube-api-access-svrgf\") pod \"maas-api-revoke-keys-e2e-worker-w2-da78c4-87kkb\" (UID: \"aaecf278-2542-4af3-b82e-233e83c24e47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-da78c4-87kkb" Oct 30 14:48:04.271580 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:04.271556 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaecf278-2542-4af3-b82e-233e83c24e47-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-da78c4-87kkb\" (UID: \"aaecf278-2542-4af3-b82e-233e83c24e47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-da78c4-87kkb" Oct 30 14:48:04.273095 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:04.273073 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaecf278-2542-4af3-b82e-233e83c24e47-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-da78c4-87kkb\" (UID: \"aaecf278-2542-4af3-b82e-233e83c24e47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-da78c4-87kkb" Oct 30 14:48:04.301148 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:04.301121 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svrgf\" (UniqueName: \"kubernetes.io/projected/aaecf278-2542-4af3-b82e-233e83c24e47-kube-api-access-svrgf\") pod \"maas-api-revoke-keys-e2e-worker-w2-da78c4-87kkb\" (UID: \"aaecf278-2542-4af3-b82e-233e83c24e47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-da78c4-87kkb" Oct 30 14:48:04.363492 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:04.363436 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-da78c4-87kkb" Oct 30 14:48:04.403476 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:04.403443 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de285ba3-daf9-4154-98b0-95b642020499" path="/var/lib/kubelet/pods/de285ba3-daf9-4154-98b0-95b642020499/volumes" Oct 30 14:48:04.679199 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:04.679169 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-da78c4-87kkb"] Oct 30 14:48:04.680899 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:48:04.680870 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaecf278_2542_4af3_b82e_233e83c24e47.slice/crio-a88ccdb7266b9c5165d6f6712155bbe293f38ee214145d2ff5afee3139a382b8 WatchSource:0}: Error finding container a88ccdb7266b9c5165d6f6712155bbe293f38ee214145d2ff5afee3139a382b8: Status 404 returned error can't find the container with id a88ccdb7266b9c5165d6f6712155bbe293f38ee214145d2ff5afee3139a382b8 Oct 30 14:48:04.814902 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:04.814867 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-da78c4-87kkb" event={"ID":"aaecf278-2542-4af3-b82e-233e83c24e47","Type":"ContainerStarted","Data":"a88ccdb7266b9c5165d6f6712155bbe293f38ee214145d2ff5afee3139a382b8"} Oct 30 14:48:05.113943 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:05.113906 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:05.824721 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:05.824690 4982 generic.go:356] "Generic (PLEG): container finished" podID="aaecf278-2542-4af3-b82e-233e83c24e47" containerID="d3c2119525ebc3e3c977a5f4c7b35033eda59f343c52dfeeee616111597afe89" exitCode=0 Oct 30 14:48:05.825120 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:05.824775 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-da78c4-87kkb" event={"ID":"aaecf278-2542-4af3-b82e-233e83c24e47","Type":"ContainerDied","Data":"d3c2119525ebc3e3c977a5f4c7b35033eda59f343c52dfeeee616111597afe89"} Oct 30 14:48:06.981781 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:06.981754 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-da78c4-87kkb" Oct 30 14:48:07.109205 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:07.109133 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaecf278-2542-4af3-b82e-233e83c24e47-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaecf278-2542-4af3-b82e-233e83c24e47-maas-api-openshift-ca\") pod \"aaecf278-2542-4af3-b82e-233e83c24e47\" (UID: \"aaecf278-2542-4af3-b82e-233e83c24e47\") " Oct 30 14:48:07.109205 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:07.109196 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaecf278-2542-4af3-b82e-233e83c24e47-kube-api-access-svrgf\" (UniqueName: \"kubernetes.io/projected/aaecf278-2542-4af3-b82e-233e83c24e47-kube-api-access-svrgf\") pod \"aaecf278-2542-4af3-b82e-233e83c24e47\" (UID: \"aaecf278-2542-4af3-b82e-233e83c24e47\") " Oct 30 14:48:07.109383 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:07.109322 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aaecf278-2542-4af3-b82e-233e83c24e47-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaecf278-2542-4af3-b82e-233e83c24e47-maas-api-xks-ca\") pod \"aaecf278-2542-4af3-b82e-233e83c24e47\" (UID: \"aaecf278-2542-4af3-b82e-233e83c24e47\") " Oct 30 14:48:07.109493 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:07.109469 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaecf278-2542-4af3-b82e-233e83c24e47-maas-api-openshift-ca" pod "aaecf278-2542-4af3-b82e-233e83c24e47" (UID: "aaecf278-2542-4af3-b82e-233e83c24e47"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:48:07.109774 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:07.109755 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaecf278-2542-4af3-b82e-233e83c24e47-maas-api-openshift-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:07.111356 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:07.111335 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaecf278-2542-4af3-b82e-233e83c24e47-kube-api-access-svrgf" pod "aaecf278-2542-4af3-b82e-233e83c24e47" (UID: "aaecf278-2542-4af3-b82e-233e83c24e47"). InnerVolumeSpecName "kube-api-access-svrgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:48:07.111427 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:07.111356 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aaecf278-2542-4af3-b82e-233e83c24e47-maas-api-xks-ca" pod "aaecf278-2542-4af3-b82e-233e83c24e47" (UID: "aaecf278-2542-4af3-b82e-233e83c24e47"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:48:07.210496 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:07.210468 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaecf278-2542-4af3-b82e-233e83c24e47-maas-api-xks-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:07.210496 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:07.210495 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svrgf\" (UniqueName: \"kubernetes.io/projected/aaecf278-2542-4af3-b82e-233e83c24e47-kube-api-access-svrgf\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:07.841461 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:07.841429 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-da78c4-87kkb" Oct 30 14:48:07.841669 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:07.841428 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-da78c4-87kkb" event={"ID":"aaecf278-2542-4af3-b82e-233e83c24e47","Type":"ContainerDied","Data":"a88ccdb7266b9c5165d6f6712155bbe293f38ee214145d2ff5afee3139a382b8"} Oct 30 14:48:07.841669 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:07.841561 4982 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a88ccdb7266b9c5165d6f6712155bbe293f38ee214145d2ff5afee3139a382b8" Oct 30 14:48:07.958819 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:07.958776 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:08.872882 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:08.872850 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf"] Oct 30 14:48:08.948865 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:08.948833 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9"] Oct 30 14:48:09.066313 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:09.066232 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:09.927925 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:09.927894 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:11.622742 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:11.622705 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:11.743955 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:11.743919 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d05e21-85db5c75d6-kg8b9"] Oct 30 14:48:11.744176 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:11.744154 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-d05e21-85db5c75d6-kg8b9" podUID="02808c74-feb9-4fe4-8fe0-9f040dd3e8a1" containerName="payload-processing" containerID="cri-o://f7712b3365b33a5b4e5c83f11dc549cafc56ef63c1c5f653719a5785624c4412" gracePeriod=30 Oct 30 14:48:11.861009 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:11.860974 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:11.870067 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:11.870033 4982 generic.go:356] "Generic (PLEG): container finished" podID="02808c74-feb9-4fe4-8fe0-9f040dd3e8a1" containerID="f7712b3365b33a5b4e5c83f11dc549cafc56ef63c1c5f653719a5785624c4412" exitCode=0 Oct 30 14:48:11.870193 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:11.870068 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d05e21-85db5c75d6-kg8b9" event={"ID":"02808c74-feb9-4fe4-8fe0-9f040dd3e8a1","Type":"ContainerDied","Data":"f7712b3365b33a5b4e5c83f11dc549cafc56ef63c1c5f653719a5785624c4412"} Oct 30 14:48:12.021045 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:12.021010 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:12.042969 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:12.042947 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d05e21-85db5c75d6-kg8b9" Oct 30 14:48:12.067059 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:12.067022 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02808c74-feb9-4fe4-8fe0-9f040dd3e8a1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02808c74-feb9-4fe4-8fe0-9f040dd3e8a1-plugins-config-volume\") pod \"02808c74-feb9-4fe4-8fe0-9f040dd3e8a1\" (UID: \"02808c74-feb9-4fe4-8fe0-9f040dd3e8a1\") " Oct 30 14:48:12.067226 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:12.067177 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02808c74-feb9-4fe4-8fe0-9f040dd3e8a1-kube-api-access-6b9m4\" (UniqueName: \"kubernetes.io/projected/02808c74-feb9-4fe4-8fe0-9f040dd3e8a1-kube-api-access-6b9m4\") pod \"02808c74-feb9-4fe4-8fe0-9f040dd3e8a1\" (UID: \"02808c74-feb9-4fe4-8fe0-9f040dd3e8a1\") " Oct 30 14:48:12.067463 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:12.067437 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02808c74-feb9-4fe4-8fe0-9f040dd3e8a1-plugins-config-volume" pod "02808c74-feb9-4fe4-8fe0-9f040dd3e8a1" (UID: "02808c74-feb9-4fe4-8fe0-9f040dd3e8a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:48:12.067591 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:12.067577 4982 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02808c74-feb9-4fe4-8fe0-9f040dd3e8a1-plugins-config-volume\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:12.069594 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:12.069563 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02808c74-feb9-4fe4-8fe0-9f040dd3e8a1-kube-api-access-6b9m4" pod "02808c74-feb9-4fe4-8fe0-9f040dd3e8a1" (UID: "02808c74-feb9-4fe4-8fe0-9f040dd3e8a1"). InnerVolumeSpecName "kube-api-access-6b9m4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:48:12.168919 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:12.168877 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6b9m4\" (UniqueName: \"kubernetes.io/projected/02808c74-feb9-4fe4-8fe0-9f040dd3e8a1-kube-api-access-6b9m4\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:12.222342 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:12.222246 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-f0e8b9/ipp-route-f0e8b9-kserve-7f4f866ccc-pz7w2" podUID="02251d15-f79c-41a6-bd91-6ec5969e1f63" containerName="main" containerID="cri-o://c9868e75fcb2c0122465e1018e29d33fea5f6ce793c786e9bebda6752d728fd3" gracePeriod=45 Oct 30 14:48:12.269862 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:12.269831 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:12.431959 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:12.431927 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:12.519903 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:12.519874 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-f0e8b9/ipp-route-f0e8b9-kserve-7f4f866ccc-pz7w2" Oct 30 14:48:12.573431 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:12.573400 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02251d15-f79c-41a6-bd91-6ec5969e1f63-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02251d15-f79c-41a6-bd91-6ec5969e1f63-model-cache\") pod \"02251d15-f79c-41a6-bd91-6ec5969e1f63\" (UID: \"02251d15-f79c-41a6-bd91-6ec5969e1f63\") " Oct 30 14:48:12.573576 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:12.573487 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02251d15-f79c-41a6-bd91-6ec5969e1f63-home\" (UniqueName: \"kubernetes.io/empty-dir/02251d15-f79c-41a6-bd91-6ec5969e1f63-home\") pod \"02251d15-f79c-41a6-bd91-6ec5969e1f63\" (UID: \"02251d15-f79c-41a6-bd91-6ec5969e1f63\") " Oct 30 14:48:12.573576 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:12.573522 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02251d15-f79c-41a6-bd91-6ec5969e1f63-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02251d15-f79c-41a6-bd91-6ec5969e1f63-tmp-dir\") pod \"02251d15-f79c-41a6-bd91-6ec5969e1f63\" (UID: \"02251d15-f79c-41a6-bd91-6ec5969e1f63\") " Oct 30 14:48:12.573712 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:12.573616 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02251d15-f79c-41a6-bd91-6ec5969e1f63-kube-api-access-8szkf\" (UniqueName: \"kubernetes.io/projected/02251d15-f79c-41a6-bd91-6ec5969e1f63-kube-api-access-8szkf\") pod \"02251d15-f79c-41a6-bd91-6ec5969e1f63\" (UID: \"02251d15-f79c-41a6-bd91-6ec5969e1f63\") " Oct 30 14:48:12.573712 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:12.573695 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02251d15-f79c-41a6-bd91-6ec5969e1f63-tls-certs\" (UniqueName: \"kubernetes.io/secret/02251d15-f79c-41a6-bd91-6ec5969e1f63-tls-certs\") pod \"02251d15-f79c-41a6-bd91-6ec5969e1f63\" (UID: \"02251d15-f79c-41a6-bd91-6ec5969e1f63\") " Oct 30 14:48:12.573813 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:12.573705 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02251d15-f79c-41a6-bd91-6ec5969e1f63-model-cache" pod "02251d15-f79c-41a6-bd91-6ec5969e1f63" (UID: "02251d15-f79c-41a6-bd91-6ec5969e1f63"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:48:12.573813 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:12.573751 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02251d15-f79c-41a6-bd91-6ec5969e1f63-dshm\" (UniqueName: \"kubernetes.io/empty-dir/02251d15-f79c-41a6-bd91-6ec5969e1f63-dshm\") pod \"02251d15-f79c-41a6-bd91-6ec5969e1f63\" (UID: \"02251d15-f79c-41a6-bd91-6ec5969e1f63\") " Oct 30 14:48:12.573813 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:12.573770 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02251d15-f79c-41a6-bd91-6ec5969e1f63-home" pod "02251d15-f79c-41a6-bd91-6ec5969e1f63" (UID: "02251d15-f79c-41a6-bd91-6ec5969e1f63"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:48:12.574166 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:12.574144 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02251d15-f79c-41a6-bd91-6ec5969e1f63-tmp-dir" pod "02251d15-f79c-41a6-bd91-6ec5969e1f63" (UID: "02251d15-f79c-41a6-bd91-6ec5969e1f63"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:48:12.574295 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:12.574285 4982 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02251d15-f79c-41a6-bd91-6ec5969e1f63-home\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:12.574374 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:12.574303 4982 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02251d15-f79c-41a6-bd91-6ec5969e1f63-tmp-dir\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:12.574374 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:12.574318 4982 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02251d15-f79c-41a6-bd91-6ec5969e1f63-model-cache\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:12.576598 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:12.576568 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02251d15-f79c-41a6-bd91-6ec5969e1f63-tls-certs" pod "02251d15-f79c-41a6-bd91-6ec5969e1f63" (UID: "02251d15-f79c-41a6-bd91-6ec5969e1f63"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:48:12.576598 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:12.576585 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02251d15-f79c-41a6-bd91-6ec5969e1f63-dshm" pod "02251d15-f79c-41a6-bd91-6ec5969e1f63" (UID: "02251d15-f79c-41a6-bd91-6ec5969e1f63"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:48:12.576952 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:12.576926 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02251d15-f79c-41a6-bd91-6ec5969e1f63-kube-api-access-8szkf" pod "02251d15-f79c-41a6-bd91-6ec5969e1f63" (UID: "02251d15-f79c-41a6-bd91-6ec5969e1f63"). InnerVolumeSpecName "kube-api-access-8szkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:48:12.675852 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:12.675810 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8szkf\" (UniqueName: \"kubernetes.io/projected/02251d15-f79c-41a6-bd91-6ec5969e1f63-kube-api-access-8szkf\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:12.675852 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:12.675849 4982 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02251d15-f79c-41a6-bd91-6ec5969e1f63-tls-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:12.675852 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:12.675859 4982 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02251d15-f79c-41a6-bd91-6ec5969e1f63-dshm\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:12.878465 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:12.878437 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d05e21-85db5c75d6-kg8b9" Oct 30 14:48:12.878635 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:12.878469 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d05e21-85db5c75d6-kg8b9" event={"ID":"02808c74-feb9-4fe4-8fe0-9f040dd3e8a1","Type":"ContainerDied","Data":"d22669199c714624e37124797cc10d78e5e4295ab166ad1116ac24134b07c67e"} Oct 30 14:48:12.878635 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:12.878526 4982 scope.go:122] "RemoveContainer" containerID="f7712b3365b33a5b4e5c83f11dc549cafc56ef63c1c5f653719a5785624c4412" Oct 30 14:48:12.879986 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:12.879962 4982 generic.go:356] "Generic (PLEG): container finished" podID="02251d15-f79c-41a6-bd91-6ec5969e1f63" containerID="c9868e75fcb2c0122465e1018e29d33fea5f6ce793c786e9bebda6752d728fd3" exitCode=0 Oct 30 14:48:12.880136 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:12.880021 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-f0e8b9/ipp-route-f0e8b9-kserve-7f4f866ccc-pz7w2" Oct 30 14:48:12.880136 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:12.880031 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-f0e8b9/ipp-route-f0e8b9-kserve-7f4f866ccc-pz7w2" event={"ID":"02251d15-f79c-41a6-bd91-6ec5969e1f63","Type":"ContainerDied","Data":"c9868e75fcb2c0122465e1018e29d33fea5f6ce793c786e9bebda6752d728fd3"} Oct 30 14:48:12.880252 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:12.880134 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-f0e8b9/ipp-route-f0e8b9-kserve-7f4f866ccc-pz7w2" event={"ID":"02251d15-f79c-41a6-bd91-6ec5969e1f63","Type":"ContainerDied","Data":"43559eb3b91b25bbb372c292d0169796798607c38c6a77eeeed0541c9dc60529"} Oct 30 14:48:12.891434 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:12.891417 4982 scope.go:122] "RemoveContainer" containerID="c9868e75fcb2c0122465e1018e29d33fea5f6ce793c786e9bebda6752d728fd3" Oct 30 14:48:12.901818 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:12.901796 4982 scope.go:122] "RemoveContainer" containerID="c9868e75fcb2c0122465e1018e29d33fea5f6ce793c786e9bebda6752d728fd3" Oct 30 14:48:12.902105 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:48:12.902083 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9868e75fcb2c0122465e1018e29d33fea5f6ce793c786e9bebda6752d728fd3\": container with ID starting with c9868e75fcb2c0122465e1018e29d33fea5f6ce793c786e9bebda6752d728fd3 not found: ID does not exist" containerID="c9868e75fcb2c0122465e1018e29d33fea5f6ce793c786e9bebda6752d728fd3" Oct 30 14:48:12.902189 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:12.902121 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9868e75fcb2c0122465e1018e29d33fea5f6ce793c786e9bebda6752d728fd3"} err="failed to get container status \"c9868e75fcb2c0122465e1018e29d33fea5f6ce793c786e9bebda6752d728fd3\": rpc error: code = NotFound desc = could not find container \"c9868e75fcb2c0122465e1018e29d33fea5f6ce793c786e9bebda6752d728fd3\": container with ID starting with c9868e75fcb2c0122465e1018e29d33fea5f6ce793c786e9bebda6752d728fd3 not found: ID does not exist" Oct 30 14:48:12.944036 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:12.944013 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-f0e8b9/ipp-route-f0e8b9-kserve-7f4f866ccc-pz7w2"] Oct 30 14:48:12.947176 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:12.947155 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-f0e8b9/ipp-route-f0e8b9-kserve-7f4f866ccc-pz7w2"] Oct 30 14:48:12.989054 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:12.989028 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d05e21-85db5c75d6-kg8b9"] Oct 30 14:48:12.995528 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:12.995497 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d05e21-85db5c75d6-kg8b9"] Oct 30 14:48:13.059147 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:13.059126 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:14.400092 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:14.400055 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02251d15-f79c-41a6-bd91-6ec5969e1f63" path="/var/lib/kubelet/pods/02251d15-f79c-41a6-bd91-6ec5969e1f63/volumes" Oct 30 14:48:14.400516 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:14.400500 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02808c74-feb9-4fe4-8fe0-9f040dd3e8a1" path="/var/lib/kubelet/pods/02808c74-feb9-4fe4-8fe0-9f040dd3e8a1/volumes" Oct 30 14:48:14.562822 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:14.562791 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:14.705807 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:14.705733 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:14.860858 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:14.860823 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:16.613767 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:16.613734 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:18.478831 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:18.478800 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:20.102790 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:20.102748 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:20.232273 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:20.232230 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2927fd-58947b47f4-wh6th"] Oct 30 14:48:20.232514 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:20.232486 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-2927fd-58947b47f4-wh6th" podUID="9703a518-851b-4a8e-9e2c-586ea7fff550" containerName="payload-processing" containerID="cri-o://60f3daab65e9ee43cd9997931d8699107a54e5a08eee0d335eb2dfa872f82684" gracePeriod=30 Oct 30 14:48:20.280460 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:20.277369 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2927fd-5cfc5f6ff9-p2hhp"] Oct 30 14:48:20.280460 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:20.277608 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2927fd-5cfc5f6ff9-p2hhp" podUID="543506fd-c659-42d5-8338-8fdc25422ecb" containerName="payload-pre-processing" containerID="cri-o://f1341bdf8345d08a3aed63ca42a6f3847be39f0caae14f641939b1ba05fe3570" gracePeriod=30 Oct 30 14:48:20.281530 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:20.281495 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:20.437556 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:20.437531 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp"] Oct 30 14:48:20.437991 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:20.437957 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp" podUID="b5d8a300-e5ec-462a-95b3-d6f1e220dc42" containerName="istio-proxy" containerID="cri-o://7da8182ed5d2a327a9f2305f9bf341b2691c4e38e16ea0b63bc4d79805f7e55a" gracePeriod=30 Oct 30 14:48:20.440173 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:20.440148 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt"] Oct 30 14:48:20.440413 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:20.440388 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt" podUID="393471d3-e435-4bf0-835d-8e4c33ae17a7" containerName="istio-proxy" containerID="cri-o://ba8ce946fc23f4ae69b314fa1433006e0f0dad111e03d0225e26bee922319aaa" gracePeriod=30 Oct 30 14:48:20.513009 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:20.512913 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:20.675187 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:20.675163 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2927fd-58947b47f4-wh6th" Oct 30 14:48:20.724426 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:20.724398 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2927fd-5cfc5f6ff9-p2hhp" Oct 30 14:48:20.728251 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:20.728206 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:20.779801 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:20.779773 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9703a518-851b-4a8e-9e2c-586ea7fff550-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9703a518-851b-4a8e-9e2c-586ea7fff550-plugins-config-volume\") pod \"9703a518-851b-4a8e-9e2c-586ea7fff550\" (UID: \"9703a518-851b-4a8e-9e2c-586ea7fff550\") " Oct 30 14:48:20.779951 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:20.779846 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9703a518-851b-4a8e-9e2c-586ea7fff550-kube-api-access-2vj28\" (UniqueName: \"kubernetes.io/projected/9703a518-851b-4a8e-9e2c-586ea7fff550-kube-api-access-2vj28\") pod \"9703a518-851b-4a8e-9e2c-586ea7fff550\" (UID: \"9703a518-851b-4a8e-9e2c-586ea7fff550\") " Oct 30 14:48:20.780198 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:20.780169 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9703a518-851b-4a8e-9e2c-586ea7fff550-plugins-config-volume" pod "9703a518-851b-4a8e-9e2c-586ea7fff550" (UID: "9703a518-851b-4a8e-9e2c-586ea7fff550"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:48:20.780415 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:20.780395 4982 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9703a518-851b-4a8e-9e2c-586ea7fff550-plugins-config-volume\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:20.781977 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:20.781952 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9703a518-851b-4a8e-9e2c-586ea7fff550-kube-api-access-2vj28" pod "9703a518-851b-4a8e-9e2c-586ea7fff550" (UID: "9703a518-851b-4a8e-9e2c-586ea7fff550"). InnerVolumeSpecName "kube-api-access-2vj28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:48:20.879848 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:20.879818 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:20.880811 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:20.880790 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/543506fd-c659-42d5-8338-8fdc25422ecb-kube-api-access-mjcg5\" (UniqueName: \"kubernetes.io/projected/543506fd-c659-42d5-8338-8fdc25422ecb-kube-api-access-mjcg5\") pod \"543506fd-c659-42d5-8338-8fdc25422ecb\" (UID: \"543506fd-c659-42d5-8338-8fdc25422ecb\") " Oct 30 14:48:20.880919 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:20.880908 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/543506fd-c659-42d5-8338-8fdc25422ecb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/543506fd-c659-42d5-8338-8fdc25422ecb-plugins-config-volume\") pod \"543506fd-c659-42d5-8338-8fdc25422ecb\" (UID: \"543506fd-c659-42d5-8338-8fdc25422ecb\") " Oct 30 14:48:20.881190 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:20.881178 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vj28\" (UniqueName: \"kubernetes.io/projected/9703a518-851b-4a8e-9e2c-586ea7fff550-kube-api-access-2vj28\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:20.881281 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:20.881254 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/543506fd-c659-42d5-8338-8fdc25422ecb-plugins-config-volume" pod "543506fd-c659-42d5-8338-8fdc25422ecb" (UID: "543506fd-c659-42d5-8338-8fdc25422ecb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:48:20.883035 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:20.883011 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/543506fd-c659-42d5-8338-8fdc25422ecb-kube-api-access-mjcg5" pod "543506fd-c659-42d5-8338-8fdc25422ecb" (UID: "543506fd-c659-42d5-8338-8fdc25422ecb"). InnerVolumeSpecName "kube-api-access-mjcg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:48:20.937090 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:20.937016 4982 generic.go:356] "Generic (PLEG): container finished" podID="543506fd-c659-42d5-8338-8fdc25422ecb" containerID="f1341bdf8345d08a3aed63ca42a6f3847be39f0caae14f641939b1ba05fe3570" exitCode=0 Oct 30 14:48:20.937090 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:20.937075 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2927fd-5cfc5f6ff9-p2hhp" Oct 30 14:48:20.937277 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:20.937107 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2927fd-5cfc5f6ff9-p2hhp" event={"ID":"543506fd-c659-42d5-8338-8fdc25422ecb","Type":"ContainerDied","Data":"f1341bdf8345d08a3aed63ca42a6f3847be39f0caae14f641939b1ba05fe3570"} Oct 30 14:48:20.937277 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:20.937189 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2927fd-5cfc5f6ff9-p2hhp" event={"ID":"543506fd-c659-42d5-8338-8fdc25422ecb","Type":"ContainerDied","Data":"99937d7286c0fc6c6e2b480727d966285cfb86cfba8a90827f38f897f39f43e7"} Oct 30 14:48:20.937277 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:20.937222 4982 scope.go:122] "RemoveContainer" containerID="f1341bdf8345d08a3aed63ca42a6f3847be39f0caae14f641939b1ba05fe3570" Oct 30 14:48:20.938664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:20.938613 4982 generic.go:356] "Generic (PLEG): container finished" podID="9703a518-851b-4a8e-9e2c-586ea7fff550" containerID="60f3daab65e9ee43cd9997931d8699107a54e5a08eee0d335eb2dfa872f82684" exitCode=0 Oct 30 14:48:20.938773 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:20.938674 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2927fd-58947b47f4-wh6th" event={"ID":"9703a518-851b-4a8e-9e2c-586ea7fff550","Type":"ContainerDied","Data":"60f3daab65e9ee43cd9997931d8699107a54e5a08eee0d335eb2dfa872f82684"} Oct 30 14:48:20.938773 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:20.938713 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2927fd-58947b47f4-wh6th" Oct 30 14:48:20.938773 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:20.938725 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2927fd-58947b47f4-wh6th" event={"ID":"9703a518-851b-4a8e-9e2c-586ea7fff550","Type":"ContainerDied","Data":"ef186bc5750618d490a73f5700ab880039b6c738d67b888bce67c3cae1c6323f"} Oct 30 14:48:20.948152 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:20.948134 4982 scope.go:122] "RemoveContainer" containerID="f1341bdf8345d08a3aed63ca42a6f3847be39f0caae14f641939b1ba05fe3570" Oct 30 14:48:20.948481 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:48:20.948464 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1341bdf8345d08a3aed63ca42a6f3847be39f0caae14f641939b1ba05fe3570\": container with ID starting with f1341bdf8345d08a3aed63ca42a6f3847be39f0caae14f641939b1ba05fe3570 not found: ID does not exist" containerID="f1341bdf8345d08a3aed63ca42a6f3847be39f0caae14f641939b1ba05fe3570" Oct 30 14:48:20.948546 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:20.948488 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1341bdf8345d08a3aed63ca42a6f3847be39f0caae14f641939b1ba05fe3570"} err="failed to get container status \"f1341bdf8345d08a3aed63ca42a6f3847be39f0caae14f641939b1ba05fe3570\": rpc error: code = NotFound desc = could not find container \"f1341bdf8345d08a3aed63ca42a6f3847be39f0caae14f641939b1ba05fe3570\": container with ID starting with f1341bdf8345d08a3aed63ca42a6f3847be39f0caae14f641939b1ba05fe3570 not found: ID does not exist" Oct 30 14:48:20.948546 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:20.948505 4982 scope.go:122] "RemoveContainer" containerID="60f3daab65e9ee43cd9997931d8699107a54e5a08eee0d335eb2dfa872f82684" Oct 30 14:48:20.957861 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:20.957838 4982 scope.go:122] "RemoveContainer" containerID="60f3daab65e9ee43cd9997931d8699107a54e5a08eee0d335eb2dfa872f82684" Oct 30 14:48:20.958118 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:48:20.958098 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60f3daab65e9ee43cd9997931d8699107a54e5a08eee0d335eb2dfa872f82684\": container with ID starting with 60f3daab65e9ee43cd9997931d8699107a54e5a08eee0d335eb2dfa872f82684 not found: ID does not exist" containerID="60f3daab65e9ee43cd9997931d8699107a54e5a08eee0d335eb2dfa872f82684" Oct 30 14:48:20.958175 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:20.958126 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60f3daab65e9ee43cd9997931d8699107a54e5a08eee0d335eb2dfa872f82684"} err="failed to get container status \"60f3daab65e9ee43cd9997931d8699107a54e5a08eee0d335eb2dfa872f82684\": rpc error: code = NotFound desc = could not find container \"60f3daab65e9ee43cd9997931d8699107a54e5a08eee0d335eb2dfa872f82684\": container with ID starting with 60f3daab65e9ee43cd9997931d8699107a54e5a08eee0d335eb2dfa872f82684 not found: ID does not exist" Oct 30 14:48:20.982353 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:20.982328 4982 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/543506fd-c659-42d5-8338-8fdc25422ecb-plugins-config-volume\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:20.982447 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:20.982357 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjcg5\" (UniqueName: \"kubernetes.io/projected/543506fd-c659-42d5-8338-8fdc25422ecb-kube-api-access-mjcg5\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:20.996204 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:20.996178 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2927fd-5cfc5f6ff9-p2hhp"] Oct 30 14:48:21.002516 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:21.002494 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2927fd-5cfc5f6ff9-p2hhp"] Oct 30 14:48:21.031731 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:21.031692 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:21.043809 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:21.043778 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2927fd-58947b47f4-wh6th"] Oct 30 14:48:21.047265 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:21.047242 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2927fd-58947b47f4-wh6th"] Oct 30 14:48:21.566486 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:21.566445 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:22.398803 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:22.398771 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="543506fd-c659-42d5-8338-8fdc25422ecb" path="/var/lib/kubelet/pods/543506fd-c659-42d5-8338-8fdc25422ecb/volumes" Oct 30 14:48:22.399155 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:22.399142 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9703a518-851b-4a8e-9e2c-586ea7fff550" path="/var/lib/kubelet/pods/9703a518-851b-4a8e-9e2c-586ea7fff550/volumes" Oct 30 14:48:23.158352 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:23.158310 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:23.893572 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:23.893526 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" podUID="00fdf663-9bc9-4349-a565-6a8dc01c0842" containerName="main" containerID="cri-o://be10650cdc5bde0f4a4fe456d284e435181637804d8ba8cbd0f1bfdfcad72f79" gracePeriod=45 Oct 30 14:48:23.907216 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:23.907185 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" podUID="00fdf663-9bc9-4349-a565-6a8dc01c0842" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": dial tcp 10.132.0.68:8000: connect: connection refused" Oct 30 14:48:23.968819 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:23.968786 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" podUID="cc3a3a28-d723-459a-93d0-ff115eac479a" containerName="main" containerID="cri-o://7a58d3b4cf4311a3f1456c31660222e83e69f0f3555a552a5ae5d549a85e8a5b" gracePeriod=45 Oct 30 14:48:24.235089 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:24.235058 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" Oct 30 14:48:24.260852 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:24.260831 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" Oct 30 14:48:24.316667 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:24.316617 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00fdf663-9bc9-4349-a565-6a8dc01c0842-dshm\" (UniqueName: \"kubernetes.io/empty-dir/00fdf663-9bc9-4349-a565-6a8dc01c0842-dshm\") pod \"00fdf663-9bc9-4349-a565-6a8dc01c0842\" (UID: \"00fdf663-9bc9-4349-a565-6a8dc01c0842\") " Oct 30 14:48:24.316872 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:24.316711 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00fdf663-9bc9-4349-a565-6a8dc01c0842-tls-certs\" (UniqueName: \"kubernetes.io/secret/00fdf663-9bc9-4349-a565-6a8dc01c0842-tls-certs\") pod \"00fdf663-9bc9-4349-a565-6a8dc01c0842\" (UID: \"00fdf663-9bc9-4349-a565-6a8dc01c0842\") " Oct 30 14:48:24.316941 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:24.316880 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00fdf663-9bc9-4349-a565-6a8dc01c0842-kube-api-access-bs6km\" (UniqueName: \"kubernetes.io/projected/00fdf663-9bc9-4349-a565-6a8dc01c0842-kube-api-access-bs6km\") pod \"00fdf663-9bc9-4349-a565-6a8dc01c0842\" (UID: \"00fdf663-9bc9-4349-a565-6a8dc01c0842\") " Oct 30 14:48:24.316941 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:24.316909 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00fdf663-9bc9-4349-a565-6a8dc01c0842-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00fdf663-9bc9-4349-a565-6a8dc01c0842-tmp-dir\") pod \"00fdf663-9bc9-4349-a565-6a8dc01c0842\" (UID: \"00fdf663-9bc9-4349-a565-6a8dc01c0842\") " Oct 30 14:48:24.317050 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:24.316939 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00fdf663-9bc9-4349-a565-6a8dc01c0842-home\" (UniqueName: \"kubernetes.io/empty-dir/00fdf663-9bc9-4349-a565-6a8dc01c0842-home\") pod \"00fdf663-9bc9-4349-a565-6a8dc01c0842\" (UID: \"00fdf663-9bc9-4349-a565-6a8dc01c0842\") " Oct 30 14:48:24.317050 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:24.316996 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00fdf663-9bc9-4349-a565-6a8dc01c0842-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00fdf663-9bc9-4349-a565-6a8dc01c0842-model-cache\") pod \"00fdf663-9bc9-4349-a565-6a8dc01c0842\" (UID: \"00fdf663-9bc9-4349-a565-6a8dc01c0842\") " Oct 30 14:48:24.318056 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:24.317786 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00fdf663-9bc9-4349-a565-6a8dc01c0842-tmp-dir" pod "00fdf663-9bc9-4349-a565-6a8dc01c0842" (UID: "00fdf663-9bc9-4349-a565-6a8dc01c0842"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:48:24.318056 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:24.317820 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00fdf663-9bc9-4349-a565-6a8dc01c0842-model-cache" pod "00fdf663-9bc9-4349-a565-6a8dc01c0842" (UID: "00fdf663-9bc9-4349-a565-6a8dc01c0842"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:48:24.318224 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:24.318184 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00fdf663-9bc9-4349-a565-6a8dc01c0842-home" pod "00fdf663-9bc9-4349-a565-6a8dc01c0842" (UID: "00fdf663-9bc9-4349-a565-6a8dc01c0842"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:48:24.320710 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:24.320688 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00fdf663-9bc9-4349-a565-6a8dc01c0842-dshm" pod "00fdf663-9bc9-4349-a565-6a8dc01c0842" (UID: "00fdf663-9bc9-4349-a565-6a8dc01c0842"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:48:24.320838 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:24.320818 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00fdf663-9bc9-4349-a565-6a8dc01c0842-tls-certs" pod "00fdf663-9bc9-4349-a565-6a8dc01c0842" (UID: "00fdf663-9bc9-4349-a565-6a8dc01c0842"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:48:24.320930 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:24.320912 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00fdf663-9bc9-4349-a565-6a8dc01c0842-kube-api-access-bs6km" pod "00fdf663-9bc9-4349-a565-6a8dc01c0842" (UID: "00fdf663-9bc9-4349-a565-6a8dc01c0842"). InnerVolumeSpecName "kube-api-access-bs6km". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:48:24.418333 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:24.418268 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc3a3a28-d723-459a-93d0-ff115eac479a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc3a3a28-d723-459a-93d0-ff115eac479a-tmp-dir\") pod \"cc3a3a28-d723-459a-93d0-ff115eac479a\" (UID: \"cc3a3a28-d723-459a-93d0-ff115eac479a\") " Oct 30 14:48:24.418333 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:24.418311 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc3a3a28-d723-459a-93d0-ff115eac479a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc3a3a28-d723-459a-93d0-ff115eac479a-model-cache\") pod \"cc3a3a28-d723-459a-93d0-ff115eac479a\" (UID: \"cc3a3a28-d723-459a-93d0-ff115eac479a\") " Oct 30 14:48:24.418519 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:24.418344 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc3a3a28-d723-459a-93d0-ff115eac479a-home\" (UniqueName: \"kubernetes.io/empty-dir/cc3a3a28-d723-459a-93d0-ff115eac479a-home\") pod \"cc3a3a28-d723-459a-93d0-ff115eac479a\" (UID: \"cc3a3a28-d723-459a-93d0-ff115eac479a\") " Oct 30 14:48:24.418519 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:24.418454 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc3a3a28-d723-459a-93d0-ff115eac479a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc3a3a28-d723-459a-93d0-ff115eac479a-dshm\") pod \"cc3a3a28-d723-459a-93d0-ff115eac479a\" (UID: \"cc3a3a28-d723-459a-93d0-ff115eac479a\") " Oct 30 14:48:24.418631 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:24.418540 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc3a3a28-d723-459a-93d0-ff115eac479a-tmp-dir" pod "cc3a3a28-d723-459a-93d0-ff115eac479a" (UID: "cc3a3a28-d723-459a-93d0-ff115eac479a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:48:24.418631 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:24.418559 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc3a3a28-d723-459a-93d0-ff115eac479a-tls-certs\" (UniqueName: \"kubernetes.io/secret/cc3a3a28-d723-459a-93d0-ff115eac479a-tls-certs\") pod \"cc3a3a28-d723-459a-93d0-ff115eac479a\" (UID: \"cc3a3a28-d723-459a-93d0-ff115eac479a\") " Oct 30 14:48:24.418631 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:24.418597 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc3a3a28-d723-459a-93d0-ff115eac479a-kube-api-access-sxcj5\" (UniqueName: \"kubernetes.io/projected/cc3a3a28-d723-459a-93d0-ff115eac479a-kube-api-access-sxcj5\") pod \"cc3a3a28-d723-459a-93d0-ff115eac479a\" (UID: \"cc3a3a28-d723-459a-93d0-ff115eac479a\") " Oct 30 14:48:24.418816 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:24.418655 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc3a3a28-d723-459a-93d0-ff115eac479a-model-cache" pod "cc3a3a28-d723-459a-93d0-ff115eac479a" (UID: "cc3a3a28-d723-459a-93d0-ff115eac479a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:48:24.418992 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:24.418965 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc3a3a28-d723-459a-93d0-ff115eac479a-home" pod "cc3a3a28-d723-459a-93d0-ff115eac479a" (UID: "cc3a3a28-d723-459a-93d0-ff115eac479a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:48:24.419111 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:24.419087 4982 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc3a3a28-d723-459a-93d0-ff115eac479a-tmp-dir\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:24.419163 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:24.419111 4982 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc3a3a28-d723-459a-93d0-ff115eac479a-model-cache\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:24.419163 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:24.419125 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bs6km\" (UniqueName: \"kubernetes.io/projected/00fdf663-9bc9-4349-a565-6a8dc01c0842-kube-api-access-bs6km\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:24.419163 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:24.419141 4982 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00fdf663-9bc9-4349-a565-6a8dc01c0842-tmp-dir\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:24.419163 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:24.419156 4982 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc3a3a28-d723-459a-93d0-ff115eac479a-home\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:24.419312 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:24.419168 4982 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00fdf663-9bc9-4349-a565-6a8dc01c0842-home\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:24.419312 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:24.419181 4982 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00fdf663-9bc9-4349-a565-6a8dc01c0842-model-cache\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:24.419312 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:24.419192 4982 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00fdf663-9bc9-4349-a565-6a8dc01c0842-dshm\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:24.419312 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:24.419204 4982 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00fdf663-9bc9-4349-a565-6a8dc01c0842-tls-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:24.420896 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:24.420870 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc3a3a28-d723-459a-93d0-ff115eac479a-tls-certs" pod "cc3a3a28-d723-459a-93d0-ff115eac479a" (UID: "cc3a3a28-d723-459a-93d0-ff115eac479a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:48:24.421017 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:24.420994 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc3a3a28-d723-459a-93d0-ff115eac479a-dshm" pod "cc3a3a28-d723-459a-93d0-ff115eac479a" (UID: "cc3a3a28-d723-459a-93d0-ff115eac479a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:48:24.421102 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:24.421085 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc3a3a28-d723-459a-93d0-ff115eac479a-kube-api-access-sxcj5" pod "cc3a3a28-d723-459a-93d0-ff115eac479a" (UID: "cc3a3a28-d723-459a-93d0-ff115eac479a"). InnerVolumeSpecName "kube-api-access-sxcj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:48:24.520237 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:24.520204 4982 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc3a3a28-d723-459a-93d0-ff115eac479a-dshm\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:24.520237 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:24.520238 4982 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc3a3a28-d723-459a-93d0-ff115eac479a-tls-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:24.520471 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:24.520253 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxcj5\" (UniqueName: \"kubernetes.io/projected/cc3a3a28-d723-459a-93d0-ff115eac479a-kube-api-access-sxcj5\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:24.967257 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:24.967220 4982 generic.go:356] "Generic (PLEG): container finished" podID="00fdf663-9bc9-4349-a565-6a8dc01c0842" containerID="be10650cdc5bde0f4a4fe456d284e435181637804d8ba8cbd0f1bfdfcad72f79" exitCode=0 Oct 30 14:48:24.967418 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:24.967261 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" Oct 30 14:48:24.967418 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:24.967292 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" event={"ID":"00fdf663-9bc9-4349-a565-6a8dc01c0842","Type":"ContainerDied","Data":"be10650cdc5bde0f4a4fe456d284e435181637804d8ba8cbd0f1bfdfcad72f79"} Oct 30 14:48:24.967418 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:24.967334 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf" event={"ID":"00fdf663-9bc9-4349-a565-6a8dc01c0842","Type":"ContainerDied","Data":"84639024ad5ef879eb97ecd486b3c3eda580e47e54b45acd81d71781efb54a0a"} Oct 30 14:48:24.967418 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:24.967356 4982 scope.go:122] "RemoveContainer" containerID="be10650cdc5bde0f4a4fe456d284e435181637804d8ba8cbd0f1bfdfcad72f79" Oct 30 14:48:24.969120 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:24.969093 4982 generic.go:356] "Generic (PLEG): container finished" podID="cc3a3a28-d723-459a-93d0-ff115eac479a" containerID="7a58d3b4cf4311a3f1456c31660222e83e69f0f3555a552a5ae5d549a85e8a5b" exitCode=0 Oct 30 14:48:24.969507 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:24.969147 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" event={"ID":"cc3a3a28-d723-459a-93d0-ff115eac479a","Type":"ContainerDied","Data":"7a58d3b4cf4311a3f1456c31660222e83e69f0f3555a552a5ae5d549a85e8a5b"} Oct 30 14:48:24.969507 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:24.969176 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" Oct 30 14:48:24.969507 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:24.969178 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9" event={"ID":"cc3a3a28-d723-459a-93d0-ff115eac479a","Type":"ContainerDied","Data":"26bb3474e07bc23b7dfd3208b530d26739d9d9d8f95de226cfe3d9b51b56bdef"} Oct 30 14:48:24.977728 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:24.977706 4982 scope.go:122] "RemoveContainer" containerID="be10650cdc5bde0f4a4fe456d284e435181637804d8ba8cbd0f1bfdfcad72f79" Oct 30 14:48:24.977973 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:48:24.977954 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be10650cdc5bde0f4a4fe456d284e435181637804d8ba8cbd0f1bfdfcad72f79\": container with ID starting with be10650cdc5bde0f4a4fe456d284e435181637804d8ba8cbd0f1bfdfcad72f79 not found: ID does not exist" containerID="be10650cdc5bde0f4a4fe456d284e435181637804d8ba8cbd0f1bfdfcad72f79" Oct 30 14:48:24.978039 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:24.977981 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be10650cdc5bde0f4a4fe456d284e435181637804d8ba8cbd0f1bfdfcad72f79"} err="failed to get container status \"be10650cdc5bde0f4a4fe456d284e435181637804d8ba8cbd0f1bfdfcad72f79\": rpc error: code = NotFound desc = could not find container \"be10650cdc5bde0f4a4fe456d284e435181637804d8ba8cbd0f1bfdfcad72f79\": container with ID starting with be10650cdc5bde0f4a4fe456d284e435181637804d8ba8cbd0f1bfdfcad72f79 not found: ID does not exist" Oct 30 14:48:24.978039 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:24.977998 4982 scope.go:122] "RemoveContainer" containerID="7a58d3b4cf4311a3f1456c31660222e83e69f0f3555a552a5ae5d549a85e8a5b" Oct 30 14:48:24.986477 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:24.986460 4982 scope.go:122] "RemoveContainer" containerID="7a58d3b4cf4311a3f1456c31660222e83e69f0f3555a552a5ae5d549a85e8a5b" Oct 30 14:48:24.986826 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:48:24.986800 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a58d3b4cf4311a3f1456c31660222e83e69f0f3555a552a5ae5d549a85e8a5b\": container with ID starting with 7a58d3b4cf4311a3f1456c31660222e83e69f0f3555a552a5ae5d549a85e8a5b not found: ID does not exist" containerID="7a58d3b4cf4311a3f1456c31660222e83e69f0f3555a552a5ae5d549a85e8a5b" Oct 30 14:48:24.986899 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:24.986838 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a58d3b4cf4311a3f1456c31660222e83e69f0f3555a552a5ae5d549a85e8a5b"} err="failed to get container status \"7a58d3b4cf4311a3f1456c31660222e83e69f0f3555a552a5ae5d549a85e8a5b\": rpc error: code = NotFound desc = could not find container \"7a58d3b4cf4311a3f1456c31660222e83e69f0f3555a552a5ae5d549a85e8a5b\": container with ID starting with 7a58d3b4cf4311a3f1456c31660222e83e69f0f3555a552a5ae5d549a85e8a5b not found: ID does not exist" Oct 30 14:48:24.994181 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:24.994144 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:25.030519 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:25.030486 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9"] Oct 30 14:48:25.042524 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:25.042498 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-da78c4/simulator-w2-da78c4-kserve-7d7c5ddfc6-lndb9"] Oct 30 14:48:25.072015 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:25.071991 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf"] Oct 30 14:48:25.081377 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:25.081350 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-da78c4/distinct-2-w2-da78c4-kserve-6cd65c4ff5-tp5rf"] Oct 30 14:48:25.738604 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:25.738579 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt" Oct 30 14:48:25.770429 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:25.770406 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp" Oct 30 14:48:25.835581 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:25.835544 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/393471d3-e435-4bf0-835d-8e4c33ae17a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/393471d3-e435-4bf0-835d-8e4c33ae17a7-istio-podinfo\") pod \"393471d3-e435-4bf0-835d-8e4c33ae17a7\" (UID: \"393471d3-e435-4bf0-835d-8e4c33ae17a7\") " Oct 30 14:48:25.835761 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:25.835693 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/393471d3-e435-4bf0-835d-8e4c33ae17a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/393471d3-e435-4bf0-835d-8e4c33ae17a7-workload-socket\") pod \"393471d3-e435-4bf0-835d-8e4c33ae17a7\" (UID: \"393471d3-e435-4bf0-835d-8e4c33ae17a7\") " Oct 30 14:48:25.835761 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:25.835742 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/393471d3-e435-4bf0-835d-8e4c33ae17a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/393471d3-e435-4bf0-835d-8e4c33ae17a7-istio-data\") pod \"393471d3-e435-4bf0-835d-8e4c33ae17a7\" (UID: \"393471d3-e435-4bf0-835d-8e4c33ae17a7\") " Oct 30 14:48:25.835860 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:25.835778 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/393471d3-e435-4bf0-835d-8e4c33ae17a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/393471d3-e435-4bf0-835d-8e4c33ae17a7-credential-socket\") pod \"393471d3-e435-4bf0-835d-8e4c33ae17a7\" (UID: \"393471d3-e435-4bf0-835d-8e4c33ae17a7\") " Oct 30 14:48:25.835860 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:25.835849 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/393471d3-e435-4bf0-835d-8e4c33ae17a7-kube-api-access-6pswf\" (UniqueName: \"kubernetes.io/projected/393471d3-e435-4bf0-835d-8e4c33ae17a7-kube-api-access-6pswf\") pod \"393471d3-e435-4bf0-835d-8e4c33ae17a7\" (UID: \"393471d3-e435-4bf0-835d-8e4c33ae17a7\") " Oct 30 14:48:25.835963 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:25.835899 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/393471d3-e435-4bf0-835d-8e4c33ae17a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/393471d3-e435-4bf0-835d-8e4c33ae17a7-workload-certs\") pod \"393471d3-e435-4bf0-835d-8e4c33ae17a7\" (UID: \"393471d3-e435-4bf0-835d-8e4c33ae17a7\") " Oct 30 14:48:25.836023 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:25.835965 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/393471d3-e435-4bf0-835d-8e4c33ae17a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/393471d3-e435-4bf0-835d-8e4c33ae17a7-istiod-ca-cert\") pod \"393471d3-e435-4bf0-835d-8e4c33ae17a7\" (UID: \"393471d3-e435-4bf0-835d-8e4c33ae17a7\") " Oct 30 14:48:25.836023 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:25.835973 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/393471d3-e435-4bf0-835d-8e4c33ae17a7-workload-socket" pod "393471d3-e435-4bf0-835d-8e4c33ae17a7" (UID: "393471d3-e435-4bf0-835d-8e4c33ae17a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:48:25.836144 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:25.836026 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/393471d3-e435-4bf0-835d-8e4c33ae17a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/393471d3-e435-4bf0-835d-8e4c33ae17a7-istio-envoy\") pod \"393471d3-e435-4bf0-835d-8e4c33ae17a7\" (UID: \"393471d3-e435-4bf0-835d-8e4c33ae17a7\") " Oct 30 14:48:25.836144 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:25.836057 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/393471d3-e435-4bf0-835d-8e4c33ae17a7-istio-token\" (UniqueName: \"kubernetes.io/projected/393471d3-e435-4bf0-835d-8e4c33ae17a7-istio-token\") pod \"393471d3-e435-4bf0-835d-8e4c33ae17a7\" (UID: \"393471d3-e435-4bf0-835d-8e4c33ae17a7\") " Oct 30 14:48:25.836144 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:25.836078 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/393471d3-e435-4bf0-835d-8e4c33ae17a7-istio-data" pod "393471d3-e435-4bf0-835d-8e4c33ae17a7" (UID: "393471d3-e435-4bf0-835d-8e4c33ae17a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:48:25.836316 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:25.836172 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/393471d3-e435-4bf0-835d-8e4c33ae17a7-workload-certs" pod "393471d3-e435-4bf0-835d-8e4c33ae17a7" (UID: "393471d3-e435-4bf0-835d-8e4c33ae17a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:48:25.836316 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:25.836188 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/393471d3-e435-4bf0-835d-8e4c33ae17a7-credential-socket" pod "393471d3-e435-4bf0-835d-8e4c33ae17a7" (UID: "393471d3-e435-4bf0-835d-8e4c33ae17a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:48:25.836490 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:25.836468 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/393471d3-e435-4bf0-835d-8e4c33ae17a7-istiod-ca-cert" pod "393471d3-e435-4bf0-835d-8e4c33ae17a7" (UID: "393471d3-e435-4bf0-835d-8e4c33ae17a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:48:25.836684 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:25.836545 4982 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/393471d3-e435-4bf0-835d-8e4c33ae17a7-workload-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:25.836828 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:25.836812 4982 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/393471d3-e435-4bf0-835d-8e4c33ae17a7-workload-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:25.836950 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:25.836935 4982 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/393471d3-e435-4bf0-835d-8e4c33ae17a7-istio-data\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:25.837068 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:25.837054 4982 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/393471d3-e435-4bf0-835d-8e4c33ae17a7-credential-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:25.838407 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:25.838385 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/393471d3-e435-4bf0-835d-8e4c33ae17a7-kube-api-access-6pswf" pod "393471d3-e435-4bf0-835d-8e4c33ae17a7" (UID: "393471d3-e435-4bf0-835d-8e4c33ae17a7"). InnerVolumeSpecName "kube-api-access-6pswf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:48:25.838407 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:25.838395 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/393471d3-e435-4bf0-835d-8e4c33ae17a7-istio-envoy" pod "393471d3-e435-4bf0-835d-8e4c33ae17a7" (UID: "393471d3-e435-4bf0-835d-8e4c33ae17a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:48:25.838694 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:25.838676 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/393471d3-e435-4bf0-835d-8e4c33ae17a7-istio-podinfo" pod "393471d3-e435-4bf0-835d-8e4c33ae17a7" (UID: "393471d3-e435-4bf0-835d-8e4c33ae17a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:48:25.838840 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:25.838820 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/393471d3-e435-4bf0-835d-8e4c33ae17a7-istio-token" pod "393471d3-e435-4bf0-835d-8e4c33ae17a7" (UID: "393471d3-e435-4bf0-835d-8e4c33ae17a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:48:25.938159 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:25.938127 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5d8a300-e5ec-462a-95b3-d6f1e220dc42-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5d8a300-e5ec-462a-95b3-d6f1e220dc42-workload-certs\") pod \"b5d8a300-e5ec-462a-95b3-d6f1e220dc42\" (UID: \"b5d8a300-e5ec-462a-95b3-d6f1e220dc42\") " Oct 30 14:48:25.938291 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:25.938192 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5d8a300-e5ec-462a-95b3-d6f1e220dc42-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d8a300-e5ec-462a-95b3-d6f1e220dc42-credential-socket\") pod \"b5d8a300-e5ec-462a-95b3-d6f1e220dc42\" (UID: \"b5d8a300-e5ec-462a-95b3-d6f1e220dc42\") " Oct 30 14:48:25.938291 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:25.938238 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5d8a300-e5ec-462a-95b3-d6f1e220dc42-kube-api-access-8lg9p\" (UniqueName: \"kubernetes.io/projected/b5d8a300-e5ec-462a-95b3-d6f1e220dc42-kube-api-access-8lg9p\") pod \"b5d8a300-e5ec-462a-95b3-d6f1e220dc42\" (UID: \"b5d8a300-e5ec-462a-95b3-d6f1e220dc42\") " Oct 30 14:48:25.938403 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:25.938297 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5d8a300-e5ec-462a-95b3-d6f1e220dc42-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d8a300-e5ec-462a-95b3-d6f1e220dc42-workload-socket\") pod \"b5d8a300-e5ec-462a-95b3-d6f1e220dc42\" (UID: \"b5d8a300-e5ec-462a-95b3-d6f1e220dc42\") " Oct 30 14:48:25.938403 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:25.938354 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5d8a300-e5ec-462a-95b3-d6f1e220dc42-istio-token\" (UniqueName: \"kubernetes.io/projected/b5d8a300-e5ec-462a-95b3-d6f1e220dc42-istio-token\") pod \"b5d8a300-e5ec-462a-95b3-d6f1e220dc42\" (UID: \"b5d8a300-e5ec-462a-95b3-d6f1e220dc42\") " Oct 30 14:48:25.938503 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:25.938406 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5d8a300-e5ec-462a-95b3-d6f1e220dc42-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5d8a300-e5ec-462a-95b3-d6f1e220dc42-istiod-ca-cert\") pod \"b5d8a300-e5ec-462a-95b3-d6f1e220dc42\" (UID: \"b5d8a300-e5ec-462a-95b3-d6f1e220dc42\") " Oct 30 14:48:25.938503 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:25.938411 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5d8a300-e5ec-462a-95b3-d6f1e220dc42-credential-socket" pod "b5d8a300-e5ec-462a-95b3-d6f1e220dc42" (UID: "b5d8a300-e5ec-462a-95b3-d6f1e220dc42"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:48:25.938503 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:25.938485 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5d8a300-e5ec-462a-95b3-d6f1e220dc42-workload-certs" pod "b5d8a300-e5ec-462a-95b3-d6f1e220dc42" (UID: "b5d8a300-e5ec-462a-95b3-d6f1e220dc42"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:48:25.938503 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:25.938497 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5d8a300-e5ec-462a-95b3-d6f1e220dc42-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5d8a300-e5ec-462a-95b3-d6f1e220dc42-istio-data\") pod \"b5d8a300-e5ec-462a-95b3-d6f1e220dc42\" (UID: \"b5d8a300-e5ec-462a-95b3-d6f1e220dc42\") " Oct 30 14:48:25.938754 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:25.938571 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5d8a300-e5ec-462a-95b3-d6f1e220dc42-workload-socket" pod "b5d8a300-e5ec-462a-95b3-d6f1e220dc42" (UID: "b5d8a300-e5ec-462a-95b3-d6f1e220dc42"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:48:25.938754 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:25.938599 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5d8a300-e5ec-462a-95b3-d6f1e220dc42-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5d8a300-e5ec-462a-95b3-d6f1e220dc42-istio-envoy\") pod \"b5d8a300-e5ec-462a-95b3-d6f1e220dc42\" (UID: \"b5d8a300-e5ec-462a-95b3-d6f1e220dc42\") " Oct 30 14:48:25.938754 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:25.938660 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b5d8a300-e5ec-462a-95b3-d6f1e220dc42-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5d8a300-e5ec-462a-95b3-d6f1e220dc42-istio-podinfo\") pod \"b5d8a300-e5ec-462a-95b3-d6f1e220dc42\" (UID: \"b5d8a300-e5ec-462a-95b3-d6f1e220dc42\") " Oct 30 14:48:25.938878 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:25.938778 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5d8a300-e5ec-462a-95b3-d6f1e220dc42-istio-data" pod "b5d8a300-e5ec-462a-95b3-d6f1e220dc42" (UID: "b5d8a300-e5ec-462a-95b3-d6f1e220dc42"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:48:25.938878 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:25.938842 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5d8a300-e5ec-462a-95b3-d6f1e220dc42-istiod-ca-cert" pod "b5d8a300-e5ec-462a-95b3-d6f1e220dc42" (UID: "b5d8a300-e5ec-462a-95b3-d6f1e220dc42"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:48:25.939189 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:25.939166 4982 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5d8a300-e5ec-462a-95b3-d6f1e220dc42-istiod-ca-cert\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:25.939275 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:25.939196 4982 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5d8a300-e5ec-462a-95b3-d6f1e220dc42-istio-data\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:25.939275 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:25.939212 4982 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/393471d3-e435-4bf0-835d-8e4c33ae17a7-istio-podinfo\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:25.939275 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:25.939227 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pswf\" (UniqueName: \"kubernetes.io/projected/393471d3-e435-4bf0-835d-8e4c33ae17a7-kube-api-access-6pswf\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:25.939275 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:25.939246 4982 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5d8a300-e5ec-462a-95b3-d6f1e220dc42-workload-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:25.939275 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:25.939260 4982 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d8a300-e5ec-462a-95b3-d6f1e220dc42-credential-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:25.939454 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:25.939275 4982 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/393471d3-e435-4bf0-835d-8e4c33ae17a7-istiod-ca-cert\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:25.939454 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:25.939290 4982 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d8a300-e5ec-462a-95b3-d6f1e220dc42-workload-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:25.939454 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:25.939303 4982 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/393471d3-e435-4bf0-835d-8e4c33ae17a7-istio-envoy\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:25.939454 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:25.939317 4982 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/393471d3-e435-4bf0-835d-8e4c33ae17a7-istio-token\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:25.940514 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:25.940486 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5d8a300-e5ec-462a-95b3-d6f1e220dc42-istio-token" pod "b5d8a300-e5ec-462a-95b3-d6f1e220dc42" (UID: "b5d8a300-e5ec-462a-95b3-d6f1e220dc42"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:48:25.940597 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:25.940534 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5d8a300-e5ec-462a-95b3-d6f1e220dc42-kube-api-access-8lg9p" pod "b5d8a300-e5ec-462a-95b3-d6f1e220dc42" (UID: "b5d8a300-e5ec-462a-95b3-d6f1e220dc42"). InnerVolumeSpecName "kube-api-access-8lg9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:48:25.940983 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:25.940965 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5d8a300-e5ec-462a-95b3-d6f1e220dc42-istio-envoy" pod "b5d8a300-e5ec-462a-95b3-d6f1e220dc42" (UID: "b5d8a300-e5ec-462a-95b3-d6f1e220dc42"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:48:25.941024 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:25.941004 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b5d8a300-e5ec-462a-95b3-d6f1e220dc42-istio-podinfo" pod "b5d8a300-e5ec-462a-95b3-d6f1e220dc42" (UID: "b5d8a300-e5ec-462a-95b3-d6f1e220dc42"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:48:25.977399 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:25.977343 4982 generic.go:356] "Generic (PLEG): container finished" podID="393471d3-e435-4bf0-835d-8e4c33ae17a7" containerID="ba8ce946fc23f4ae69b314fa1433006e0f0dad111e03d0225e26bee922319aaa" exitCode=0 Oct 30 14:48:25.977518 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:25.977423 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt" Oct 30 14:48:25.977518 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:25.977449 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt" event={"ID":"393471d3-e435-4bf0-835d-8e4c33ae17a7","Type":"ContainerDied","Data":"ba8ce946fc23f4ae69b314fa1433006e0f0dad111e03d0225e26bee922319aaa"} Oct 30 14:48:25.977626 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:25.977580 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt" event={"ID":"393471d3-e435-4bf0-835d-8e4c33ae17a7","Type":"ContainerDied","Data":"a7246c961ec1f7f9f9f5308277a43fa925a5ae2a010bdcbd27936009cae35b43"} Oct 30 14:48:25.977693 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:25.977630 4982 scope.go:122] "RemoveContainer" containerID="ba8ce946fc23f4ae69b314fa1433006e0f0dad111e03d0225e26bee922319aaa" Oct 30 14:48:25.980535 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:25.980515 4982 generic.go:356] "Generic (PLEG): container finished" podID="b5d8a300-e5ec-462a-95b3-d6f1e220dc42" containerID="7da8182ed5d2a327a9f2305f9bf341b2691c4e38e16ea0b63bc4d79805f7e55a" exitCode=0 Oct 30 14:48:25.980665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:25.980590 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp" event={"ID":"b5d8a300-e5ec-462a-95b3-d6f1e220dc42","Type":"ContainerDied","Data":"7da8182ed5d2a327a9f2305f9bf341b2691c4e38e16ea0b63bc4d79805f7e55a"} Oct 30 14:48:25.980665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:25.980607 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp" Oct 30 14:48:25.980665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:25.980624 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp" event={"ID":"b5d8a300-e5ec-462a-95b3-d6f1e220dc42","Type":"ContainerDied","Data":"34836d38fe63faa50ac1150b32ce716caf168f02fe2d1ae0df7344660f2be0ab"} Oct 30 14:48:25.990137 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:25.990118 4982 scope.go:122] "RemoveContainer" containerID="ba8ce946fc23f4ae69b314fa1433006e0f0dad111e03d0225e26bee922319aaa" Oct 30 14:48:25.990421 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:48:25.990393 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba8ce946fc23f4ae69b314fa1433006e0f0dad111e03d0225e26bee922319aaa\": container with ID starting with ba8ce946fc23f4ae69b314fa1433006e0f0dad111e03d0225e26bee922319aaa not found: ID does not exist" containerID="ba8ce946fc23f4ae69b314fa1433006e0f0dad111e03d0225e26bee922319aaa" Oct 30 14:48:25.990505 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:25.990432 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba8ce946fc23f4ae69b314fa1433006e0f0dad111e03d0225e26bee922319aaa"} err="failed to get container status \"ba8ce946fc23f4ae69b314fa1433006e0f0dad111e03d0225e26bee922319aaa\": rpc error: code = NotFound desc = could not find container \"ba8ce946fc23f4ae69b314fa1433006e0f0dad111e03d0225e26bee922319aaa\": container with ID starting with ba8ce946fc23f4ae69b314fa1433006e0f0dad111e03d0225e26bee922319aaa not found: ID does not exist" Oct 30 14:48:25.990505 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:25.990456 4982 scope.go:122] "RemoveContainer" containerID="7da8182ed5d2a327a9f2305f9bf341b2691c4e38e16ea0b63bc4d79805f7e55a" Oct 30 14:48:26.001854 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:26.001830 4982 scope.go:122] "RemoveContainer" containerID="b8ce2f3cc2284ca2be475a1f1e9b3f16132f1c5a2534b8f9dc61223fc01176ad" Oct 30 14:48:26.010476 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:26.010455 4982 scope.go:122] "RemoveContainer" containerID="7da8182ed5d2a327a9f2305f9bf341b2691c4e38e16ea0b63bc4d79805f7e55a" Oct 30 14:48:26.010775 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:48:26.010754 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7da8182ed5d2a327a9f2305f9bf341b2691c4e38e16ea0b63bc4d79805f7e55a\": container with ID starting with 7da8182ed5d2a327a9f2305f9bf341b2691c4e38e16ea0b63bc4d79805f7e55a not found: ID does not exist" containerID="7da8182ed5d2a327a9f2305f9bf341b2691c4e38e16ea0b63bc4d79805f7e55a" Oct 30 14:48:26.010870 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:26.010780 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7da8182ed5d2a327a9f2305f9bf341b2691c4e38e16ea0b63bc4d79805f7e55a"} err="failed to get container status \"7da8182ed5d2a327a9f2305f9bf341b2691c4e38e16ea0b63bc4d79805f7e55a\": rpc error: code = NotFound desc = could not find container \"7da8182ed5d2a327a9f2305f9bf341b2691c4e38e16ea0b63bc4d79805f7e55a\": container with ID starting with 7da8182ed5d2a327a9f2305f9bf341b2691c4e38e16ea0b63bc4d79805f7e55a not found: ID does not exist" Oct 30 14:48:26.010870 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:26.010797 4982 scope.go:122] "RemoveContainer" containerID="b8ce2f3cc2284ca2be475a1f1e9b3f16132f1c5a2534b8f9dc61223fc01176ad" Oct 30 14:48:26.011051 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:48:26.011033 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8ce2f3cc2284ca2be475a1f1e9b3f16132f1c5a2534b8f9dc61223fc01176ad\": container with ID starting with b8ce2f3cc2284ca2be475a1f1e9b3f16132f1c5a2534b8f9dc61223fc01176ad not found: ID does not exist" containerID="b8ce2f3cc2284ca2be475a1f1e9b3f16132f1c5a2534b8f9dc61223fc01176ad" Oct 30 14:48:26.011111 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:26.011057 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8ce2f3cc2284ca2be475a1f1e9b3f16132f1c5a2534b8f9dc61223fc01176ad"} err="failed to get container status \"b8ce2f3cc2284ca2be475a1f1e9b3f16132f1c5a2534b8f9dc61223fc01176ad\": rpc error: code = NotFound desc = could not find container \"b8ce2f3cc2284ca2be475a1f1e9b3f16132f1c5a2534b8f9dc61223fc01176ad\": container with ID starting with b8ce2f3cc2284ca2be475a1f1e9b3f16132f1c5a2534b8f9dc61223fc01176ad not found: ID does not exist" Oct 30 14:48:26.040158 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:26.040132 4982 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5d8a300-e5ec-462a-95b3-d6f1e220dc42-istio-token\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:26.040158 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:26.040152 4982 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5d8a300-e5ec-462a-95b3-d6f1e220dc42-istio-envoy\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:26.040158 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:26.040161 4982 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5d8a300-e5ec-462a-95b3-d6f1e220dc42-istio-podinfo\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:26.040311 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:26.040170 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lg9p\" (UniqueName: \"kubernetes.io/projected/b5d8a300-e5ec-462a-95b3-d6f1e220dc42-kube-api-access-8lg9p\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:26.060113 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:26.060091 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp"] Oct 30 14:48:26.089442 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:26.089423 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-t9mhp"] Oct 30 14:48:26.125977 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:26.125955 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt"] Oct 30 14:48:26.142458 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:26.142436 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-brwmt"] Oct 30 14:48:26.402230 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:26.402199 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00fdf663-9bc9-4349-a565-6a8dc01c0842" path="/var/lib/kubelet/pods/00fdf663-9bc9-4349-a565-6a8dc01c0842/volumes" Oct 30 14:48:26.402843 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:26.402825 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="393471d3-e435-4bf0-835d-8e4c33ae17a7" path="/var/lib/kubelet/pods/393471d3-e435-4bf0-835d-8e4c33ae17a7/volumes" Oct 30 14:48:26.403395 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:26.403381 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5d8a300-e5ec-462a-95b3-d6f1e220dc42" path="/var/lib/kubelet/pods/b5d8a300-e5ec-462a-95b3-d6f1e220dc42/volumes" Oct 30 14:48:26.404356 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:26.404337 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc3a3a28-d723-459a-93d0-ff115eac479a" path="/var/lib/kubelet/pods/cc3a3a28-d723-459a-93d0-ff115eac479a/volumes" Oct 30 14:48:27.166989 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:27.166953 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:28.012098 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:48:28.012063 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:48:28.334315 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:28.334284 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:28.482190 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:28.482156 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:28.742135 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:28.742057 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:29.634582 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:29.634555 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:31.681927 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:31.681893 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:31.820426 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:31.820370 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:34.442108 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:34.442065 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:34.796788 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:34.796690 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:36.046689 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:36.046649 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:36.364543 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:36.364508 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:37.960580 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:37.960541 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50f76dcc-fngln"] Oct 30 14:48:37.961061 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:37.961047 4982 cpu_manager.go:414] "containerMap: removing container" podUID="00fdf663-9bc9-4349-a565-6a8dc01c0842" containerName="main" Oct 30 14:48:37.961106 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:37.961062 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00fdf663-9bc9-4349-a565-6a8dc01c0842" containerName="main" Oct 30 14:48:37.961106 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:37.961073 4982 cpu_manager.go:414] "containerMap: removing container" podUID="9703a518-851b-4a8e-9e2c-586ea7fff550" containerName="payload-processing" Oct 30 14:48:37.961106 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:37.961078 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9703a518-851b-4a8e-9e2c-586ea7fff550" containerName="payload-processing" Oct 30 14:48:37.961106 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:37.961089 4982 cpu_manager.go:414] "containerMap: removing container" podUID="02808c74-feb9-4fe4-8fe0-9f040dd3e8a1" containerName="payload-processing" Oct 30 14:48:37.961106 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:37.961094 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02808c74-feb9-4fe4-8fe0-9f040dd3e8a1" containerName="payload-processing" Oct 30 14:48:37.961319 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:37.961109 4982 cpu_manager.go:414] "containerMap: removing container" podUID="02251d15-f79c-41a6-bd91-6ec5969e1f63" containerName="main" Oct 30 14:48:37.961319 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:37.961115 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02251d15-f79c-41a6-bd91-6ec5969e1f63" containerName="main" Oct 30 14:48:37.961319 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:37.961130 4982 cpu_manager.go:414] "containerMap: removing container" podUID="cc3a3a28-d723-459a-93d0-ff115eac479a" containerName="main" Oct 30 14:48:37.961319 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:37.961134 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc3a3a28-d723-459a-93d0-ff115eac479a" containerName="main" Oct 30 14:48:37.961319 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:37.961145 4982 cpu_manager.go:414] "containerMap: removing container" podUID="b5d8a300-e5ec-462a-95b3-d6f1e220dc42" containerName="istio-proxy" Oct 30 14:48:37.961319 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:37.961149 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5d8a300-e5ec-462a-95b3-d6f1e220dc42" containerName="istio-proxy" Oct 30 14:48:37.961319 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:37.961158 4982 cpu_manager.go:414] "containerMap: removing container" podUID="543506fd-c659-42d5-8338-8fdc25422ecb" containerName="payload-pre-processing" Oct 30 14:48:37.961319 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:37.961162 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="543506fd-c659-42d5-8338-8fdc25422ecb" containerName="payload-pre-processing" Oct 30 14:48:37.961319 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:37.961173 4982 cpu_manager.go:414] "containerMap: removing container" podUID="aaecf278-2542-4af3-b82e-233e83c24e47" containerName="revoke-keys" Oct 30 14:48:37.961319 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:37.961178 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaecf278-2542-4af3-b82e-233e83c24e47" containerName="revoke-keys" Oct 30 14:48:37.961319 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:37.961184 4982 cpu_manager.go:414] "containerMap: removing container" podUID="393471d3-e435-4bf0-835d-8e4c33ae17a7" containerName="istio-proxy" Oct 30 14:48:37.961319 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:37.961189 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="393471d3-e435-4bf0-835d-8e4c33ae17a7" containerName="istio-proxy" Oct 30 14:48:37.961319 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:37.961197 4982 cpu_manager.go:414] "containerMap: removing container" podUID="b5d8a300-e5ec-462a-95b3-d6f1e220dc42" containerName="istio-proxy" Oct 30 14:48:37.961319 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:37.961201 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5d8a300-e5ec-462a-95b3-d6f1e220dc42" containerName="istio-proxy" Oct 30 14:48:37.961319 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:37.961264 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="02251d15-f79c-41a6-bd91-6ec5969e1f63" containerName="main" Oct 30 14:48:37.961319 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:37.961276 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc3a3a28-d723-459a-93d0-ff115eac479a" containerName="main" Oct 30 14:48:37.961319 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:37.961286 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5d8a300-e5ec-462a-95b3-d6f1e220dc42" containerName="istio-proxy" Oct 30 14:48:37.961319 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:37.961293 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="543506fd-c659-42d5-8338-8fdc25422ecb" containerName="payload-pre-processing" Oct 30 14:48:37.961319 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:37.961299 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaecf278-2542-4af3-b82e-233e83c24e47" containerName="revoke-keys" Oct 30 14:48:37.961319 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:37.961305 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="393471d3-e435-4bf0-835d-8e4c33ae17a7" containerName="istio-proxy" Oct 30 14:48:37.961319 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:37.961311 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5d8a300-e5ec-462a-95b3-d6f1e220dc42" containerName="istio-proxy" Oct 30 14:48:37.961319 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:37.961317 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="00fdf663-9bc9-4349-a565-6a8dc01c0842" containerName="main" Oct 30 14:48:37.961319 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:37.961323 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="9703a518-851b-4a8e-9e2c-586ea7fff550" containerName="payload-processing" Oct 30 14:48:37.961319 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:37.961328 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="02808c74-feb9-4fe4-8fe0-9f040dd3e8a1" containerName="payload-processing" Oct 30 14:48:37.966283 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:37.966257 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50f76dcc-fngln" Oct 30 14:48:38.009741 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:38.009707 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50f76dcc-fngln"] Oct 30 14:48:38.060372 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:38.060342 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31ed90a7-a67c-4ece-b5b0-0f88f6860ec9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50f76dcc-fngln\" (UID: \"31ed90a7-a67c-4ece-b5b0-0f88f6860ec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50f76dcc-fngln" Oct 30 14:48:38.060518 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:38.060460 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p9df\" (UniqueName: \"kubernetes.io/projected/31ed90a7-a67c-4ece-b5b0-0f88f6860ec9-kube-api-access-6p9df\") pod \"maas-api-revoke-sub-50f76dcc-fngln\" (UID: \"31ed90a7-a67c-4ece-b5b0-0f88f6860ec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50f76dcc-fngln" Oct 30 14:48:38.060561 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:38.060523 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31ed90a7-a67c-4ece-b5b0-0f88f6860ec9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50f76dcc-fngln\" (UID: \"31ed90a7-a67c-4ece-b5b0-0f88f6860ec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50f76dcc-fngln" Oct 30 14:48:38.068514 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:38.068488 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:48:38.068514 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:38.068505 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:48:38.162009 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:38.161980 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p9df\" (UniqueName: \"kubernetes.io/projected/31ed90a7-a67c-4ece-b5b0-0f88f6860ec9-kube-api-access-6p9df\") pod \"maas-api-revoke-sub-50f76dcc-fngln\" (UID: \"31ed90a7-a67c-4ece-b5b0-0f88f6860ec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50f76dcc-fngln" Oct 30 14:48:38.162158 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:38.162033 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31ed90a7-a67c-4ece-b5b0-0f88f6860ec9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50f76dcc-fngln\" (UID: \"31ed90a7-a67c-4ece-b5b0-0f88f6860ec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50f76dcc-fngln" Oct 30 14:48:38.162158 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:38.162106 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31ed90a7-a67c-4ece-b5b0-0f88f6860ec9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50f76dcc-fngln\" (UID: \"31ed90a7-a67c-4ece-b5b0-0f88f6860ec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50f76dcc-fngln" Oct 30 14:48:38.162721 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:38.162692 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31ed90a7-a67c-4ece-b5b0-0f88f6860ec9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50f76dcc-fngln\" (UID: \"31ed90a7-a67c-4ece-b5b0-0f88f6860ec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50f76dcc-fngln" Oct 30 14:48:38.164427 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:38.164404 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31ed90a7-a67c-4ece-b5b0-0f88f6860ec9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50f76dcc-fngln\" (UID: \"31ed90a7-a67c-4ece-b5b0-0f88f6860ec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50f76dcc-fngln" Oct 30 14:48:38.187691 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:38.187666 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p9df\" (UniqueName: \"kubernetes.io/projected/31ed90a7-a67c-4ece-b5b0-0f88f6860ec9-kube-api-access-6p9df\") pod \"maas-api-revoke-sub-50f76dcc-fngln\" (UID: \"31ed90a7-a67c-4ece-b5b0-0f88f6860ec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50f76dcc-fngln" Oct 30 14:48:38.283617 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:38.283557 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50f76dcc-fngln" Oct 30 14:48:38.554535 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:38.554421 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:38.589513 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:38.589475 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50f76dcc-fngln"] Oct 30 14:48:38.593904 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:48:38.593875 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31ed90a7_a67c_4ece_b5b0_0f88f6860ec9.slice/crio-e65bf17064f3d7164e88df431257ee8c88d0899b247ed2826ca208757def7a86 WatchSource:0}: Error finding container e65bf17064f3d7164e88df431257ee8c88d0899b247ed2826ca208757def7a86: Status 404 returned error can't find the container with id e65bf17064f3d7164e88df431257ee8c88d0899b247ed2826ca208757def7a86 Oct 30 14:48:39.069761 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:39.069713 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50f76dcc-fngln" event={"ID":"31ed90a7-a67c-4ece-b5b0-0f88f6860ec9","Type":"ContainerStarted","Data":"a4283f1bf83083d0bc18ac2cb920d901a76b53449add986e9a9941a550c6d097"} Oct 30 14:48:39.070208 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:39.069774 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50f76dcc-fngln" event={"ID":"31ed90a7-a67c-4ece-b5b0-0f88f6860ec9","Type":"ContainerStarted","Data":"e65bf17064f3d7164e88df431257ee8c88d0899b247ed2826ca208757def7a86"} Oct 30 14:48:39.114227 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:39.114176 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50f76dcc-fngln" podStartSLOduration=2.114161192 podStartE2EDuration="2.114161192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:48:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:48:39.113475611 +0000 UTC m=+3588.857609652" watchObservedRunningTime="2026-10-30 14:48:39.114161192 +0000 UTC m=+3588.858295234" Oct 30 14:48:40.036806 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:40.036768 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-90b406-56b44b6b55-9nd98"] Oct 30 14:48:40.037299 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:40.037265 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90b406-56b44b6b55-9nd98" podUID="4d6e8423-5356-4a73-9c9b-43c29eedb02b" containerName="maas-api" containerID="cri-o://f784b6390f0eca03503631854eeb0eb38f2de7eb5f066775b3ff258fb87d2bd4" gracePeriod=30 Oct 30 14:48:40.079613 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:40.079586 4982 generic.go:356] "Generic (PLEG): container finished" podID="31ed90a7-a67c-4ece-b5b0-0f88f6860ec9" containerID="a4283f1bf83083d0bc18ac2cb920d901a76b53449add986e9a9941a550c6d097" exitCode=0 Oct 30 14:48:40.082081 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:40.079666 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50f76dcc-fngln" event={"ID":"31ed90a7-a67c-4ece-b5b0-0f88f6860ec9","Type":"ContainerDied","Data":"a4283f1bf83083d0bc18ac2cb920d901a76b53449add986e9a9941a550c6d097"} Oct 30 14:48:40.178620 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:40.178587 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:40.287546 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:40.287457 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-90b406-5f96f96db6-dwhwj"] Oct 30 14:48:40.287774 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:40.287747 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-90b406-5f96f96db6-dwhwj" podUID="ffeee019-eb71-4340-87a4-8451e5c670c6" containerName="payload-processing" containerID="cri-o://d44884feb547f6306b257870176c264162b7ec58824d761a9163c843f0a55f88" gracePeriod=30 Oct 30 14:48:40.345278 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:40.344014 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-90b406-5dd68cb667-gfd5c"] Oct 30 14:48:40.345278 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:40.344273 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90b406-5dd68cb667-gfd5c" podUID="5424ec16-5768-428c-b826-fc0df029900b" containerName="payload-pre-processing" containerID="cri-o://57eb031ffd8d278aaaec427e96ad9bff53d542f7fcaaba8e7a22cd6415fe8110" gracePeriod=30 Oct 30 14:48:40.448734 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:40.448702 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:40.464818 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:40.464794 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90b406-56b44b6b55-9nd98" Oct 30 14:48:40.488569 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:40.488534 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d6e8423-5356-4a73-9c9b-43c29eedb02b-kube-api-access-5298h\" (UniqueName: \"kubernetes.io/projected/4d6e8423-5356-4a73-9c9b-43c29eedb02b-kube-api-access-5298h\") pod \"4d6e8423-5356-4a73-9c9b-43c29eedb02b\" (UID: \"4d6e8423-5356-4a73-9c9b-43c29eedb02b\") " Oct 30 14:48:40.488760 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:40.488629 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d6e8423-5356-4a73-9c9b-43c29eedb02b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d6e8423-5356-4a73-9c9b-43c29eedb02b-openshift-service-ca\") pod \"4d6e8423-5356-4a73-9c9b-43c29eedb02b\" (UID: \"4d6e8423-5356-4a73-9c9b-43c29eedb02b\") " Oct 30 14:48:40.488760 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:40.488744 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d6e8423-5356-4a73-9c9b-43c29eedb02b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d6e8423-5356-4a73-9c9b-43c29eedb02b-metrics-certs\") pod \"4d6e8423-5356-4a73-9c9b-43c29eedb02b\" (UID: \"4d6e8423-5356-4a73-9c9b-43c29eedb02b\") " Oct 30 14:48:40.488886 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:40.488846 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d6e8423-5356-4a73-9c9b-43c29eedb02b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d6e8423-5356-4a73-9c9b-43c29eedb02b-maas-api-tls\") pod \"4d6e8423-5356-4a73-9c9b-43c29eedb02b\" (UID: \"4d6e8423-5356-4a73-9c9b-43c29eedb02b\") " Oct 30 14:48:40.489771 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:40.489740 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d6e8423-5356-4a73-9c9b-43c29eedb02b-openshift-service-ca" pod "4d6e8423-5356-4a73-9c9b-43c29eedb02b" (UID: "4d6e8423-5356-4a73-9c9b-43c29eedb02b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:48:40.499008 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:40.492350 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d6e8423-5356-4a73-9c9b-43c29eedb02b-kube-api-access-5298h" pod "4d6e8423-5356-4a73-9c9b-43c29eedb02b" (UID: "4d6e8423-5356-4a73-9c9b-43c29eedb02b"). InnerVolumeSpecName "kube-api-access-5298h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:48:40.500738 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:40.500701 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d6e8423-5356-4a73-9c9b-43c29eedb02b-maas-api-tls" pod "4d6e8423-5356-4a73-9c9b-43c29eedb02b" (UID: "4d6e8423-5356-4a73-9c9b-43c29eedb02b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:48:40.503818 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:40.503768 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d6e8423-5356-4a73-9c9b-43c29eedb02b-metrics-certs" pod "4d6e8423-5356-4a73-9c9b-43c29eedb02b" (UID: "4d6e8423-5356-4a73-9c9b-43c29eedb02b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:48:40.578000 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:40.577969 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-90b406-5f96f96db6-dwhwj" Oct 30 14:48:40.590305 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:40.590279 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d6e8423-5356-4a73-9c9b-43c29eedb02b-maas-api-tls\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:40.590305 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:40.590304 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5298h\" (UniqueName: \"kubernetes.io/projected/4d6e8423-5356-4a73-9c9b-43c29eedb02b-kube-api-access-5298h\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:40.590480 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:40.590314 4982 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d6e8423-5356-4a73-9c9b-43c29eedb02b-openshift-service-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:40.590480 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:40.590325 4982 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d6e8423-5356-4a73-9c9b-43c29eedb02b-metrics-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:40.616461 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:40.616427 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:40.693223 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:40.693180 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffeee019-eb71-4340-87a4-8451e5c670c6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffeee019-eb71-4340-87a4-8451e5c670c6-plugins-config-volume\") pod \"ffeee019-eb71-4340-87a4-8451e5c670c6\" (UID: \"ffeee019-eb71-4340-87a4-8451e5c670c6\") " Oct 30 14:48:40.693437 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:40.693348 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffeee019-eb71-4340-87a4-8451e5c670c6-kube-api-access-6xv79\" (UniqueName: \"kubernetes.io/projected/ffeee019-eb71-4340-87a4-8451e5c670c6-kube-api-access-6xv79\") pod \"ffeee019-eb71-4340-87a4-8451e5c670c6\" (UID: \"ffeee019-eb71-4340-87a4-8451e5c670c6\") " Oct 30 14:48:40.693766 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:40.693728 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffeee019-eb71-4340-87a4-8451e5c670c6-plugins-config-volume" pod "ffeee019-eb71-4340-87a4-8451e5c670c6" (UID: "ffeee019-eb71-4340-87a4-8451e5c670c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:48:40.696580 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:40.696531 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffeee019-eb71-4340-87a4-8451e5c670c6-kube-api-access-6xv79" pod "ffeee019-eb71-4340-87a4-8451e5c670c6" (UID: "ffeee019-eb71-4340-87a4-8451e5c670c6"). InnerVolumeSpecName "kube-api-access-6xv79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:48:40.700098 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:40.700062 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90b406-5dd68cb667-gfd5c" Oct 30 14:48:40.794600 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:40.794571 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5424ec16-5768-428c-b826-fc0df029900b-kube-api-access-cv9hn\" (UniqueName: \"kubernetes.io/projected/5424ec16-5768-428c-b826-fc0df029900b-kube-api-access-cv9hn\") pod \"5424ec16-5768-428c-b826-fc0df029900b\" (UID: \"5424ec16-5768-428c-b826-fc0df029900b\") " Oct 30 14:48:40.794807 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:40.794632 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5424ec16-5768-428c-b826-fc0df029900b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5424ec16-5768-428c-b826-fc0df029900b-plugins-config-volume\") pod \"5424ec16-5768-428c-b826-fc0df029900b\" (UID: \"5424ec16-5768-428c-b826-fc0df029900b\") " Oct 30 14:48:40.795123 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:40.795096 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5424ec16-5768-428c-b826-fc0df029900b-plugins-config-volume" pod "5424ec16-5768-428c-b826-fc0df029900b" (UID: "5424ec16-5768-428c-b826-fc0df029900b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:48:40.795206 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:40.795188 4982 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffeee019-eb71-4340-87a4-8451e5c670c6-plugins-config-volume\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:40.795206 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:40.795203 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xv79\" (UniqueName: \"kubernetes.io/projected/ffeee019-eb71-4340-87a4-8451e5c670c6-kube-api-access-6xv79\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:40.796761 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:40.796733 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5424ec16-5768-428c-b826-fc0df029900b-kube-api-access-cv9hn" pod "5424ec16-5768-428c-b826-fc0df029900b" (UID: "5424ec16-5768-428c-b826-fc0df029900b"). InnerVolumeSpecName "kube-api-access-cv9hn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:48:40.831893 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:40.831863 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:40.896218 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:40.896186 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cv9hn\" (UniqueName: \"kubernetes.io/projected/5424ec16-5768-428c-b826-fc0df029900b-kube-api-access-cv9hn\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:40.896218 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:40.896218 4982 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5424ec16-5768-428c-b826-fc0df029900b-plugins-config-volume\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:40.959337 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:40.959305 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90b406-6jzrm"] Oct 30 14:48:40.964337 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:40.964310 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-90b406-6jzrm"] Oct 30 14:48:41.065636 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:41.065585 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:41.088825 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:41.088756 4982 generic.go:356] "Generic (PLEG): container finished" podID="5424ec16-5768-428c-b826-fc0df029900b" containerID="57eb031ffd8d278aaaec427e96ad9bff53d542f7fcaaba8e7a22cd6415fe8110" exitCode=0 Oct 30 14:48:41.088825 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:41.088807 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90b406-5dd68cb667-gfd5c" Oct 30 14:48:41.089344 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:41.088835 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90b406-5dd68cb667-gfd5c" event={"ID":"5424ec16-5768-428c-b826-fc0df029900b","Type":"ContainerDied","Data":"57eb031ffd8d278aaaec427e96ad9bff53d542f7fcaaba8e7a22cd6415fe8110"} Oct 30 14:48:41.089344 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:41.088894 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90b406-5dd68cb667-gfd5c" event={"ID":"5424ec16-5768-428c-b826-fc0df029900b","Type":"ContainerDied","Data":"f33252183b943df4d42b8757e0090dc337a74939abf6b33c0a42245988845479"} Oct 30 14:48:41.089344 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:41.088924 4982 scope.go:122] "RemoveContainer" containerID="57eb031ffd8d278aaaec427e96ad9bff53d542f7fcaaba8e7a22cd6415fe8110" Oct 30 14:48:41.090492 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:41.090329 4982 generic.go:356] "Generic (PLEG): container finished" podID="ffeee019-eb71-4340-87a4-8451e5c670c6" containerID="d44884feb547f6306b257870176c264162b7ec58824d761a9163c843f0a55f88" exitCode=0 Oct 30 14:48:41.090492 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:41.090369 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-90b406-5f96f96db6-dwhwj" Oct 30 14:48:41.090492 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:41.090409 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-90b406-5f96f96db6-dwhwj" event={"ID":"ffeee019-eb71-4340-87a4-8451e5c670c6","Type":"ContainerDied","Data":"d44884feb547f6306b257870176c264162b7ec58824d761a9163c843f0a55f88"} Oct 30 14:48:41.090492 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:41.090471 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-90b406-5f96f96db6-dwhwj" event={"ID":"ffeee019-eb71-4340-87a4-8451e5c670c6","Type":"ContainerDied","Data":"7c20e8c5736bddbdc743f42ff2c540485a2d69d359ec9bf5047d79219cd858e1"} Oct 30 14:48:41.092247 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:41.092225 4982 generic.go:356] "Generic (PLEG): container finished" podID="4d6e8423-5356-4a73-9c9b-43c29eedb02b" containerID="f784b6390f0eca03503631854eeb0eb38f2de7eb5f066775b3ff258fb87d2bd4" exitCode=0 Oct 30 14:48:41.092458 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:41.092296 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90b406-56b44b6b55-9nd98" event={"ID":"4d6e8423-5356-4a73-9c9b-43c29eedb02b","Type":"ContainerDied","Data":"f784b6390f0eca03503631854eeb0eb38f2de7eb5f066775b3ff258fb87d2bd4"} Oct 30 14:48:41.096758 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:41.092339 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90b406-56b44b6b55-9nd98" Oct 30 14:48:41.096758 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:41.093728 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90b406-56b44b6b55-9nd98" event={"ID":"4d6e8423-5356-4a73-9c9b-43c29eedb02b","Type":"ContainerDied","Data":"50d4d09d95830e17e014e056d50c54edc4050913f1afabab02852b941604b8a7"} Oct 30 14:48:41.103625 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:41.103587 4982 scope.go:122] "RemoveContainer" containerID="57eb031ffd8d278aaaec427e96ad9bff53d542f7fcaaba8e7a22cd6415fe8110" Oct 30 14:48:41.103965 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:48:41.103941 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57eb031ffd8d278aaaec427e96ad9bff53d542f7fcaaba8e7a22cd6415fe8110\": container with ID starting with 57eb031ffd8d278aaaec427e96ad9bff53d542f7fcaaba8e7a22cd6415fe8110 not found: ID does not exist" containerID="57eb031ffd8d278aaaec427e96ad9bff53d542f7fcaaba8e7a22cd6415fe8110" Oct 30 14:48:41.104059 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:41.103972 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57eb031ffd8d278aaaec427e96ad9bff53d542f7fcaaba8e7a22cd6415fe8110"} err="failed to get container status \"57eb031ffd8d278aaaec427e96ad9bff53d542f7fcaaba8e7a22cd6415fe8110\": rpc error: code = NotFound desc = could not find container \"57eb031ffd8d278aaaec427e96ad9bff53d542f7fcaaba8e7a22cd6415fe8110\": container with ID starting with 57eb031ffd8d278aaaec427e96ad9bff53d542f7fcaaba8e7a22cd6415fe8110 not found: ID does not exist" Oct 30 14:48:41.104059 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:41.103995 4982 scope.go:122] "RemoveContainer" containerID="d44884feb547f6306b257870176c264162b7ec58824d761a9163c843f0a55f88" Oct 30 14:48:41.112986 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:41.112968 4982 scope.go:122] "RemoveContainer" containerID="d44884feb547f6306b257870176c264162b7ec58824d761a9163c843f0a55f88" Oct 30 14:48:41.113256 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:48:41.113236 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d44884feb547f6306b257870176c264162b7ec58824d761a9163c843f0a55f88\": container with ID starting with d44884feb547f6306b257870176c264162b7ec58824d761a9163c843f0a55f88 not found: ID does not exist" containerID="d44884feb547f6306b257870176c264162b7ec58824d761a9163c843f0a55f88" Oct 30 14:48:41.113331 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:41.113263 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d44884feb547f6306b257870176c264162b7ec58824d761a9163c843f0a55f88"} err="failed to get container status \"d44884feb547f6306b257870176c264162b7ec58824d761a9163c843f0a55f88\": rpc error: code = NotFound desc = could not find container \"d44884feb547f6306b257870176c264162b7ec58824d761a9163c843f0a55f88\": container with ID starting with d44884feb547f6306b257870176c264162b7ec58824d761a9163c843f0a55f88 not found: ID does not exist" Oct 30 14:48:41.113331 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:41.113279 4982 scope.go:122] "RemoveContainer" containerID="f784b6390f0eca03503631854eeb0eb38f2de7eb5f066775b3ff258fb87d2bd4" Oct 30 14:48:41.123893 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:41.123877 4982 scope.go:122] "RemoveContainer" containerID="f784b6390f0eca03503631854eeb0eb38f2de7eb5f066775b3ff258fb87d2bd4" Oct 30 14:48:41.124146 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:48:41.124130 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f784b6390f0eca03503631854eeb0eb38f2de7eb5f066775b3ff258fb87d2bd4\": container with ID starting with f784b6390f0eca03503631854eeb0eb38f2de7eb5f066775b3ff258fb87d2bd4 not found: ID does not exist" containerID="f784b6390f0eca03503631854eeb0eb38f2de7eb5f066775b3ff258fb87d2bd4" Oct 30 14:48:41.124204 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:41.124162 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f784b6390f0eca03503631854eeb0eb38f2de7eb5f066775b3ff258fb87d2bd4"} err="failed to get container status \"f784b6390f0eca03503631854eeb0eb38f2de7eb5f066775b3ff258fb87d2bd4\": rpc error: code = NotFound desc = could not find container \"f784b6390f0eca03503631854eeb0eb38f2de7eb5f066775b3ff258fb87d2bd4\": container with ID starting with f784b6390f0eca03503631854eeb0eb38f2de7eb5f066775b3ff258fb87d2bd4 not found: ID does not exist" Oct 30 14:48:41.169034 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:41.169000 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-90b406-5dd68cb667-gfd5c"] Oct 30 14:48:41.173976 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:41.173945 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-90b406-5dd68cb667-gfd5c"] Oct 30 14:48:41.214670 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:41.214532 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-90b406-5f96f96db6-dwhwj"] Oct 30 14:48:41.222535 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:41.221520 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-90b406-5f96f96db6-dwhwj"] Oct 30 14:48:41.238299 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:41.238277 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50f76dcc-fngln" Oct 30 14:48:41.255558 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:41.255536 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-90b406-56b44b6b55-9nd98"] Oct 30 14:48:41.261084 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:41.261060 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-90b406-56b44b6b55-9nd98"] Oct 30 14:48:41.301069 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:41.301047 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31ed90a7-a67c-4ece-b5b0-0f88f6860ec9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31ed90a7-a67c-4ece-b5b0-0f88f6860ec9-maas-api-openshift-ca\") pod \"31ed90a7-a67c-4ece-b5b0-0f88f6860ec9\" (UID: \"31ed90a7-a67c-4ece-b5b0-0f88f6860ec9\") " Oct 30 14:48:41.301191 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:41.301084 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31ed90a7-a67c-4ece-b5b0-0f88f6860ec9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31ed90a7-a67c-4ece-b5b0-0f88f6860ec9-maas-api-xks-ca\") pod \"31ed90a7-a67c-4ece-b5b0-0f88f6860ec9\" (UID: \"31ed90a7-a67c-4ece-b5b0-0f88f6860ec9\") " Oct 30 14:48:41.301191 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:41.301168 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31ed90a7-a67c-4ece-b5b0-0f88f6860ec9-kube-api-access-6p9df\" (UniqueName: \"kubernetes.io/projected/31ed90a7-a67c-4ece-b5b0-0f88f6860ec9-kube-api-access-6p9df\") pod \"31ed90a7-a67c-4ece-b5b0-0f88f6860ec9\" (UID: \"31ed90a7-a67c-4ece-b5b0-0f88f6860ec9\") " Oct 30 14:48:41.301388 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:41.301364 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31ed90a7-a67c-4ece-b5b0-0f88f6860ec9-maas-api-openshift-ca" pod "31ed90a7-a67c-4ece-b5b0-0f88f6860ec9" (UID: "31ed90a7-a67c-4ece-b5b0-0f88f6860ec9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:48:41.301521 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:41.301497 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31ed90a7-a67c-4ece-b5b0-0f88f6860ec9-maas-api-openshift-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:41.303356 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:41.303322 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31ed90a7-a67c-4ece-b5b0-0f88f6860ec9-kube-api-access-6p9df" pod "31ed90a7-a67c-4ece-b5b0-0f88f6860ec9" (UID: "31ed90a7-a67c-4ece-b5b0-0f88f6860ec9"). InnerVolumeSpecName "kube-api-access-6p9df". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:48:41.303443 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:41.303382 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31ed90a7-a67c-4ece-b5b0-0f88f6860ec9-maas-api-xks-ca" pod "31ed90a7-a67c-4ece-b5b0-0f88f6860ec9" (UID: "31ed90a7-a67c-4ece-b5b0-0f88f6860ec9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:48:41.402218 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:41.402167 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6p9df\" (UniqueName: \"kubernetes.io/projected/31ed90a7-a67c-4ece-b5b0-0f88f6860ec9-kube-api-access-6p9df\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:41.402218 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:41.402207 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31ed90a7-a67c-4ece-b5b0-0f88f6860ec9-maas-api-xks-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:41.488193 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:41.488147 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:42.102947 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:42.102909 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50f76dcc-fngln" event={"ID":"31ed90a7-a67c-4ece-b5b0-0f88f6860ec9","Type":"ContainerDied","Data":"e65bf17064f3d7164e88df431257ee8c88d0899b247ed2826ca208757def7a86"} Oct 30 14:48:42.102947 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:42.102941 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50f76dcc-fngln" Oct 30 14:48:42.103363 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:42.102971 4982 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e65bf17064f3d7164e88df431257ee8c88d0899b247ed2826ca208757def7a86" Oct 30 14:48:42.399836 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:42.399764 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12350227-cab2-45e5-aadc-f65f4e48bca6" path="/var/lib/kubelet/pods/12350227-cab2-45e5-aadc-f65f4e48bca6/volumes" Oct 30 14:48:42.400203 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:42.400189 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d6e8423-5356-4a73-9c9b-43c29eedb02b" path="/var/lib/kubelet/pods/4d6e8423-5356-4a73-9c9b-43c29eedb02b/volumes" Oct 30 14:48:42.400541 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:42.400528 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5424ec16-5768-428c-b826-fc0df029900b" path="/var/lib/kubelet/pods/5424ec16-5768-428c-b826-fc0df029900b/volumes" Oct 30 14:48:42.401285 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:42.401263 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffeee019-eb71-4340-87a4-8451e5c670c6" path="/var/lib/kubelet/pods/ffeee019-eb71-4340-87a4-8451e5c670c6/volumes" Oct 30 14:48:43.020560 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:43.020528 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:43.483009 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:43.482970 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:43.662984 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:43.662950 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:43.722048 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:43.722007 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx"] Oct 30 14:48:43.722262 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:43.722234 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx" podUID="19c9ad1b-96d7-41de-a3cc-0960ef9c31b8" containerName="istio-proxy" containerID="cri-o://0343b9bee9368249e29c13772a84c585addf6942ee565d663c9f8ff2541e86b5" gracePeriod=30 Oct 30 14:48:43.815343 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:43.815308 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:44.304116 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:44.304046 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3b95dd-2qfpx"] Oct 30 14:48:44.304554 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:44.304540 4982 cpu_manager.go:414] "containerMap: removing container" podUID="4d6e8423-5356-4a73-9c9b-43c29eedb02b" containerName="maas-api" Oct 30 14:48:44.304605 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:44.304555 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d6e8423-5356-4a73-9c9b-43c29eedb02b" containerName="maas-api" Oct 30 14:48:44.304605 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:44.304565 4982 cpu_manager.go:414] "containerMap: removing container" podUID="ffeee019-eb71-4340-87a4-8451e5c670c6" containerName="payload-processing" Oct 30 14:48:44.304605 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:44.304571 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffeee019-eb71-4340-87a4-8451e5c670c6" containerName="payload-processing" Oct 30 14:48:44.304605 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:44.304580 4982 cpu_manager.go:414] "containerMap: removing container" podUID="5424ec16-5768-428c-b826-fc0df029900b" containerName="payload-pre-processing" Oct 30 14:48:44.304605 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:44.304586 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5424ec16-5768-428c-b826-fc0df029900b" containerName="payload-pre-processing" Oct 30 14:48:44.304605 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:44.304604 4982 cpu_manager.go:414] "containerMap: removing container" podUID="31ed90a7-a67c-4ece-b5b0-0f88f6860ec9" containerName="revoke-keys" Oct 30 14:48:44.304809 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:44.304609 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31ed90a7-a67c-4ece-b5b0-0f88f6860ec9" containerName="revoke-keys" Oct 30 14:48:44.304809 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:44.304728 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d6e8423-5356-4a73-9c9b-43c29eedb02b" containerName="maas-api" Oct 30 14:48:44.304809 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:44.304736 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffeee019-eb71-4340-87a4-8451e5c670c6" containerName="payload-processing" Oct 30 14:48:44.304809 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:44.304744 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="5424ec16-5768-428c-b826-fc0df029900b" containerName="payload-pre-processing" Oct 30 14:48:44.304809 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:44.304756 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="31ed90a7-a67c-4ece-b5b0-0f88f6860ec9" containerName="revoke-keys" Oct 30 14:48:44.309325 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:44.309306 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3b95dd-2qfpx" Oct 30 14:48:44.351590 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:44.351562 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3b95dd-2qfpx"] Oct 30 14:48:44.365367 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:44.365342 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:48:44.365465 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:44.365362 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:48:44.434493 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:44.434466 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/757b0c4a-6c88-4a4e-9aaa-04a17c7592c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-3b95dd-2qfpx\" (UID: \"757b0c4a-6c88-4a4e-9aaa-04a17c7592c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3b95dd-2qfpx" Oct 30 14:48:44.434608 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:44.434508 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/757b0c4a-6c88-4a4e-9aaa-04a17c7592c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-3b95dd-2qfpx\" (UID: \"757b0c4a-6c88-4a4e-9aaa-04a17c7592c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3b95dd-2qfpx" Oct 30 14:48:44.434608 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:44.434585 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwxsx\" (UniqueName: \"kubernetes.io/projected/757b0c4a-6c88-4a4e-9aaa-04a17c7592c5-kube-api-access-qwxsx\") pod \"maas-api-revoke-keys-e2e-ipp-b-3b95dd-2qfpx\" (UID: \"757b0c4a-6c88-4a4e-9aaa-04a17c7592c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3b95dd-2qfpx" Oct 30 14:48:44.535299 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:44.535274 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/757b0c4a-6c88-4a4e-9aaa-04a17c7592c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-3b95dd-2qfpx\" (UID: \"757b0c4a-6c88-4a4e-9aaa-04a17c7592c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3b95dd-2qfpx" Oct 30 14:48:44.535629 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:44.535315 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/757b0c4a-6c88-4a4e-9aaa-04a17c7592c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-3b95dd-2qfpx\" (UID: \"757b0c4a-6c88-4a4e-9aaa-04a17c7592c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3b95dd-2qfpx" Oct 30 14:48:44.535629 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:44.535355 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwxsx\" (UniqueName: \"kubernetes.io/projected/757b0c4a-6c88-4a4e-9aaa-04a17c7592c5-kube-api-access-qwxsx\") pod \"maas-api-revoke-keys-e2e-ipp-b-3b95dd-2qfpx\" (UID: \"757b0c4a-6c88-4a4e-9aaa-04a17c7592c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3b95dd-2qfpx" Oct 30 14:48:44.535930 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:44.535903 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/757b0c4a-6c88-4a4e-9aaa-04a17c7592c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-3b95dd-2qfpx\" (UID: \"757b0c4a-6c88-4a4e-9aaa-04a17c7592c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3b95dd-2qfpx" Oct 30 14:48:44.537599 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:44.537572 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/757b0c4a-6c88-4a4e-9aaa-04a17c7592c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-3b95dd-2qfpx\" (UID: \"757b0c4a-6c88-4a4e-9aaa-04a17c7592c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3b95dd-2qfpx" Oct 30 14:48:44.561381 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:44.561354 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwxsx\" (UniqueName: \"kubernetes.io/projected/757b0c4a-6c88-4a4e-9aaa-04a17c7592c5-kube-api-access-qwxsx\") pod \"maas-api-revoke-keys-e2e-ipp-b-3b95dd-2qfpx\" (UID: \"757b0c4a-6c88-4a4e-9aaa-04a17c7592c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3b95dd-2qfpx" Oct 30 14:48:44.622690 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:44.622665 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3b95dd-2qfpx" Oct 30 14:48:44.922833 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:44.922809 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3b95dd-2qfpx"] Oct 30 14:48:44.925146 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:48:44.925115 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod757b0c4a_6c88_4a4e_9aaa_04a17c7592c5.slice/crio-4027f9306e5f711b028bc61f9e348f7934060b6cfe5d9cb9a4124267829fba2b WatchSource:0}: Error finding container 4027f9306e5f711b028bc61f9e348f7934060b6cfe5d9cb9a4124267829fba2b: Status 404 returned error can't find the container with id 4027f9306e5f711b028bc61f9e348f7934060b6cfe5d9cb9a4124267829fba2b Oct 30 14:48:45.124259 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:45.124222 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:45.124713 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:45.124685 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3b95dd-2qfpx" event={"ID":"757b0c4a-6c88-4a4e-9aaa-04a17c7592c5","Type":"ContainerStarted","Data":"4027f9306e5f711b028bc61f9e348f7934060b6cfe5d9cb9a4124267829fba2b"} Oct 30 14:48:45.875435 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:45.875390 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:46.018825 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:46.018783 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:46.133537 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:46.133504 4982 generic.go:356] "Generic (PLEG): container finished" podID="757b0c4a-6c88-4a4e-9aaa-04a17c7592c5" containerID="f5b48ff42807153a83eb8dc1d90fc3c9e36a26a90877017134bead26bec531e5" exitCode=0 Oct 30 14:48:46.133773 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:46.133585 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3b95dd-2qfpx" event={"ID":"757b0c4a-6c88-4a4e-9aaa-04a17c7592c5","Type":"ContainerDied","Data":"f5b48ff42807153a83eb8dc1d90fc3c9e36a26a90877017134bead26bec531e5"} Oct 30 14:48:47.056829 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:47.056796 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:47.294284 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:47.294260 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3b95dd-2qfpx" Oct 30 14:48:47.364609 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:47.364538 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/757b0c4a-6c88-4a4e-9aaa-04a17c7592c5-kube-api-access-qwxsx\" (UniqueName: \"kubernetes.io/projected/757b0c4a-6c88-4a4e-9aaa-04a17c7592c5-kube-api-access-qwxsx\") pod \"757b0c4a-6c88-4a4e-9aaa-04a17c7592c5\" (UID: \"757b0c4a-6c88-4a4e-9aaa-04a17c7592c5\") " Oct 30 14:48:47.364771 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:47.364606 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/757b0c4a-6c88-4a4e-9aaa-04a17c7592c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/757b0c4a-6c88-4a4e-9aaa-04a17c7592c5-maas-api-xks-ca\") pod \"757b0c4a-6c88-4a4e-9aaa-04a17c7592c5\" (UID: \"757b0c4a-6c88-4a4e-9aaa-04a17c7592c5\") " Oct 30 14:48:47.365202 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:47.365176 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/757b0c4a-6c88-4a4e-9aaa-04a17c7592c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/757b0c4a-6c88-4a4e-9aaa-04a17c7592c5-maas-api-openshift-ca\") pod \"757b0c4a-6c88-4a4e-9aaa-04a17c7592c5\" (UID: \"757b0c4a-6c88-4a4e-9aaa-04a17c7592c5\") " Oct 30 14:48:47.365718 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:47.365668 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/757b0c4a-6c88-4a4e-9aaa-04a17c7592c5-maas-api-openshift-ca" pod "757b0c4a-6c88-4a4e-9aaa-04a17c7592c5" (UID: "757b0c4a-6c88-4a4e-9aaa-04a17c7592c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:48:47.367361 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:47.367205 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/757b0c4a-6c88-4a4e-9aaa-04a17c7592c5-maas-api-xks-ca" pod "757b0c4a-6c88-4a4e-9aaa-04a17c7592c5" (UID: "757b0c4a-6c88-4a4e-9aaa-04a17c7592c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:48:47.367361 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:47.367324 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/757b0c4a-6c88-4a4e-9aaa-04a17c7592c5-kube-api-access-qwxsx" pod "757b0c4a-6c88-4a4e-9aaa-04a17c7592c5" (UID: "757b0c4a-6c88-4a4e-9aaa-04a17c7592c5"). InnerVolumeSpecName "kube-api-access-qwxsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:48:47.467054 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:47.467017 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/757b0c4a-6c88-4a4e-9aaa-04a17c7592c5-maas-api-openshift-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:47.467054 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:47.467056 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwxsx\" (UniqueName: \"kubernetes.io/projected/757b0c4a-6c88-4a4e-9aaa-04a17c7592c5-kube-api-access-qwxsx\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:47.467248 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:47.467067 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/757b0c4a-6c88-4a4e-9aaa-04a17c7592c5-maas-api-xks-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:47.703806 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:47.703728 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:48.152613 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:48.152569 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3b95dd-2qfpx" event={"ID":"757b0c4a-6c88-4a4e-9aaa-04a17c7592c5","Type":"ContainerDied","Data":"4027f9306e5f711b028bc61f9e348f7934060b6cfe5d9cb9a4124267829fba2b"} Oct 30 14:48:48.152985 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:48.152624 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3b95dd-2qfpx" Oct 30 14:48:48.152985 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:48.152627 4982 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4027f9306e5f711b028bc61f9e348f7934060b6cfe5d9cb9a4124267829fba2b" Oct 30 14:48:48.844996 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:48.844446 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:48.985771 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:48.985709 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:48.996572 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:48.996549 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx" Oct 30 14:48:49.083197 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:49.083169 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19c9ad1b-96d7-41de-a3cc-0960ef9c31b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19c9ad1b-96d7-41de-a3cc-0960ef9c31b8-workload-certs\") pod \"19c9ad1b-96d7-41de-a3cc-0960ef9c31b8\" (UID: \"19c9ad1b-96d7-41de-a3cc-0960ef9c31b8\") " Oct 30 14:48:49.083355 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:49.083206 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/19c9ad1b-96d7-41de-a3cc-0960ef9c31b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19c9ad1b-96d7-41de-a3cc-0960ef9c31b8-istio-podinfo\") pod \"19c9ad1b-96d7-41de-a3cc-0960ef9c31b8\" (UID: \"19c9ad1b-96d7-41de-a3cc-0960ef9c31b8\") " Oct 30 14:48:49.083355 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:49.083323 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19c9ad1b-96d7-41de-a3cc-0960ef9c31b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19c9ad1b-96d7-41de-a3cc-0960ef9c31b8-istio-envoy\") pod \"19c9ad1b-96d7-41de-a3cc-0960ef9c31b8\" (UID: \"19c9ad1b-96d7-41de-a3cc-0960ef9c31b8\") " Oct 30 14:48:49.083483 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:49.083381 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19c9ad1b-96d7-41de-a3cc-0960ef9c31b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c9ad1b-96d7-41de-a3cc-0960ef9c31b8-workload-socket\") pod \"19c9ad1b-96d7-41de-a3cc-0960ef9c31b8\" (UID: \"19c9ad1b-96d7-41de-a3cc-0960ef9c31b8\") " Oct 30 14:48:49.083483 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:49.083426 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19c9ad1b-96d7-41de-a3cc-0960ef9c31b8-kube-api-access-cvc7w\" (UniqueName: \"kubernetes.io/projected/19c9ad1b-96d7-41de-a3cc-0960ef9c31b8-kube-api-access-cvc7w\") pod \"19c9ad1b-96d7-41de-a3cc-0960ef9c31b8\" (UID: \"19c9ad1b-96d7-41de-a3cc-0960ef9c31b8\") " Oct 30 14:48:49.083483 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:49.083463 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19c9ad1b-96d7-41de-a3cc-0960ef9c31b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c9ad1b-96d7-41de-a3cc-0960ef9c31b8-credential-socket\") pod \"19c9ad1b-96d7-41de-a3cc-0960ef9c31b8\" (UID: \"19c9ad1b-96d7-41de-a3cc-0960ef9c31b8\") " Oct 30 14:48:49.083634 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:49.083473 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19c9ad1b-96d7-41de-a3cc-0960ef9c31b8-workload-certs" pod "19c9ad1b-96d7-41de-a3cc-0960ef9c31b8" (UID: "19c9ad1b-96d7-41de-a3cc-0960ef9c31b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:48:49.083634 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:49.083505 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19c9ad1b-96d7-41de-a3cc-0960ef9c31b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19c9ad1b-96d7-41de-a3cc-0960ef9c31b8-istio-data\") pod \"19c9ad1b-96d7-41de-a3cc-0960ef9c31b8\" (UID: \"19c9ad1b-96d7-41de-a3cc-0960ef9c31b8\") " Oct 30 14:48:49.083634 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:49.083544 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19c9ad1b-96d7-41de-a3cc-0960ef9c31b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19c9ad1b-96d7-41de-a3cc-0960ef9c31b8-istiod-ca-cert\") pod \"19c9ad1b-96d7-41de-a3cc-0960ef9c31b8\" (UID: \"19c9ad1b-96d7-41de-a3cc-0960ef9c31b8\") " Oct 30 14:48:49.083634 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:49.083585 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19c9ad1b-96d7-41de-a3cc-0960ef9c31b8-istio-token\" (UniqueName: \"kubernetes.io/projected/19c9ad1b-96d7-41de-a3cc-0960ef9c31b8-istio-token\") pod \"19c9ad1b-96d7-41de-a3cc-0960ef9c31b8\" (UID: \"19c9ad1b-96d7-41de-a3cc-0960ef9c31b8\") " Oct 30 14:48:49.083904 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:49.083725 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19c9ad1b-96d7-41de-a3cc-0960ef9c31b8-workload-socket" pod "19c9ad1b-96d7-41de-a3cc-0960ef9c31b8" (UID: "19c9ad1b-96d7-41de-a3cc-0960ef9c31b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:48:49.083904 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:49.083837 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19c9ad1b-96d7-41de-a3cc-0960ef9c31b8-istio-data" pod "19c9ad1b-96d7-41de-a3cc-0960ef9c31b8" (UID: "19c9ad1b-96d7-41de-a3cc-0960ef9c31b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:48:49.084010 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:49.083942 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19c9ad1b-96d7-41de-a3cc-0960ef9c31b8-credential-socket" pod "19c9ad1b-96d7-41de-a3cc-0960ef9c31b8" (UID: "19c9ad1b-96d7-41de-a3cc-0960ef9c31b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:48:49.084180 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:49.084104 4982 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c9ad1b-96d7-41de-a3cc-0960ef9c31b8-workload-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:49.084180 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:49.084129 4982 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c9ad1b-96d7-41de-a3cc-0960ef9c31b8-credential-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:49.084180 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:49.084143 4982 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19c9ad1b-96d7-41de-a3cc-0960ef9c31b8-istio-data\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:49.084180 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:49.084156 4982 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19c9ad1b-96d7-41de-a3cc-0960ef9c31b8-workload-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:49.084441 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:49.084206 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19c9ad1b-96d7-41de-a3cc-0960ef9c31b8-istiod-ca-cert" pod "19c9ad1b-96d7-41de-a3cc-0960ef9c31b8" (UID: "19c9ad1b-96d7-41de-a3cc-0960ef9c31b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:48:49.085875 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:49.085848 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19c9ad1b-96d7-41de-a3cc-0960ef9c31b8-kube-api-access-cvc7w" pod "19c9ad1b-96d7-41de-a3cc-0960ef9c31b8" (UID: "19c9ad1b-96d7-41de-a3cc-0960ef9c31b8"). InnerVolumeSpecName "kube-api-access-cvc7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:48:49.086070 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:49.086047 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19c9ad1b-96d7-41de-a3cc-0960ef9c31b8-istio-envoy" pod "19c9ad1b-96d7-41de-a3cc-0960ef9c31b8" (UID: "19c9ad1b-96d7-41de-a3cc-0960ef9c31b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:48:49.086350 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:49.086333 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19c9ad1b-96d7-41de-a3cc-0960ef9c31b8-istio-token" pod "19c9ad1b-96d7-41de-a3cc-0960ef9c31b8" (UID: "19c9ad1b-96d7-41de-a3cc-0960ef9c31b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:48:49.086447 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:49.086388 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19c9ad1b-96d7-41de-a3cc-0960ef9c31b8-istio-podinfo" pod "19c9ad1b-96d7-41de-a3cc-0960ef9c31b8" (UID: "19c9ad1b-96d7-41de-a3cc-0960ef9c31b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:48:49.160455 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:49.160424 4982 generic.go:356] "Generic (PLEG): container finished" podID="19c9ad1b-96d7-41de-a3cc-0960ef9c31b8" containerID="0343b9bee9368249e29c13772a84c585addf6942ee565d663c9f8ff2541e86b5" exitCode=0 Oct 30 14:48:49.160876 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:49.160464 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx" event={"ID":"19c9ad1b-96d7-41de-a3cc-0960ef9c31b8","Type":"ContainerDied","Data":"0343b9bee9368249e29c13772a84c585addf6942ee565d663c9f8ff2541e86b5"} Oct 30 14:48:49.160876 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:49.160516 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx" event={"ID":"19c9ad1b-96d7-41de-a3cc-0960ef9c31b8","Type":"ContainerDied","Data":"92b27f1b26638f5ec435a3333ba39e849a39db954688f7492490450f14c3a5f2"} Oct 30 14:48:49.160876 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:49.160537 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx" Oct 30 14:48:49.160876 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:49.160545 4982 scope.go:122] "RemoveContainer" containerID="0343b9bee9368249e29c13772a84c585addf6942ee565d663c9f8ff2541e86b5" Oct 30 14:48:49.171718 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:49.171691 4982 scope.go:122] "RemoveContainer" containerID="0343b9bee9368249e29c13772a84c585addf6942ee565d663c9f8ff2541e86b5" Oct 30 14:48:49.171993 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:48:49.171971 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0343b9bee9368249e29c13772a84c585addf6942ee565d663c9f8ff2541e86b5\": container with ID starting with 0343b9bee9368249e29c13772a84c585addf6942ee565d663c9f8ff2541e86b5 not found: ID does not exist" containerID="0343b9bee9368249e29c13772a84c585addf6942ee565d663c9f8ff2541e86b5" Oct 30 14:48:49.172057 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:49.172001 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0343b9bee9368249e29c13772a84c585addf6942ee565d663c9f8ff2541e86b5"} err="failed to get container status \"0343b9bee9368249e29c13772a84c585addf6942ee565d663c9f8ff2541e86b5\": rpc error: code = NotFound desc = could not find container \"0343b9bee9368249e29c13772a84c585addf6942ee565d663c9f8ff2541e86b5\": container with ID starting with 0343b9bee9368249e29c13772a84c585addf6942ee565d663c9f8ff2541e86b5 not found: ID does not exist" Oct 30 14:48:49.185694 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:49.185658 4982 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19c9ad1b-96d7-41de-a3cc-0960ef9c31b8-istio-envoy\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:49.185694 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:49.185682 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvc7w\" (UniqueName: \"kubernetes.io/projected/19c9ad1b-96d7-41de-a3cc-0960ef9c31b8-kube-api-access-cvc7w\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:49.185694 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:49.185693 4982 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19c9ad1b-96d7-41de-a3cc-0960ef9c31b8-istiod-ca-cert\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:49.185860 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:49.185702 4982 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19c9ad1b-96d7-41de-a3cc-0960ef9c31b8-istio-token\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:49.185860 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:49.185710 4982 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19c9ad1b-96d7-41de-a3cc-0960ef9c31b8-istio-podinfo\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:49.220595 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:49.220569 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx"] Oct 30 14:48:49.226176 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:49.226156 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-f0e8b9-openshift-default-6f75d78b5f-9nrsx"] Oct 30 14:48:50.172690 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:50.172658 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:50.344346 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:50.344313 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:50.400001 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:50.399967 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19c9ad1b-96d7-41de-a3cc-0960ef9c31b8" path="/var/lib/kubelet/pods/19c9ad1b-96d7-41de-a3cc-0960ef9c31b8/volumes" Oct 30 14:48:50.445623 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:50.445529 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27"] Oct 30 14:48:50.446101 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:50.445817 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27" podUID="9b853a87-0adf-46f0-b45d-c17b69a68495" containerName="istio-proxy" containerID="cri-o://07975312845b9ab2a9b567ef181257a9f0085811e94eea60ceda6f26185542dc" gracePeriod=30 Oct 30 14:48:50.513787 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:50.513759 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:50.982555 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:50.982514 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:50.988527 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:50.988499 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ntlqq_dc967875-1546-40bb-8376-fde09416c204/console-operator/2.log" Oct 30 14:48:51.001124 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:51.001098 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhxjf_aca47fe8-7068-4743-b6ab-283ee0874f66/ovn-acl-logging/0.log" Oct 30 14:48:51.009471 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:51.009447 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ntlqq_dc967875-1546-40bb-8376-fde09416c204/console-operator/2.log" Oct 30 14:48:51.015132 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:51.015114 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhxjf_aca47fe8-7068-4743-b6ab-283ee0874f66/ovn-acl-logging/0.log" Oct 30 14:48:51.031879 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:51.031855 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e371c3-pjr6w"] Oct 30 14:48:51.032363 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:51.032347 4982 cpu_manager.go:414] "containerMap: removing container" podUID="757b0c4a-6c88-4a4e-9aaa-04a17c7592c5" containerName="revoke-keys" Oct 30 14:48:51.032421 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:51.032364 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="757b0c4a-6c88-4a4e-9aaa-04a17c7592c5" containerName="revoke-keys" Oct 30 14:48:51.032421 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:51.032406 4982 cpu_manager.go:414] "containerMap: removing container" podUID="19c9ad1b-96d7-41de-a3cc-0960ef9c31b8" containerName="istio-proxy" Oct 30 14:48:51.032421 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:51.032412 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19c9ad1b-96d7-41de-a3cc-0960ef9c31b8" containerName="istio-proxy" Oct 30 14:48:51.032520 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:51.032490 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="19c9ad1b-96d7-41de-a3cc-0960ef9c31b8" containerName="istio-proxy" Oct 30 14:48:51.032520 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:51.032500 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="757b0c4a-6c88-4a4e-9aaa-04a17c7592c5" containerName="revoke-keys" Oct 30 14:48:51.037271 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:51.037254 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e371c3-pjr6w" Oct 30 14:48:51.056044 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:51.056024 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:48:51.057656 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:51.057623 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e371c3-pjr6w"] Oct 30 14:48:51.068131 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:51.068109 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:48:51.106724 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:51.106695 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86df208c-1b1f-4068-8fed-3e43281724d0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-e371c3-pjr6w\" (UID: \"86df208c-1b1f-4068-8fed-3e43281724d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e371c3-pjr6w" Oct 30 14:48:51.106858 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:51.106736 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86df208c-1b1f-4068-8fed-3e43281724d0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-e371c3-pjr6w\" (UID: \"86df208c-1b1f-4068-8fed-3e43281724d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e371c3-pjr6w" Oct 30 14:48:51.106921 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:51.106896 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg96g\" (UniqueName: \"kubernetes.io/projected/86df208c-1b1f-4068-8fed-3e43281724d0-kube-api-access-bg96g\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-e371c3-pjr6w\" (UID: \"86df208c-1b1f-4068-8fed-3e43281724d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e371c3-pjr6w" Oct 30 14:48:51.207749 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:51.207709 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg96g\" (UniqueName: \"kubernetes.io/projected/86df208c-1b1f-4068-8fed-3e43281724d0-kube-api-access-bg96g\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-e371c3-pjr6w\" (UID: \"86df208c-1b1f-4068-8fed-3e43281724d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e371c3-pjr6w" Oct 30 14:48:51.208155 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:51.207920 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86df208c-1b1f-4068-8fed-3e43281724d0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-e371c3-pjr6w\" (UID: \"86df208c-1b1f-4068-8fed-3e43281724d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e371c3-pjr6w" Oct 30 14:48:51.208155 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:51.207950 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86df208c-1b1f-4068-8fed-3e43281724d0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-e371c3-pjr6w\" (UID: \"86df208c-1b1f-4068-8fed-3e43281724d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e371c3-pjr6w" Oct 30 14:48:51.208775 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:51.208750 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86df208c-1b1f-4068-8fed-3e43281724d0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-e371c3-pjr6w\" (UID: \"86df208c-1b1f-4068-8fed-3e43281724d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e371c3-pjr6w" Oct 30 14:48:51.210435 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:51.210407 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86df208c-1b1f-4068-8fed-3e43281724d0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-e371c3-pjr6w\" (UID: \"86df208c-1b1f-4068-8fed-3e43281724d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e371c3-pjr6w" Oct 30 14:48:51.234558 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:51.234499 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg96g\" (UniqueName: \"kubernetes.io/projected/86df208c-1b1f-4068-8fed-3e43281724d0-kube-api-access-bg96g\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-e371c3-pjr6w\" (UID: \"86df208c-1b1f-4068-8fed-3e43281724d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e371c3-pjr6w" Oct 30 14:48:51.350778 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:51.350747 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e371c3-pjr6w" Oct 30 14:48:51.669406 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:51.669374 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e371c3-pjr6w"] Oct 30 14:48:51.671049 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:48:51.671015 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86df208c_1b1f_4068_8fed_3e43281724d0.slice/crio-1b65fc4134e2b2e8d76ec475927ebd0c475df8df9adaa2543b03373f8fe68d23 WatchSource:0}: Error finding container 1b65fc4134e2b2e8d76ec475927ebd0c475df8df9adaa2543b03373f8fe68d23: Status 404 returned error can't find the container with id 1b65fc4134e2b2e8d76ec475927ebd0c475df8df9adaa2543b03373f8fe68d23 Oct 30 14:48:52.184418 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:52.184382 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e371c3-pjr6w" event={"ID":"86df208c-1b1f-4068-8fed-3e43281724d0","Type":"ContainerStarted","Data":"b6e2587053490171965f4d54570c62cc10dca4cfc29d50dabbfc8751c10d9ff6"} Oct 30 14:48:52.184582 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:52.184445 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e371c3-pjr6w" event={"ID":"86df208c-1b1f-4068-8fed-3e43281724d0","Type":"ContainerStarted","Data":"1b65fc4134e2b2e8d76ec475927ebd0c475df8df9adaa2543b03373f8fe68d23"} Oct 30 14:48:52.232621 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:52.232574 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e371c3-pjr6w" podStartSLOduration=2.232562279 podStartE2EDuration="2.232562279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:48:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:48:52.232314537 +0000 UTC m=+3601.976448580" watchObservedRunningTime="2026-10-30 14:48:52.232562279 +0000 UTC m=+3601.976696332" Oct 30 14:48:52.796038 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:52.796007 4982 scope.go:122] "RemoveContainer" containerID="ac9aeeb6af766fce81e3ca832637fb090e41d7229a4f2160126386801b259256" Oct 30 14:48:53.021316 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:53.020460 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:53.192708 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:53.192672 4982 generic.go:356] "Generic (PLEG): container finished" podID="86df208c-1b1f-4068-8fed-3e43281724d0" containerID="b6e2587053490171965f4d54570c62cc10dca4cfc29d50dabbfc8751c10d9ff6" exitCode=0 Oct 30 14:48:53.192874 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:53.192764 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e371c3-pjr6w" event={"ID":"86df208c-1b1f-4068-8fed-3e43281724d0","Type":"ContainerDied","Data":"b6e2587053490171965f4d54570c62cc10dca4cfc29d50dabbfc8751c10d9ff6"} Oct 30 14:48:54.346818 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:54.346789 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e371c3-pjr6w" Oct 30 14:48:54.449450 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:54.449376 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86df208c-1b1f-4068-8fed-3e43281724d0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86df208c-1b1f-4068-8fed-3e43281724d0-maas-api-xks-ca\") pod \"86df208c-1b1f-4068-8fed-3e43281724d0\" (UID: \"86df208c-1b1f-4068-8fed-3e43281724d0\") " Oct 30 14:48:54.449450 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:54.449414 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86df208c-1b1f-4068-8fed-3e43281724d0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86df208c-1b1f-4068-8fed-3e43281724d0-maas-api-openshift-ca\") pod \"86df208c-1b1f-4068-8fed-3e43281724d0\" (UID: \"86df208c-1b1f-4068-8fed-3e43281724d0\") " Oct 30 14:48:54.449610 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:54.449539 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86df208c-1b1f-4068-8fed-3e43281724d0-kube-api-access-bg96g\" (UniqueName: \"kubernetes.io/projected/86df208c-1b1f-4068-8fed-3e43281724d0-kube-api-access-bg96g\") pod \"86df208c-1b1f-4068-8fed-3e43281724d0\" (UID: \"86df208c-1b1f-4068-8fed-3e43281724d0\") " Oct 30 14:48:54.449861 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:54.449833 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86df208c-1b1f-4068-8fed-3e43281724d0-maas-api-openshift-ca" pod "86df208c-1b1f-4068-8fed-3e43281724d0" (UID: "86df208c-1b1f-4068-8fed-3e43281724d0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:48:54.449970 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:54.449937 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86df208c-1b1f-4068-8fed-3e43281724d0-maas-api-openshift-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:54.451399 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:54.451369 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86df208c-1b1f-4068-8fed-3e43281724d0-maas-api-xks-ca" pod "86df208c-1b1f-4068-8fed-3e43281724d0" (UID: "86df208c-1b1f-4068-8fed-3e43281724d0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:48:54.451569 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:54.451546 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86df208c-1b1f-4068-8fed-3e43281724d0-kube-api-access-bg96g" pod "86df208c-1b1f-4068-8fed-3e43281724d0" (UID: "86df208c-1b1f-4068-8fed-3e43281724d0"). InnerVolumeSpecName "kube-api-access-bg96g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:48:54.551803 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:54.551735 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86df208c-1b1f-4068-8fed-3e43281724d0-maas-api-xks-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:54.551803 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:54.551769 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bg96g\" (UniqueName: \"kubernetes.io/projected/86df208c-1b1f-4068-8fed-3e43281724d0-kube-api-access-bg96g\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:54.592525 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:54.592482 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:55.127966 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:55.127926 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:55.207347 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:55.207317 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e371c3-pjr6w" Oct 30 14:48:55.207535 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:55.207354 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e371c3-pjr6w" event={"ID":"86df208c-1b1f-4068-8fed-3e43281724d0","Type":"ContainerDied","Data":"1b65fc4134e2b2e8d76ec475927ebd0c475df8df9adaa2543b03373f8fe68d23"} Oct 30 14:48:55.207535 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:55.207432 4982 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b65fc4134e2b2e8d76ec475927ebd0c475df8df9adaa2543b03373f8fe68d23" Oct 30 14:48:55.275109 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:55.275073 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:55.724263 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:55.724204 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27" Oct 30 14:48:55.865380 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:55.865348 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b853a87-0adf-46f0-b45d-c17b69a68495-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b853a87-0adf-46f0-b45d-c17b69a68495-istio-envoy\") pod \"9b853a87-0adf-46f0-b45d-c17b69a68495\" (UID: \"9b853a87-0adf-46f0-b45d-c17b69a68495\") " Oct 30 14:48:55.865548 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:55.865401 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b853a87-0adf-46f0-b45d-c17b69a68495-kube-api-access-dr4qg\" (UniqueName: \"kubernetes.io/projected/9b853a87-0adf-46f0-b45d-c17b69a68495-kube-api-access-dr4qg\") pod \"9b853a87-0adf-46f0-b45d-c17b69a68495\" (UID: \"9b853a87-0adf-46f0-b45d-c17b69a68495\") " Oct 30 14:48:55.865617 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:55.865580 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9b853a87-0adf-46f0-b45d-c17b69a68495-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b853a87-0adf-46f0-b45d-c17b69a68495-istio-podinfo\") pod \"9b853a87-0adf-46f0-b45d-c17b69a68495\" (UID: \"9b853a87-0adf-46f0-b45d-c17b69a68495\") " Oct 30 14:48:55.865722 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:55.865620 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b853a87-0adf-46f0-b45d-c17b69a68495-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b853a87-0adf-46f0-b45d-c17b69a68495-workload-socket\") pod \"9b853a87-0adf-46f0-b45d-c17b69a68495\" (UID: \"9b853a87-0adf-46f0-b45d-c17b69a68495\") " Oct 30 14:48:55.865778 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:55.865726 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b853a87-0adf-46f0-b45d-c17b69a68495-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b853a87-0adf-46f0-b45d-c17b69a68495-workload-certs\") pod \"9b853a87-0adf-46f0-b45d-c17b69a68495\" (UID: \"9b853a87-0adf-46f0-b45d-c17b69a68495\") " Oct 30 14:48:55.865778 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:55.865766 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b853a87-0adf-46f0-b45d-c17b69a68495-istio-token\" (UniqueName: \"kubernetes.io/projected/9b853a87-0adf-46f0-b45d-c17b69a68495-istio-token\") pod \"9b853a87-0adf-46f0-b45d-c17b69a68495\" (UID: \"9b853a87-0adf-46f0-b45d-c17b69a68495\") " Oct 30 14:48:55.865871 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:55.865797 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b853a87-0adf-46f0-b45d-c17b69a68495-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b853a87-0adf-46f0-b45d-c17b69a68495-istiod-ca-cert\") pod \"9b853a87-0adf-46f0-b45d-c17b69a68495\" (UID: \"9b853a87-0adf-46f0-b45d-c17b69a68495\") " Oct 30 14:48:55.865871 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:55.865823 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b853a87-0adf-46f0-b45d-c17b69a68495-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b853a87-0adf-46f0-b45d-c17b69a68495-istio-data\") pod \"9b853a87-0adf-46f0-b45d-c17b69a68495\" (UID: \"9b853a87-0adf-46f0-b45d-c17b69a68495\") " Oct 30 14:48:55.865871 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:55.865848 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b853a87-0adf-46f0-b45d-c17b69a68495-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b853a87-0adf-46f0-b45d-c17b69a68495-credential-socket\") pod \"9b853a87-0adf-46f0-b45d-c17b69a68495\" (UID: \"9b853a87-0adf-46f0-b45d-c17b69a68495\") " Oct 30 14:48:55.866038 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:55.866000 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b853a87-0adf-46f0-b45d-c17b69a68495-workload-socket" pod "9b853a87-0adf-46f0-b45d-c17b69a68495" (UID: "9b853a87-0adf-46f0-b45d-c17b69a68495"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:48:55.866110 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:55.866044 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b853a87-0adf-46f0-b45d-c17b69a68495-workload-certs" pod "9b853a87-0adf-46f0-b45d-c17b69a68495" (UID: "9b853a87-0adf-46f0-b45d-c17b69a68495"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:48:55.866437 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:55.866177 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b853a87-0adf-46f0-b45d-c17b69a68495-istio-data" pod "9b853a87-0adf-46f0-b45d-c17b69a68495" (UID: "9b853a87-0adf-46f0-b45d-c17b69a68495"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:48:55.866437 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:55.866239 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b853a87-0adf-46f0-b45d-c17b69a68495-credential-socket" pod "9b853a87-0adf-46f0-b45d-c17b69a68495" (UID: "9b853a87-0adf-46f0-b45d-c17b69a68495"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:48:55.866437 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:55.866287 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b853a87-0adf-46f0-b45d-c17b69a68495-istiod-ca-cert" pod "9b853a87-0adf-46f0-b45d-c17b69a68495" (UID: "9b853a87-0adf-46f0-b45d-c17b69a68495"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:48:55.866437 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:55.866299 4982 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b853a87-0adf-46f0-b45d-c17b69a68495-workload-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:55.866437 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:55.866346 4982 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b853a87-0adf-46f0-b45d-c17b69a68495-istio-data\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:55.866437 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:55.866364 4982 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b853a87-0adf-46f0-b45d-c17b69a68495-workload-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:55.868032 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:55.867999 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b853a87-0adf-46f0-b45d-c17b69a68495-kube-api-access-dr4qg" pod "9b853a87-0adf-46f0-b45d-c17b69a68495" (UID: "9b853a87-0adf-46f0-b45d-c17b69a68495"). InnerVolumeSpecName "kube-api-access-dr4qg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:48:55.868765 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:55.868737 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b853a87-0adf-46f0-b45d-c17b69a68495-istio-podinfo" pod "9b853a87-0adf-46f0-b45d-c17b69a68495" (UID: "9b853a87-0adf-46f0-b45d-c17b69a68495"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:48:55.868765 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:55.868759 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b853a87-0adf-46f0-b45d-c17b69a68495-istio-token" pod "9b853a87-0adf-46f0-b45d-c17b69a68495" (UID: "9b853a87-0adf-46f0-b45d-c17b69a68495"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:48:55.868905 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:55.868760 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b853a87-0adf-46f0-b45d-c17b69a68495-istio-envoy" pod "9b853a87-0adf-46f0-b45d-c17b69a68495" (UID: "9b853a87-0adf-46f0-b45d-c17b69a68495"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:48:55.967031 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:55.967003 4982 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b853a87-0adf-46f0-b45d-c17b69a68495-istio-podinfo\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:55.967031 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:55.967031 4982 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b853a87-0adf-46f0-b45d-c17b69a68495-istio-token\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:55.967190 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:55.967040 4982 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b853a87-0adf-46f0-b45d-c17b69a68495-istiod-ca-cert\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:55.967190 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:55.967049 4982 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b853a87-0adf-46f0-b45d-c17b69a68495-credential-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:55.967190 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:55.967058 4982 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b853a87-0adf-46f0-b45d-c17b69a68495-istio-envoy\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:55.967190 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:55.967066 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dr4qg\" (UniqueName: \"kubernetes.io/projected/9b853a87-0adf-46f0-b45d-c17b69a68495-kube-api-access-dr4qg\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:48:56.215936 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:56.215902 4982 generic.go:356] "Generic (PLEG): container finished" podID="9b853a87-0adf-46f0-b45d-c17b69a68495" containerID="07975312845b9ab2a9b567ef181257a9f0085811e94eea60ceda6f26185542dc" exitCode=0 Oct 30 14:48:56.216114 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:56.215958 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27" Oct 30 14:48:56.216114 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:56.215986 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27" event={"ID":"9b853a87-0adf-46f0-b45d-c17b69a68495","Type":"ContainerDied","Data":"07975312845b9ab2a9b567ef181257a9f0085811e94eea60ceda6f26185542dc"} Oct 30 14:48:56.216114 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:56.216044 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27" event={"ID":"9b853a87-0adf-46f0-b45d-c17b69a68495","Type":"ContainerDied","Data":"8137093d76c5b51d2d6cdcfc7d27313f5dea91c5331556ce0f0a26d95dbb2749"} Oct 30 14:48:56.216114 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:56.216068 4982 scope.go:122] "RemoveContainer" containerID="07975312845b9ab2a9b567ef181257a9f0085811e94eea60ceda6f26185542dc" Oct 30 14:48:56.227200 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:56.227179 4982 scope.go:122] "RemoveContainer" containerID="f40b22d496fc456deef3f35cbfb2bd4fce960f0bd73d35e1396e9f5ea8ab6e7a" Oct 30 14:48:56.236262 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:56.236245 4982 scope.go:122] "RemoveContainer" containerID="07975312845b9ab2a9b567ef181257a9f0085811e94eea60ceda6f26185542dc" Oct 30 14:48:56.236530 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:48:56.236509 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07975312845b9ab2a9b567ef181257a9f0085811e94eea60ceda6f26185542dc\": container with ID starting with 07975312845b9ab2a9b567ef181257a9f0085811e94eea60ceda6f26185542dc not found: ID does not exist" containerID="07975312845b9ab2a9b567ef181257a9f0085811e94eea60ceda6f26185542dc" Oct 30 14:48:56.236578 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:56.236538 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07975312845b9ab2a9b567ef181257a9f0085811e94eea60ceda6f26185542dc"} err="failed to get container status \"07975312845b9ab2a9b567ef181257a9f0085811e94eea60ceda6f26185542dc\": rpc error: code = NotFound desc = could not find container \"07975312845b9ab2a9b567ef181257a9f0085811e94eea60ceda6f26185542dc\": container with ID starting with 07975312845b9ab2a9b567ef181257a9f0085811e94eea60ceda6f26185542dc not found: ID does not exist" Oct 30 14:48:56.236578 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:56.236554 4982 scope.go:122] "RemoveContainer" containerID="f40b22d496fc456deef3f35cbfb2bd4fce960f0bd73d35e1396e9f5ea8ab6e7a" Oct 30 14:48:56.236783 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:48:56.236763 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f40b22d496fc456deef3f35cbfb2bd4fce960f0bd73d35e1396e9f5ea8ab6e7a\": container with ID starting with f40b22d496fc456deef3f35cbfb2bd4fce960f0bd73d35e1396e9f5ea8ab6e7a not found: ID does not exist" containerID="f40b22d496fc456deef3f35cbfb2bd4fce960f0bd73d35e1396e9f5ea8ab6e7a" Oct 30 14:48:56.236831 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:56.236789 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f40b22d496fc456deef3f35cbfb2bd4fce960f0bd73d35e1396e9f5ea8ab6e7a"} err="failed to get container status \"f40b22d496fc456deef3f35cbfb2bd4fce960f0bd73d35e1396e9f5ea8ab6e7a\": rpc error: code = NotFound desc = could not find container \"f40b22d496fc456deef3f35cbfb2bd4fce960f0bd73d35e1396e9f5ea8ab6e7a\": container with ID starting with f40b22d496fc456deef3f35cbfb2bd4fce960f0bd73d35e1396e9f5ea8ab6e7a not found: ID does not exist" Oct 30 14:48:56.278028 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:56.278006 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27"] Oct 30 14:48:56.291653 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:56.291619 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-90b406-openshift-default-68c56c8459-bgw27"] Oct 30 14:48:56.398833 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:56.398806 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b853a87-0adf-46f0-b45d-c17b69a68495" path="/var/lib/kubelet/pods/9b853a87-0adf-46f0-b45d-c17b69a68495/volumes" Oct 30 14:48:56.667380 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:56.667348 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:56.817600 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:56.817561 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:56.959657 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:56.959530 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:57.163260 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:57.163224 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:57.306499 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:57.306402 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:57.542875 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:57.542843 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:57.692960 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:57.692925 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:58.015079 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:48:58.014988 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:48:58.169416 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:58.169386 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:59.375612 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:59.375568 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:59.483217 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:59.483163 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3b95dd-6969b8789f-cwgl2"] Oct 30 14:48:59.484296 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:59.483727 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b95dd-6969b8789f-cwgl2" podUID="ef2a7ae4-27bb-4854-9736-b9f765bcb83c" containerName="payload-pre-processing" containerID="cri-o://7fcc56797e6fb75ac4944dfa04509deedc4a5cb7837ba6b8bf0f26a8c88c4807" gracePeriod=30 Oct 30 14:48:59.573306 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:59.573276 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:59.736463 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:59.736433 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:59.757155 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:59.757133 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b95dd-6969b8789f-cwgl2" Oct 30 14:48:59.929191 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:59.929107 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef2a7ae4-27bb-4854-9736-b9f765bcb83c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef2a7ae4-27bb-4854-9736-b9f765bcb83c-plugins-config-volume\") pod \"ef2a7ae4-27bb-4854-9736-b9f765bcb83c\" (UID: \"ef2a7ae4-27bb-4854-9736-b9f765bcb83c\") " Oct 30 14:48:59.929191 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:59.929153 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef2a7ae4-27bb-4854-9736-b9f765bcb83c-kube-api-access-7jzml\" (UniqueName: \"kubernetes.io/projected/ef2a7ae4-27bb-4854-9736-b9f765bcb83c-kube-api-access-7jzml\") pod \"ef2a7ae4-27bb-4854-9736-b9f765bcb83c\" (UID: \"ef2a7ae4-27bb-4854-9736-b9f765bcb83c\") " Oct 30 14:48:59.929575 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:59.929543 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef2a7ae4-27bb-4854-9736-b9f765bcb83c-plugins-config-volume" pod "ef2a7ae4-27bb-4854-9736-b9f765bcb83c" (UID: "ef2a7ae4-27bb-4854-9736-b9f765bcb83c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:48:59.931400 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:48:59.931377 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef2a7ae4-27bb-4854-9736-b9f765bcb83c-kube-api-access-7jzml" pod "ef2a7ae4-27bb-4854-9736-b9f765bcb83c" (UID: "ef2a7ae4-27bb-4854-9736-b9f765bcb83c"). InnerVolumeSpecName "kube-api-access-7jzml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:49:00.031098 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:00.031061 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jzml\" (UniqueName: \"kubernetes.io/projected/ef2a7ae4-27bb-4854-9736-b9f765bcb83c-kube-api-access-7jzml\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:49:00.031098 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:00.031094 4982 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef2a7ae4-27bb-4854-9736-b9f765bcb83c-plugins-config-volume\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:49:00.135522 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:00.135489 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:00.243970 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:00.243881 4982 generic.go:356] "Generic (PLEG): container finished" podID="ef2a7ae4-27bb-4854-9736-b9f765bcb83c" containerID="7fcc56797e6fb75ac4944dfa04509deedc4a5cb7837ba6b8bf0f26a8c88c4807" exitCode=0 Oct 30 14:49:00.243970 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:00.243966 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b95dd-6969b8789f-cwgl2" Oct 30 14:49:00.244156 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:00.243960 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b95dd-6969b8789f-cwgl2" event={"ID":"ef2a7ae4-27bb-4854-9736-b9f765bcb83c","Type":"ContainerDied","Data":"7fcc56797e6fb75ac4944dfa04509deedc4a5cb7837ba6b8bf0f26a8c88c4807"} Oct 30 14:49:00.244156 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:00.244069 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b95dd-6969b8789f-cwgl2" event={"ID":"ef2a7ae4-27bb-4854-9736-b9f765bcb83c","Type":"ContainerDied","Data":"672ddc28eec223e9f517a6f88ffec52e9660762e5f2983f7effd78dc0bb98199"} Oct 30 14:49:00.244156 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:00.244101 4982 scope.go:122] "RemoveContainer" containerID="7fcc56797e6fb75ac4944dfa04509deedc4a5cb7837ba6b8bf0f26a8c88c4807" Oct 30 14:49:00.256859 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:00.256677 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3b95dd-2qfpx"] Oct 30 14:49:00.260075 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:00.260045 4982 scope.go:122] "RemoveContainer" containerID="7fcc56797e6fb75ac4944dfa04509deedc4a5cb7837ba6b8bf0f26a8c88c4807" Oct 30 14:49:00.260997 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:49:00.260956 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fcc56797e6fb75ac4944dfa04509deedc4a5cb7837ba6b8bf0f26a8c88c4807\": container with ID starting with 7fcc56797e6fb75ac4944dfa04509deedc4a5cb7837ba6b8bf0f26a8c88c4807 not found: ID does not exist" containerID="7fcc56797e6fb75ac4944dfa04509deedc4a5cb7837ba6b8bf0f26a8c88c4807" Oct 30 14:49:00.261103 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:00.261009 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fcc56797e6fb75ac4944dfa04509deedc4a5cb7837ba6b8bf0f26a8c88c4807"} err="failed to get container status \"7fcc56797e6fb75ac4944dfa04509deedc4a5cb7837ba6b8bf0f26a8c88c4807\": rpc error: code = NotFound desc = could not find container \"7fcc56797e6fb75ac4944dfa04509deedc4a5cb7837ba6b8bf0f26a8c88c4807\": container with ID starting with 7fcc56797e6fb75ac4944dfa04509deedc4a5cb7837ba6b8bf0f26a8c88c4807 not found: ID does not exist" Oct 30 14:49:00.266252 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:00.266227 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3b95dd-2qfpx"] Oct 30 14:49:00.325987 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:00.325962 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3b95dd-6969b8789f-cwgl2"] Oct 30 14:49:00.333925 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:00.333890 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:00.335548 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:00.335526 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3b95dd-6969b8789f-cwgl2"] Oct 30 14:49:00.401085 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:00.401061 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="757b0c4a-6c88-4a4e-9aaa-04a17c7592c5" path="/var/lib/kubelet/pods/757b0c4a-6c88-4a4e-9aaa-04a17c7592c5/volumes" Oct 30 14:49:00.401759 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:00.401739 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef2a7ae4-27bb-4854-9736-b9f765bcb83c" path="/var/lib/kubelet/pods/ef2a7ae4-27bb-4854-9736-b9f765bcb83c/volumes" Oct 30 14:49:00.784395 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:00.784354 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:01.659665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:01.659616 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:01.818389 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:01.818353 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:02.259121 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:02.259093 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:03.891574 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:03.891533 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:04.437314 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:04.437282 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:04.538362 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:04.538331 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-da78c4-87kkb"] Oct 30 14:49:04.552448 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:04.552399 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-da78c4-87kkb"] Oct 30 14:49:04.582421 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:04.582384 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:05.075174 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:05.075142 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:05.471430 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:05.471355 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:05.616414 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:05.616378 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:05.842606 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:05.842574 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:05.979470 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:05.979434 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:06.399087 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:06.399053 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aaecf278-2542-4af3-b82e-233e83c24e47" path="/var/lib/kubelet/pods/aaecf278-2542-4af3-b82e-233e83c24e47/volumes" Oct 30 14:49:06.589153 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:06.589086 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:08.247590 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:08.247542 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:08.398746 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:08.398709 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:08.576851 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:08.576819 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:08.731361 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:08.731326 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:08.993292 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:08.993211 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5704d6-vvj4s"] Oct 30 14:49:08.993924 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:08.993904 4982 cpu_manager.go:414] "containerMap: removing container" podUID="9b853a87-0adf-46f0-b45d-c17b69a68495" containerName="istio-proxy" Oct 30 14:49:08.993979 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:08.993927 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b853a87-0adf-46f0-b45d-c17b69a68495" containerName="istio-proxy" Oct 30 14:49:08.993979 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:08.993967 4982 cpu_manager.go:414] "containerMap: removing container" podUID="86df208c-1b1f-4068-8fed-3e43281724d0" containerName="revoke-keys" Oct 30 14:49:08.993979 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:08.993976 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86df208c-1b1f-4068-8fed-3e43281724d0" containerName="revoke-keys" Oct 30 14:49:08.994080 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:08.994014 4982 cpu_manager.go:414] "containerMap: removing container" podUID="ef2a7ae4-27bb-4854-9736-b9f765bcb83c" containerName="payload-pre-processing" Oct 30 14:49:08.994080 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:08.994022 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef2a7ae4-27bb-4854-9736-b9f765bcb83c" containerName="payload-pre-processing" Oct 30 14:49:08.994150 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:08.994144 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef2a7ae4-27bb-4854-9736-b9f765bcb83c" containerName="payload-pre-processing" Oct 30 14:49:08.994183 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:08.994158 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b853a87-0adf-46f0-b45d-c17b69a68495" containerName="istio-proxy" Oct 30 14:49:08.994183 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:08.994166 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b853a87-0adf-46f0-b45d-c17b69a68495" containerName="istio-proxy" Oct 30 14:49:08.994183 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:08.994181 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="86df208c-1b1f-4068-8fed-3e43281724d0" containerName="revoke-keys" Oct 30 14:49:08.999439 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:08.999418 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5704d6-vvj4s" Oct 30 14:49:09.018606 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:09.018582 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5704d6-vvj4s"] Oct 30 14:49:09.028494 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:09.028463 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:49:09.028701 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:09.028548 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:49:09.135974 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:09.135940 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c3e3a0f-43a7-41d3-acf7-c6299b1f5668-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-5704d6-vvj4s\" (UID: \"6c3e3a0f-43a7-41d3-acf7-c6299b1f5668\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5704d6-vvj4s" Oct 30 14:49:09.136114 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:09.135994 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j8w8\" (UniqueName: \"kubernetes.io/projected/6c3e3a0f-43a7-41d3-acf7-c6299b1f5668-kube-api-access-7j8w8\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-5704d6-vvj4s\" (UID: \"6c3e3a0f-43a7-41d3-acf7-c6299b1f5668\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5704d6-vvj4s" Oct 30 14:49:09.136114 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:09.136070 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c3e3a0f-43a7-41d3-acf7-c6299b1f5668-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-5704d6-vvj4s\" (UID: \"6c3e3a0f-43a7-41d3-acf7-c6299b1f5668\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5704d6-vvj4s" Oct 30 14:49:09.237076 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:09.237051 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c3e3a0f-43a7-41d3-acf7-c6299b1f5668-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-5704d6-vvj4s\" (UID: \"6c3e3a0f-43a7-41d3-acf7-c6299b1f5668\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5704d6-vvj4s" Oct 30 14:49:09.237220 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:09.237090 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j8w8\" (UniqueName: \"kubernetes.io/projected/6c3e3a0f-43a7-41d3-acf7-c6299b1f5668-kube-api-access-7j8w8\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-5704d6-vvj4s\" (UID: \"6c3e3a0f-43a7-41d3-acf7-c6299b1f5668\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5704d6-vvj4s" Oct 30 14:49:09.237220 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:09.237115 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c3e3a0f-43a7-41d3-acf7-c6299b1f5668-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-5704d6-vvj4s\" (UID: \"6c3e3a0f-43a7-41d3-acf7-c6299b1f5668\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5704d6-vvj4s" Oct 30 14:49:09.237756 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:09.237734 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c3e3a0f-43a7-41d3-acf7-c6299b1f5668-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-5704d6-vvj4s\" (UID: \"6c3e3a0f-43a7-41d3-acf7-c6299b1f5668\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5704d6-vvj4s" Oct 30 14:49:09.239470 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:09.239444 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c3e3a0f-43a7-41d3-acf7-c6299b1f5668-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-5704d6-vvj4s\" (UID: \"6c3e3a0f-43a7-41d3-acf7-c6299b1f5668\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5704d6-vvj4s" Oct 30 14:49:09.263309 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:09.263253 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j8w8\" (UniqueName: \"kubernetes.io/projected/6c3e3a0f-43a7-41d3-acf7-c6299b1f5668-kube-api-access-7j8w8\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-5704d6-vvj4s\" (UID: \"6c3e3a0f-43a7-41d3-acf7-c6299b1f5668\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5704d6-vvj4s" Oct 30 14:49:09.315218 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:09.315195 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5704d6-vvj4s" Oct 30 14:49:09.627345 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:09.627258 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5704d6-vvj4s"] Oct 30 14:49:09.630301 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:49:09.630271 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c3e3a0f_43a7_41d3_acf7_c6299b1f5668.slice/crio-86405d9aa26c7cceabcbb7a9f517c5e8c3578bd908ef72a6bd87ec173cb5f9ce WatchSource:0}: Error finding container 86405d9aa26c7cceabcbb7a9f517c5e8c3578bd908ef72a6bd87ec173cb5f9ce: Status 404 returned error can't find the container with id 86405d9aa26c7cceabcbb7a9f517c5e8c3578bd908ef72a6bd87ec173cb5f9ce Oct 30 14:49:10.307047 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:10.307008 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5704d6-vvj4s" event={"ID":"6c3e3a0f-43a7-41d3-acf7-c6299b1f5668","Type":"ContainerStarted","Data":"0ca0030799625f39defcd6871a7cdf06e3f2c6361168594c474aa860231a0a51"} Oct 30 14:49:10.307451 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:10.307107 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5704d6-vvj4s" event={"ID":"6c3e3a0f-43a7-41d3-acf7-c6299b1f5668","Type":"ContainerStarted","Data":"86405d9aa26c7cceabcbb7a9f517c5e8c3578bd908ef72a6bd87ec173cb5f9ce"} Oct 30 14:49:10.354437 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:10.354390 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5704d6-vvj4s" podStartSLOduration=2.354378532 podStartE2EDuration="2.354378532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:49:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:49:10.353804516 +0000 UTC m=+3620.097938556" watchObservedRunningTime="2026-10-30 14:49:10.354378532 +0000 UTC m=+3620.098512573" Oct 30 14:49:10.599203 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:10.599122 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:11.314700 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:11.314659 4982 generic.go:356] "Generic (PLEG): container finished" podID="6c3e3a0f-43a7-41d3-acf7-c6299b1f5668" containerID="0ca0030799625f39defcd6871a7cdf06e3f2c6361168594c474aa860231a0a51" exitCode=0 Oct 30 14:49:11.315109 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:11.314703 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5704d6-vvj4s" event={"ID":"6c3e3a0f-43a7-41d3-acf7-c6299b1f5668","Type":"ContainerDied","Data":"0ca0030799625f39defcd6871a7cdf06e3f2c6361168594c474aa860231a0a51"} Oct 30 14:49:12.227584 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:12.227525 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:12.502102 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:12.501965 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5704d6-vvj4s" Oct 30 14:49:12.572981 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:12.572949 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c3e3a0f-43a7-41d3-acf7-c6299b1f5668-kube-api-access-7j8w8\" (UniqueName: \"kubernetes.io/projected/6c3e3a0f-43a7-41d3-acf7-c6299b1f5668-kube-api-access-7j8w8\") pod \"6c3e3a0f-43a7-41d3-acf7-c6299b1f5668\" (UID: \"6c3e3a0f-43a7-41d3-acf7-c6299b1f5668\") " Oct 30 14:49:12.573153 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:12.573009 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c3e3a0f-43a7-41d3-acf7-c6299b1f5668-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c3e3a0f-43a7-41d3-acf7-c6299b1f5668-maas-api-xks-ca\") pod \"6c3e3a0f-43a7-41d3-acf7-c6299b1f5668\" (UID: \"6c3e3a0f-43a7-41d3-acf7-c6299b1f5668\") " Oct 30 14:49:12.573153 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:12.573048 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c3e3a0f-43a7-41d3-acf7-c6299b1f5668-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c3e3a0f-43a7-41d3-acf7-c6299b1f5668-maas-api-openshift-ca\") pod \"6c3e3a0f-43a7-41d3-acf7-c6299b1f5668\" (UID: \"6c3e3a0f-43a7-41d3-acf7-c6299b1f5668\") " Oct 30 14:49:12.573522 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:12.573493 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c3e3a0f-43a7-41d3-acf7-c6299b1f5668-maas-api-openshift-ca" pod "6c3e3a0f-43a7-41d3-acf7-c6299b1f5668" (UID: "6c3e3a0f-43a7-41d3-acf7-c6299b1f5668"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:49:12.575165 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:12.575136 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c3e3a0f-43a7-41d3-acf7-c6299b1f5668-maas-api-xks-ca" pod "6c3e3a0f-43a7-41d3-acf7-c6299b1f5668" (UID: "6c3e3a0f-43a7-41d3-acf7-c6299b1f5668"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:49:12.575300 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:12.575279 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c3e3a0f-43a7-41d3-acf7-c6299b1f5668-kube-api-access-7j8w8" pod "6c3e3a0f-43a7-41d3-acf7-c6299b1f5668" (UID: "6c3e3a0f-43a7-41d3-acf7-c6299b1f5668"). InnerVolumeSpecName "kube-api-access-7j8w8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:49:12.674777 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:12.674744 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c3e3a0f-43a7-41d3-acf7-c6299b1f5668-maas-api-openshift-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:49:12.674777 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:12.674773 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j8w8\" (UniqueName: \"kubernetes.io/projected/6c3e3a0f-43a7-41d3-acf7-c6299b1f5668-kube-api-access-7j8w8\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:49:12.674997 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:12.674786 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c3e3a0f-43a7-41d3-acf7-c6299b1f5668-maas-api-xks-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:49:12.906353 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:12.906317 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:13.032791 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:13.032747 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e371c3-pjr6w"] Oct 30 14:49:13.037238 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:13.037207 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e371c3-pjr6w"] Oct 30 14:49:13.063730 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:13.063699 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:13.328852 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:13.328828 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5704d6-vvj4s" Oct 30 14:49:13.329008 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:13.328823 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5704d6-vvj4s" event={"ID":"6c3e3a0f-43a7-41d3-acf7-c6299b1f5668","Type":"ContainerDied","Data":"86405d9aa26c7cceabcbb7a9f517c5e8c3578bd908ef72a6bd87ec173cb5f9ce"} Oct 30 14:49:13.329008 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:13.328953 4982 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86405d9aa26c7cceabcbb7a9f517c5e8c3578bd908ef72a6bd87ec173cb5f9ce" Oct 30 14:49:13.545866 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:13.545830 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:13.722250 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:13.722173 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:13.870298 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:13.870253 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:14.059215 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:14.059179 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:14.217615 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:14.217582 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:14.373773 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:14.373685 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:14.398809 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:14.398775 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86df208c-1b1f-4068-8fed-3e43281724d0" path="/var/lib/kubelet/pods/86df208c-1b1f-4068-8fed-3e43281724d0/volumes" Oct 30 14:49:15.036095 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:15.036065 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:16.799544 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:16.799512 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-75443e-6b6cd45ccb-fnql4"] Oct 30 14:49:16.835241 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:16.835210 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-75443e-7c475f6fbc-d7wdc"] Oct 30 14:49:16.975874 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:16.975837 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:17.127171 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:17.127135 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:18.922178 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:18.922114 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:19.644390 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:19.644356 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:19.796146 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:19.796111 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:20.912836 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:20.912797 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:21.800396 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:21.800351 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-75443e-6b6cd45ccb-fnql4" podUID="42ebc63e-3f5f-4aa3-b2cc-b9d92e3f9e0c" containerName="payload-processing" containerID="cri-o://504ade04f400781a538cd542d18de720c7dadb4351a7d4c6470084648d32f1a6" gracePeriod=25 Oct 30 14:49:21.836080 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:21.836027 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-75443e-7c475f6fbc-d7wdc" podUID="02ab3f3b-3aac-4acd-8340-3f63c789ec8c" containerName="payload-pre-processing" containerID="cri-o://287ae69eacdb8c07d50ca49b7e5029e6777a3d56d09f2668eaca1fb2aa8960a3" gracePeriod=25 Oct 30 14:49:21.991292 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:21.991265 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:22.140323 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:22.140279 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:22.349837 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:22.349799 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:22.737782 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:22.737730 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:24.904271 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:24.904228 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:25.356854 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:25.356822 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:25.511993 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:25.511949 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:26.916328 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:26.916282 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:28.011453 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:49:28.011418 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:49:28.651026 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:28.650986 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:29.221190 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:29.221153 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:29.349095 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:29.349037 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5704d6-vvj4s"] Oct 30 14:49:29.361608 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:29.361579 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5704d6-vvj4s"] Oct 30 14:49:29.498094 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:29.498018 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:29.872083 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:29.872052 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:30.399308 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:30.399262 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c3e3a0f-43a7-41d3-acf7-c6299b1f5668" path="/var/lib/kubelet/pods/6c3e3a0f-43a7-41d3-acf7-c6299b1f5668/volumes" Oct 30 14:49:31.001708 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:31.001608 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:31.030859 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:31.030821 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-75443e-6b6cd45ccb-fnql4" podUID="42ebc63e-3f5f-4aa3-b2cc-b9d92e3f9e0c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:49:31.031044 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:31.030860 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-75443e-7c475f6fbc-d7wdc" podUID="02ab3f3b-3aac-4acd-8340-3f63c789ec8c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:49:31.457729 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:31.457694 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:31.623861 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:31.623817 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:31.791251 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:31.791148 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:32.001549 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:32.001517 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:32.185825 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:32.185787 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:32.330672 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:32.330606 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:32.726433 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:32.726402 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:33.413060 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:33.412965 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:33.559892 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:33.559861 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:33.886529 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:33.886486 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:34.643691 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:34.643631 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:36.432184 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:36.432147 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:36.812033 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:36.811997 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:37.006451 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:37.006416 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:38.282095 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:38.282050 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:39.738868 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:39.738831 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:39.894544 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:39.894494 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:40.124278 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:40.124240 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:40.266963 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:40.266917 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:40.611492 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:40.611438 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:40.989376 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:40.989280 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:41.031100 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:41.031066 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-75443e-7c475f6fbc-d7wdc" podUID="02ab3f3b-3aac-4acd-8340-3f63c789ec8c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:49:41.031281 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:41.031130 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-75443e-6b6cd45ccb-fnql4" podUID="42ebc63e-3f5f-4aa3-b2cc-b9d92e3f9e0c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:49:41.241745 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:41.241625 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:41.433976 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:41.433941 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:42.140745 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:42.140716 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-75443e-6b6cd45ccb-fnql4" Oct 30 14:49:42.171301 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:42.171280 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-75443e-7c475f6fbc-d7wdc" Oct 30 14:49:42.210238 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:42.210216 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02ab3f3b-3aac-4acd-8340-3f63c789ec8c-kube-api-access-fxvj6\" (UniqueName: \"kubernetes.io/projected/02ab3f3b-3aac-4acd-8340-3f63c789ec8c-kube-api-access-fxvj6\") pod \"02ab3f3b-3aac-4acd-8340-3f63c789ec8c\" (UID: \"02ab3f3b-3aac-4acd-8340-3f63c789ec8c\") " Oct 30 14:49:42.210357 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:42.210318 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42ebc63e-3f5f-4aa3-b2cc-b9d92e3f9e0c-kube-api-access-sg22h\" (UniqueName: \"kubernetes.io/projected/42ebc63e-3f5f-4aa3-b2cc-b9d92e3f9e0c-kube-api-access-sg22h\") pod \"42ebc63e-3f5f-4aa3-b2cc-b9d92e3f9e0c\" (UID: \"42ebc63e-3f5f-4aa3-b2cc-b9d92e3f9e0c\") " Oct 30 14:49:42.210422 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:42.210365 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02ab3f3b-3aac-4acd-8340-3f63c789ec8c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02ab3f3b-3aac-4acd-8340-3f63c789ec8c-plugins-config-volume\") pod \"02ab3f3b-3aac-4acd-8340-3f63c789ec8c\" (UID: \"02ab3f3b-3aac-4acd-8340-3f63c789ec8c\") " Oct 30 14:49:42.210487 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:42.210423 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42ebc63e-3f5f-4aa3-b2cc-b9d92e3f9e0c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42ebc63e-3f5f-4aa3-b2cc-b9d92e3f9e0c-plugins-config-volume\") pod \"42ebc63e-3f5f-4aa3-b2cc-b9d92e3f9e0c\" (UID: \"42ebc63e-3f5f-4aa3-b2cc-b9d92e3f9e0c\") " Oct 30 14:49:42.210928 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:42.210900 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02ab3f3b-3aac-4acd-8340-3f63c789ec8c-plugins-config-volume" pod "02ab3f3b-3aac-4acd-8340-3f63c789ec8c" (UID: "02ab3f3b-3aac-4acd-8340-3f63c789ec8c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:49:42.211096 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:42.211004 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42ebc63e-3f5f-4aa3-b2cc-b9d92e3f9e0c-plugins-config-volume" pod "42ebc63e-3f5f-4aa3-b2cc-b9d92e3f9e0c" (UID: "42ebc63e-3f5f-4aa3-b2cc-b9d92e3f9e0c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:49:42.212458 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:42.212433 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42ebc63e-3f5f-4aa3-b2cc-b9d92e3f9e0c-kube-api-access-sg22h" pod "42ebc63e-3f5f-4aa3-b2cc-b9d92e3f9e0c" (UID: "42ebc63e-3f5f-4aa3-b2cc-b9d92e3f9e0c"). InnerVolumeSpecName "kube-api-access-sg22h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:49:42.212538 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:42.212520 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02ab3f3b-3aac-4acd-8340-3f63c789ec8c-kube-api-access-fxvj6" pod "02ab3f3b-3aac-4acd-8340-3f63c789ec8c" (UID: "02ab3f3b-3aac-4acd-8340-3f63c789ec8c"). InnerVolumeSpecName "kube-api-access-fxvj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:49:42.311547 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:42.311513 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxvj6\" (UniqueName: \"kubernetes.io/projected/02ab3f3b-3aac-4acd-8340-3f63c789ec8c-kube-api-access-fxvj6\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:49:42.311547 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:42.311544 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg22h\" (UniqueName: \"kubernetes.io/projected/42ebc63e-3f5f-4aa3-b2cc-b9d92e3f9e0c-kube-api-access-sg22h\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:49:42.311547 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:42.311553 4982 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02ab3f3b-3aac-4acd-8340-3f63c789ec8c-plugins-config-volume\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:49:42.311845 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:42.311562 4982 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42ebc63e-3f5f-4aa3-b2cc-b9d92e3f9e0c-plugins-config-volume\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:49:42.311845 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:42.311630 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:42.502524 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:42.502488 4982 generic.go:356] "Generic (PLEG): container finished" podID="02ab3f3b-3aac-4acd-8340-3f63c789ec8c" containerID="287ae69eacdb8c07d50ca49b7e5029e6777a3d56d09f2668eaca1fb2aa8960a3" exitCode=0 Oct 30 14:49:42.502710 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:42.502531 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-75443e-7c475f6fbc-d7wdc" Oct 30 14:49:42.502710 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:42.502538 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-75443e-7c475f6fbc-d7wdc" event={"ID":"02ab3f3b-3aac-4acd-8340-3f63c789ec8c","Type":"ContainerDied","Data":"287ae69eacdb8c07d50ca49b7e5029e6777a3d56d09f2668eaca1fb2aa8960a3"} Oct 30 14:49:42.502710 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:42.502600 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-75443e-7c475f6fbc-d7wdc" event={"ID":"02ab3f3b-3aac-4acd-8340-3f63c789ec8c","Type":"ContainerDied","Data":"3b450840fb30f2d4d0edb16c61782617308c248d6aa7cb2345bec420b69e157f"} Oct 30 14:49:42.502710 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:42.502627 4982 scope.go:122] "RemoveContainer" containerID="287ae69eacdb8c07d50ca49b7e5029e6777a3d56d09f2668eaca1fb2aa8960a3" Oct 30 14:49:42.504794 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:42.504597 4982 generic.go:356] "Generic (PLEG): container finished" podID="42ebc63e-3f5f-4aa3-b2cc-b9d92e3f9e0c" containerID="504ade04f400781a538cd542d18de720c7dadb4351a7d4c6470084648d32f1a6" exitCode=0 Oct 30 14:49:42.506226 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:42.505049 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-75443e-6b6cd45ccb-fnql4" event={"ID":"42ebc63e-3f5f-4aa3-b2cc-b9d92e3f9e0c","Type":"ContainerDied","Data":"504ade04f400781a538cd542d18de720c7dadb4351a7d4c6470084648d32f1a6"} Oct 30 14:49:42.506226 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:42.505095 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-75443e-6b6cd45ccb-fnql4" event={"ID":"42ebc63e-3f5f-4aa3-b2cc-b9d92e3f9e0c","Type":"ContainerDied","Data":"b6365238ae88307a361849612b1244e0ca1f9229e749bece7d7da00fb9945edb"} Oct 30 14:49:42.506226 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:42.505208 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-75443e-6b6cd45ccb-fnql4" Oct 30 14:49:42.514350 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:42.514331 4982 scope.go:122] "RemoveContainer" containerID="287ae69eacdb8c07d50ca49b7e5029e6777a3d56d09f2668eaca1fb2aa8960a3" Oct 30 14:49:42.514691 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:49:42.514665 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"287ae69eacdb8c07d50ca49b7e5029e6777a3d56d09f2668eaca1fb2aa8960a3\": container with ID starting with 287ae69eacdb8c07d50ca49b7e5029e6777a3d56d09f2668eaca1fb2aa8960a3 not found: ID does not exist" containerID="287ae69eacdb8c07d50ca49b7e5029e6777a3d56d09f2668eaca1fb2aa8960a3" Oct 30 14:49:42.514778 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:42.514701 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"287ae69eacdb8c07d50ca49b7e5029e6777a3d56d09f2668eaca1fb2aa8960a3"} err="failed to get container status \"287ae69eacdb8c07d50ca49b7e5029e6777a3d56d09f2668eaca1fb2aa8960a3\": rpc error: code = NotFound desc = could not find container \"287ae69eacdb8c07d50ca49b7e5029e6777a3d56d09f2668eaca1fb2aa8960a3\": container with ID starting with 287ae69eacdb8c07d50ca49b7e5029e6777a3d56d09f2668eaca1fb2aa8960a3 not found: ID does not exist" Oct 30 14:49:42.514778 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:42.514726 4982 scope.go:122] "RemoveContainer" containerID="504ade04f400781a538cd542d18de720c7dadb4351a7d4c6470084648d32f1a6" Oct 30 14:49:42.523582 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:42.523563 4982 scope.go:122] "RemoveContainer" containerID="504ade04f400781a538cd542d18de720c7dadb4351a7d4c6470084648d32f1a6" Oct 30 14:49:42.523864 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:49:42.523843 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"504ade04f400781a538cd542d18de720c7dadb4351a7d4c6470084648d32f1a6\": container with ID starting with 504ade04f400781a538cd542d18de720c7dadb4351a7d4c6470084648d32f1a6 not found: ID does not exist" containerID="504ade04f400781a538cd542d18de720c7dadb4351a7d4c6470084648d32f1a6" Oct 30 14:49:42.523941 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:42.523872 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"504ade04f400781a538cd542d18de720c7dadb4351a7d4c6470084648d32f1a6"} err="failed to get container status \"504ade04f400781a538cd542d18de720c7dadb4351a7d4c6470084648d32f1a6\": rpc error: code = NotFound desc = could not find container \"504ade04f400781a538cd542d18de720c7dadb4351a7d4c6470084648d32f1a6\": container with ID starting with 504ade04f400781a538cd542d18de720c7dadb4351a7d4c6470084648d32f1a6 not found: ID does not exist" Oct 30 14:49:42.560440 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:42.560416 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-75443e-6b6cd45ccb-fnql4"] Oct 30 14:49:42.572696 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:42.572672 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-75443e-6b6cd45ccb-fnql4"] Oct 30 14:49:42.605702 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:42.605678 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-75443e-7c475f6fbc-d7wdc"] Oct 30 14:49:42.609474 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:42.609448 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-75443e-7c475f6fbc-d7wdc"] Oct 30 14:49:44.273840 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:44.273803 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:44.398976 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:44.398944 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02ab3f3b-3aac-4acd-8340-3f63c789ec8c" path="/var/lib/kubelet/pods/02ab3f3b-3aac-4acd-8340-3f63c789ec8c/volumes" Oct 30 14:49:44.399312 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:44.399297 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42ebc63e-3f5f-4aa3-b2cc-b9d92e3f9e0c" path="/var/lib/kubelet/pods/42ebc63e-3f5f-4aa3-b2cc-b9d92e3f9e0c/volumes" Oct 30 14:49:45.559846 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:45.559816 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9"] Oct 30 14:49:45.669073 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:45.669034 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:46.449613 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:46.449519 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:47.158065 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:47.158033 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:47.821119 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:47.821081 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:48.658935 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:48.658906 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:48.801905 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:48.801867 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:49.287432 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:49.287396 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:50.863613 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:50.863579 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:51.410596 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:51.410560 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:51.568051 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:51.568021 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:52.222873 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:52.222838 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:52.518890 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:52.518813 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:52.669097 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:52.669066 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:52.851594 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:52.851565 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:52.884160 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:52.884130 4982 scope.go:122] "RemoveContainer" containerID="3268e201a8fef1d71f2aa8aa6e1852040a8432144c145298c5afdbcb6067335a" Oct 30 14:49:52.894601 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:52.894576 4982 scope.go:122] "RemoveContainer" containerID="82359b46efd57ac7d58e57ca35221ea9e544965124dda220b3084916fe0a3c22" Oct 30 14:49:52.903880 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:52.903860 4982 scope.go:122] "RemoveContainer" containerID="b0a4f021c4c7a943ae2d1067b7fdb54688a2a26e00010134a39bbe883bcc33c8" Oct 30 14:49:53.247001 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:53.246914 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:54.671001 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:54.670967 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:56.416275 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:56.416236 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:56.574215 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:56.574180 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:56.755727 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:56.755650 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:56.935109 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:56.935075 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:57.432049 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:57.432013 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:58.012029 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:49:58.011990 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:49:58.171490 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:58.171441 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:58.321830 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:58.321789 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:58.799704 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:49:58.799580 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:00.573767 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:00.573731 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:00.580285 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:00.580250 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" podUID="946696b9-91bd-437a-9f23-c16b73a5a9a6" containerName="main" containerID="cri-o://9f40a74941b4c90cad3417db554d163394dbd2ddd03fc61e55b86e2aec49c0c2" gracePeriod=45 Oct 30 14:50:00.854109 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:00.854087 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" Oct 30 14:50:00.993347 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:00.993313 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/946696b9-91bd-437a-9f23-c16b73a5a9a6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/946696b9-91bd-437a-9f23-c16b73a5a9a6-tmp-dir\") pod \"946696b9-91bd-437a-9f23-c16b73a5a9a6\" (UID: \"946696b9-91bd-437a-9f23-c16b73a5a9a6\") " Oct 30 14:50:00.993347 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:00.993350 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/946696b9-91bd-437a-9f23-c16b73a5a9a6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/946696b9-91bd-437a-9f23-c16b73a5a9a6-dshm\") pod \"946696b9-91bd-437a-9f23-c16b73a5a9a6\" (UID: \"946696b9-91bd-437a-9f23-c16b73a5a9a6\") " Oct 30 14:50:00.993608 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:00.993415 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/946696b9-91bd-437a-9f23-c16b73a5a9a6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/946696b9-91bd-437a-9f23-c16b73a5a9a6-model-cache\") pod \"946696b9-91bd-437a-9f23-c16b73a5a9a6\" (UID: \"946696b9-91bd-437a-9f23-c16b73a5a9a6\") " Oct 30 14:50:00.993608 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:00.993474 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/946696b9-91bd-437a-9f23-c16b73a5a9a6-kube-api-access-l8m8h\" (UniqueName: \"kubernetes.io/projected/946696b9-91bd-437a-9f23-c16b73a5a9a6-kube-api-access-l8m8h\") pod \"946696b9-91bd-437a-9f23-c16b73a5a9a6\" (UID: \"946696b9-91bd-437a-9f23-c16b73a5a9a6\") " Oct 30 14:50:00.993608 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:00.993535 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/946696b9-91bd-437a-9f23-c16b73a5a9a6-tls-certs\" (UniqueName: \"kubernetes.io/secret/946696b9-91bd-437a-9f23-c16b73a5a9a6-tls-certs\") pod \"946696b9-91bd-437a-9f23-c16b73a5a9a6\" (UID: \"946696b9-91bd-437a-9f23-c16b73a5a9a6\") " Oct 30 14:50:00.993608 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:00.993597 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/946696b9-91bd-437a-9f23-c16b73a5a9a6-home\" (UniqueName: \"kubernetes.io/empty-dir/946696b9-91bd-437a-9f23-c16b73a5a9a6-home\") pod \"946696b9-91bd-437a-9f23-c16b73a5a9a6\" (UID: \"946696b9-91bd-437a-9f23-c16b73a5a9a6\") " Oct 30 14:50:00.993880 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:00.993621 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/946696b9-91bd-437a-9f23-c16b73a5a9a6-tmp-dir" pod "946696b9-91bd-437a-9f23-c16b73a5a9a6" (UID: "946696b9-91bd-437a-9f23-c16b73a5a9a6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:00.994364 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:00.993997 4982 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/946696b9-91bd-437a-9f23-c16b73a5a9a6-tmp-dir\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:00.994364 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:00.994258 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/946696b9-91bd-437a-9f23-c16b73a5a9a6-model-cache" pod "946696b9-91bd-437a-9f23-c16b73a5a9a6" (UID: "946696b9-91bd-437a-9f23-c16b73a5a9a6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:00.994605 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:00.994583 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/946696b9-91bd-437a-9f23-c16b73a5a9a6-home" pod "946696b9-91bd-437a-9f23-c16b73a5a9a6" (UID: "946696b9-91bd-437a-9f23-c16b73a5a9a6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:00.996720 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:00.996555 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/946696b9-91bd-437a-9f23-c16b73a5a9a6-dshm" pod "946696b9-91bd-437a-9f23-c16b73a5a9a6" (UID: "946696b9-91bd-437a-9f23-c16b73a5a9a6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:00.996720 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:00.996687 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/946696b9-91bd-437a-9f23-c16b73a5a9a6-kube-api-access-l8m8h" pod "946696b9-91bd-437a-9f23-c16b73a5a9a6" (UID: "946696b9-91bd-437a-9f23-c16b73a5a9a6"). InnerVolumeSpecName "kube-api-access-l8m8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:50:00.997916 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:00.997890 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/946696b9-91bd-437a-9f23-c16b73a5a9a6-tls-certs" pod "946696b9-91bd-437a-9f23-c16b73a5a9a6" (UID: "946696b9-91bd-437a-9f23-c16b73a5a9a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:50:01.029899 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:01.029870 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:01.095081 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:01.094933 4982 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/946696b9-91bd-437a-9f23-c16b73a5a9a6-tls-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:01.095081 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:01.094967 4982 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/946696b9-91bd-437a-9f23-c16b73a5a9a6-home\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:01.095081 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:01.094981 4982 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/946696b9-91bd-437a-9f23-c16b73a5a9a6-dshm\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:01.095081 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:01.094993 4982 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/946696b9-91bd-437a-9f23-c16b73a5a9a6-model-cache\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:01.095081 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:01.095007 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8m8h\" (UniqueName: \"kubernetes.io/projected/946696b9-91bd-437a-9f23-c16b73a5a9a6-kube-api-access-l8m8h\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:01.176447 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:01.176410 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:01.328167 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:01.328118 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:01.620276 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:01.620234 4982 generic.go:356] "Generic (PLEG): container finished" podID="946696b9-91bd-437a-9f23-c16b73a5a9a6" containerID="9f40a74941b4c90cad3417db554d163394dbd2ddd03fc61e55b86e2aec49c0c2" exitCode=0 Oct 30 14:50:01.620688 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:01.620285 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" Oct 30 14:50:01.620688 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:01.620335 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" event={"ID":"946696b9-91bd-437a-9f23-c16b73a5a9a6","Type":"ContainerDied","Data":"9f40a74941b4c90cad3417db554d163394dbd2ddd03fc61e55b86e2aec49c0c2"} Oct 30 14:50:01.620688 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:01.620413 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9" event={"ID":"946696b9-91bd-437a-9f23-c16b73a5a9a6","Type":"ContainerDied","Data":"502f1ed5b19d3c740f6785d767cdbfdba64fbe3caeeb768085ed71cc0c6e43cf"} Oct 30 14:50:01.620688 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:01.620441 4982 scope.go:122] "RemoveContainer" containerID="9f40a74941b4c90cad3417db554d163394dbd2ddd03fc61e55b86e2aec49c0c2" Oct 30 14:50:01.631417 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:01.631399 4982 scope.go:122] "RemoveContainer" containerID="9f40a74941b4c90cad3417db554d163394dbd2ddd03fc61e55b86e2aec49c0c2" Oct 30 14:50:01.631737 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:50:01.631708 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f40a74941b4c90cad3417db554d163394dbd2ddd03fc61e55b86e2aec49c0c2\": container with ID starting with 9f40a74941b4c90cad3417db554d163394dbd2ddd03fc61e55b86e2aec49c0c2 not found: ID does not exist" containerID="9f40a74941b4c90cad3417db554d163394dbd2ddd03fc61e55b86e2aec49c0c2" Oct 30 14:50:01.631825 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:01.631751 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f40a74941b4c90cad3417db554d163394dbd2ddd03fc61e55b86e2aec49c0c2"} err="failed to get container status \"9f40a74941b4c90cad3417db554d163394dbd2ddd03fc61e55b86e2aec49c0c2\": rpc error: code = NotFound desc = could not find container \"9f40a74941b4c90cad3417db554d163394dbd2ddd03fc61e55b86e2aec49c0c2\": container with ID starting with 9f40a74941b4c90cad3417db554d163394dbd2ddd03fc61e55b86e2aec49c0c2 not found: ID does not exist" Oct 30 14:50:01.679771 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:01.679742 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9"] Oct 30 14:50:01.692173 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:01.692144 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-2df0ab/premium-w0-2df0ab-kserve-85cdc8bbff-gr2z9"] Oct 30 14:50:01.743707 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:01.743672 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:01.898756 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:01.895377 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:02.401466 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:02.401431 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="946696b9-91bd-437a-9f23-c16b73a5a9a6" path="/var/lib/kubelet/pods/946696b9-91bd-437a-9f23-c16b73a5a9a6/volumes" Oct 30 14:50:02.421432 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:02.421398 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:02.589313 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:02.589279 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:03.895168 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:03.895120 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:05.668181 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:05.668151 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:07.641846 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:07.641803 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:08.172241 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:08.172204 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:08.317319 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:08.317284 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:09.696484 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:09.696451 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:09.917136 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:09.917096 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:10.064145 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:10.064071 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:10.209116 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:10.209077 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b"] Oct 30 14:50:10.209363 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:10.209332 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b" podUID="52b260c5-122f-4e2a-8344-56d8efbb66f6" containerName="istio-proxy" containerID="cri-o://f581e9dc825db0ec024b40e0f6cf9f589ec9508428476950a821eb17bb42d707" gracePeriod=30 Oct 30 14:50:10.254261 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:10.254235 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:10.407456 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:10.407375 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:10.548085 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:10.548050 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b"] Oct 30 14:50:10.548322 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:10.548279 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b" podUID="c99896fe-79b1-4368-aeb5-a81182e7ae13" containerName="istio-proxy" containerID="cri-o://8702c319b369afd4d5d80fad6ade9493e1cdb09f96cbf3a40b0cdb727cf5759d" gracePeriod=30 Oct 30 14:50:10.616720 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:10.616693 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:10.782410 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:10.782239 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:11.089053 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:11.089012 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:11.846344 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:11.846308 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:13.454438 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:13.454397 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-2df0ab-58454c8987-cjnqk"] Oct 30 14:50:13.454923 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:13.454615 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2df0ab-58454c8987-cjnqk" podUID="757d5732-c143-453c-9149-5744017ea635" containerName="maas-api" containerID="cri-o://650906439f75a7905bac8c85cda95633476909dd2e6361fa52494a36efd02472" gracePeriod=30 Oct 30 14:50:13.556934 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:13.556899 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:13.712230 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:13.712074 4982 generic.go:356] "Generic (PLEG): container finished" podID="757d5732-c143-453c-9149-5744017ea635" containerID="650906439f75a7905bac8c85cda95633476909dd2e6361fa52494a36efd02472" exitCode=0 Oct 30 14:50:13.712230 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:13.712140 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2df0ab-58454c8987-cjnqk" event={"ID":"757d5732-c143-453c-9149-5744017ea635","Type":"ContainerDied","Data":"650906439f75a7905bac8c85cda95633476909dd2e6361fa52494a36efd02472"} Oct 30 14:50:13.744176 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:13.744151 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2df0ab-58454c8987-cjnqk" Oct 30 14:50:13.832429 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:13.832363 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/757d5732-c143-453c-9149-5744017ea635-kube-api-access-t789t\" (UniqueName: \"kubernetes.io/projected/757d5732-c143-453c-9149-5744017ea635-kube-api-access-t789t\") pod \"757d5732-c143-453c-9149-5744017ea635\" (UID: \"757d5732-c143-453c-9149-5744017ea635\") " Oct 30 14:50:13.832429 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:13.832430 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/757d5732-c143-453c-9149-5744017ea635-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/757d5732-c143-453c-9149-5744017ea635-openshift-service-ca\") pod \"757d5732-c143-453c-9149-5744017ea635\" (UID: \"757d5732-c143-453c-9149-5744017ea635\") " Oct 30 14:50:13.832750 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:13.832597 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/757d5732-c143-453c-9149-5744017ea635-metrics-certs\" (UniqueName: \"kubernetes.io/secret/757d5732-c143-453c-9149-5744017ea635-metrics-certs\") pod \"757d5732-c143-453c-9149-5744017ea635\" (UID: \"757d5732-c143-453c-9149-5744017ea635\") " Oct 30 14:50:13.832750 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:13.832627 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/757d5732-c143-453c-9149-5744017ea635-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/757d5732-c143-453c-9149-5744017ea635-maas-api-tls\") pod \"757d5732-c143-453c-9149-5744017ea635\" (UID: \"757d5732-c143-453c-9149-5744017ea635\") " Oct 30 14:50:13.832909 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:13.832877 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/757d5732-c143-453c-9149-5744017ea635-openshift-service-ca" pod "757d5732-c143-453c-9149-5744017ea635" (UID: "757d5732-c143-453c-9149-5744017ea635"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:50:13.834921 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:13.834884 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/757d5732-c143-453c-9149-5744017ea635-maas-api-tls" pod "757d5732-c143-453c-9149-5744017ea635" (UID: "757d5732-c143-453c-9149-5744017ea635"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:50:13.835039 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:13.835006 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/757d5732-c143-453c-9149-5744017ea635-metrics-certs" pod "757d5732-c143-453c-9149-5744017ea635" (UID: "757d5732-c143-453c-9149-5744017ea635"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:50:13.835150 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:13.835133 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/757d5732-c143-453c-9149-5744017ea635-kube-api-access-t789t" pod "757d5732-c143-453c-9149-5744017ea635" (UID: "757d5732-c143-453c-9149-5744017ea635"). InnerVolumeSpecName "kube-api-access-t789t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:50:13.934018 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:13.933981 4982 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/757d5732-c143-453c-9149-5744017ea635-metrics-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:13.934018 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:13.934016 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/757d5732-c143-453c-9149-5744017ea635-maas-api-tls\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:13.934228 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:13.934029 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t789t\" (UniqueName: \"kubernetes.io/projected/757d5732-c143-453c-9149-5744017ea635-kube-api-access-t789t\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:13.934228 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:13.934043 4982 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/757d5732-c143-453c-9149-5744017ea635-openshift-service-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:14.225438 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:14.225408 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:14.398956 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:14.398917 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:14.558125 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:14.558093 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:14.719590 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:14.719561 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2df0ab-58454c8987-cjnqk" Oct 30 14:50:14.719590 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:14.719572 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2df0ab-58454c8987-cjnqk" event={"ID":"757d5732-c143-453c-9149-5744017ea635","Type":"ContainerDied","Data":"61ead91e528a60d02ba4ba82344925d8de92b728bdde5c74c6b7828d1a9d14f6"} Oct 30 14:50:14.719871 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:14.719636 4982 scope.go:122] "RemoveContainer" containerID="650906439f75a7905bac8c85cda95633476909dd2e6361fa52494a36efd02472" Oct 30 14:50:14.778083 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:14.778053 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-2df0ab-58454c8987-cjnqk"] Oct 30 14:50:14.791352 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:14.791327 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-2df0ab-58454c8987-cjnqk"] Oct 30 14:50:14.883025 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:14.882959 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:15.033699 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.033663 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f219c0fa-c9x9b"] Oct 30 14:50:15.039865 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.039839 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f219c0fa-c9x9b"] Oct 30 14:50:15.466284 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.466257 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b" Oct 30 14:50:15.548939 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.548857 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52b260c5-122f-4e2a-8344-56d8efbb66f6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52b260c5-122f-4e2a-8344-56d8efbb66f6-istio-envoy\") pod \"52b260c5-122f-4e2a-8344-56d8efbb66f6\" (UID: \"52b260c5-122f-4e2a-8344-56d8efbb66f6\") " Oct 30 14:50:15.548939 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.548932 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52b260c5-122f-4e2a-8344-56d8efbb66f6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52b260c5-122f-4e2a-8344-56d8efbb66f6-credential-socket\") pod \"52b260c5-122f-4e2a-8344-56d8efbb66f6\" (UID: \"52b260c5-122f-4e2a-8344-56d8efbb66f6\") " Oct 30 14:50:15.549193 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.548998 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52b260c5-122f-4e2a-8344-56d8efbb66f6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52b260c5-122f-4e2a-8344-56d8efbb66f6-workload-certs\") pod \"52b260c5-122f-4e2a-8344-56d8efbb66f6\" (UID: \"52b260c5-122f-4e2a-8344-56d8efbb66f6\") " Oct 30 14:50:15.549193 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.549052 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/52b260c5-122f-4e2a-8344-56d8efbb66f6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52b260c5-122f-4e2a-8344-56d8efbb66f6-istio-podinfo\") pod \"52b260c5-122f-4e2a-8344-56d8efbb66f6\" (UID: \"52b260c5-122f-4e2a-8344-56d8efbb66f6\") " Oct 30 14:50:15.549193 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.549090 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52b260c5-122f-4e2a-8344-56d8efbb66f6-istio-token\" (UniqueName: \"kubernetes.io/projected/52b260c5-122f-4e2a-8344-56d8efbb66f6-istio-token\") pod \"52b260c5-122f-4e2a-8344-56d8efbb66f6\" (UID: \"52b260c5-122f-4e2a-8344-56d8efbb66f6\") " Oct 30 14:50:15.549193 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.549119 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52b260c5-122f-4e2a-8344-56d8efbb66f6-kube-api-access-hq6s6\" (UniqueName: \"kubernetes.io/projected/52b260c5-122f-4e2a-8344-56d8efbb66f6-kube-api-access-hq6s6\") pod \"52b260c5-122f-4e2a-8344-56d8efbb66f6\" (UID: \"52b260c5-122f-4e2a-8344-56d8efbb66f6\") " Oct 30 14:50:15.549193 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.549186 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52b260c5-122f-4e2a-8344-56d8efbb66f6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52b260c5-122f-4e2a-8344-56d8efbb66f6-istiod-ca-cert\") pod \"52b260c5-122f-4e2a-8344-56d8efbb66f6\" (UID: \"52b260c5-122f-4e2a-8344-56d8efbb66f6\") " Oct 30 14:50:15.549489 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.549236 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52b260c5-122f-4e2a-8344-56d8efbb66f6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52b260c5-122f-4e2a-8344-56d8efbb66f6-workload-socket\") pod \"52b260c5-122f-4e2a-8344-56d8efbb66f6\" (UID: \"52b260c5-122f-4e2a-8344-56d8efbb66f6\") " Oct 30 14:50:15.549489 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.549244 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52b260c5-122f-4e2a-8344-56d8efbb66f6-workload-certs" pod "52b260c5-122f-4e2a-8344-56d8efbb66f6" (UID: "52b260c5-122f-4e2a-8344-56d8efbb66f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:15.549489 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.549260 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52b260c5-122f-4e2a-8344-56d8efbb66f6-credential-socket" pod "52b260c5-122f-4e2a-8344-56d8efbb66f6" (UID: "52b260c5-122f-4e2a-8344-56d8efbb66f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:15.549489 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.549271 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52b260c5-122f-4e2a-8344-56d8efbb66f6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52b260c5-122f-4e2a-8344-56d8efbb66f6-istio-data\") pod \"52b260c5-122f-4e2a-8344-56d8efbb66f6\" (UID: \"52b260c5-122f-4e2a-8344-56d8efbb66f6\") " Oct 30 14:50:15.549489 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.549478 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52b260c5-122f-4e2a-8344-56d8efbb66f6-workload-socket" pod "52b260c5-122f-4e2a-8344-56d8efbb66f6" (UID: "52b260c5-122f-4e2a-8344-56d8efbb66f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:15.549804 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.549574 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52b260c5-122f-4e2a-8344-56d8efbb66f6-istiod-ca-cert" pod "52b260c5-122f-4e2a-8344-56d8efbb66f6" (UID: "52b260c5-122f-4e2a-8344-56d8efbb66f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:50:15.549804 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.549679 4982 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52b260c5-122f-4e2a-8344-56d8efbb66f6-workload-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:15.549804 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.549693 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52b260c5-122f-4e2a-8344-56d8efbb66f6-istio-data" pod "52b260c5-122f-4e2a-8344-56d8efbb66f6" (UID: "52b260c5-122f-4e2a-8344-56d8efbb66f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:15.549804 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.549699 4982 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52b260c5-122f-4e2a-8344-56d8efbb66f6-credential-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:15.549804 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.549720 4982 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52b260c5-122f-4e2a-8344-56d8efbb66f6-workload-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:15.551592 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.551566 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52b260c5-122f-4e2a-8344-56d8efbb66f6-istio-token" pod "52b260c5-122f-4e2a-8344-56d8efbb66f6" (UID: "52b260c5-122f-4e2a-8344-56d8efbb66f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:50:15.551748 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.551597 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52b260c5-122f-4e2a-8344-56d8efbb66f6-istio-envoy" pod "52b260c5-122f-4e2a-8344-56d8efbb66f6" (UID: "52b260c5-122f-4e2a-8344-56d8efbb66f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:15.551748 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.551611 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52b260c5-122f-4e2a-8344-56d8efbb66f6-istio-podinfo" pod "52b260c5-122f-4e2a-8344-56d8efbb66f6" (UID: "52b260c5-122f-4e2a-8344-56d8efbb66f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:50:15.551748 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.551700 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52b260c5-122f-4e2a-8344-56d8efbb66f6-kube-api-access-hq6s6" pod "52b260c5-122f-4e2a-8344-56d8efbb66f6" (UID: "52b260c5-122f-4e2a-8344-56d8efbb66f6"). InnerVolumeSpecName "kube-api-access-hq6s6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:50:15.650169 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.650144 4982 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52b260c5-122f-4e2a-8344-56d8efbb66f6-istio-podinfo\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:15.650169 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.650168 4982 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52b260c5-122f-4e2a-8344-56d8efbb66f6-istio-token\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:15.650521 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.650177 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hq6s6\" (UniqueName: \"kubernetes.io/projected/52b260c5-122f-4e2a-8344-56d8efbb66f6-kube-api-access-hq6s6\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:15.650521 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.650186 4982 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52b260c5-122f-4e2a-8344-56d8efbb66f6-istiod-ca-cert\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:15.650521 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.650194 4982 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52b260c5-122f-4e2a-8344-56d8efbb66f6-istio-data\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:15.650521 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.650202 4982 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52b260c5-122f-4e2a-8344-56d8efbb66f6-istio-envoy\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:15.673665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.673625 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-00926c/distinct-2-w3-00926c-kserve-746b64d6b5-z66nw"] Oct 30 14:50:15.681139 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.681102 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-00926c/distinct-w3-00926c-kserve-774df5b58-cv8vt"] Oct 30 14:50:15.699307 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.699279 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb"] Oct 30 14:50:15.732074 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.732035 4982 generic.go:356] "Generic (PLEG): container finished" podID="c99896fe-79b1-4368-aeb5-a81182e7ae13" containerID="8702c319b369afd4d5d80fad6ade9493e1cdb09f96cbf3a40b0cdb727cf5759d" exitCode=0 Oct 30 14:50:15.732074 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.732068 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b" event={"ID":"c99896fe-79b1-4368-aeb5-a81182e7ae13","Type":"ContainerDied","Data":"8702c319b369afd4d5d80fad6ade9493e1cdb09f96cbf3a40b0cdb727cf5759d"} Oct 30 14:50:15.732317 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.732145 4982 scope.go:122] "RemoveContainer" containerID="26ba0205e31735dc2452611301762a84e2575620db52858330c39d279e7b54f8" Oct 30 14:50:15.734690 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.734632 4982 generic.go:356] "Generic (PLEG): container finished" podID="52b260c5-122f-4e2a-8344-56d8efbb66f6" containerID="f581e9dc825db0ec024b40e0f6cf9f589ec9508428476950a821eb17bb42d707" exitCode=0 Oct 30 14:50:15.734824 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.734739 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b" event={"ID":"52b260c5-122f-4e2a-8344-56d8efbb66f6","Type":"ContainerDied","Data":"f581e9dc825db0ec024b40e0f6cf9f589ec9508428476950a821eb17bb42d707"} Oct 30 14:50:15.734824 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.734772 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b" event={"ID":"52b260c5-122f-4e2a-8344-56d8efbb66f6","Type":"ContainerDied","Data":"98bc3ae3d4b9fa4ebcd8a3022f9e9a8e99ec3232c5168e72971ce1d7dcb0fb52"} Oct 30 14:50:15.735236 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.735211 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b" Oct 30 14:50:15.744466 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.744450 4982 scope.go:122] "RemoveContainer" containerID="f581e9dc825db0ec024b40e0f6cf9f589ec9508428476950a821eb17bb42d707" Oct 30 14:50:15.762329 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.762309 4982 scope.go:122] "RemoveContainer" containerID="1eb60eea9f4ff8eebbfc71960921badc5a46788f9c55856ed6dea790469f8e6b" Oct 30 14:50:15.772096 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.771997 4982 scope.go:122] "RemoveContainer" containerID="f581e9dc825db0ec024b40e0f6cf9f589ec9508428476950a821eb17bb42d707" Oct 30 14:50:15.772536 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:50:15.772508 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f581e9dc825db0ec024b40e0f6cf9f589ec9508428476950a821eb17bb42d707\": container with ID starting with f581e9dc825db0ec024b40e0f6cf9f589ec9508428476950a821eb17bb42d707 not found: ID does not exist" containerID="f581e9dc825db0ec024b40e0f6cf9f589ec9508428476950a821eb17bb42d707" Oct 30 14:50:15.772671 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.772548 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f581e9dc825db0ec024b40e0f6cf9f589ec9508428476950a821eb17bb42d707"} err="failed to get container status \"f581e9dc825db0ec024b40e0f6cf9f589ec9508428476950a821eb17bb42d707\": rpc error: code = NotFound desc = could not find container \"f581e9dc825db0ec024b40e0f6cf9f589ec9508428476950a821eb17bb42d707\": container with ID starting with f581e9dc825db0ec024b40e0f6cf9f589ec9508428476950a821eb17bb42d707 not found: ID does not exist" Oct 30 14:50:15.772671 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.772589 4982 scope.go:122] "RemoveContainer" containerID="1eb60eea9f4ff8eebbfc71960921badc5a46788f9c55856ed6dea790469f8e6b" Oct 30 14:50:15.773224 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:50:15.773136 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1eb60eea9f4ff8eebbfc71960921badc5a46788f9c55856ed6dea790469f8e6b\": container with ID starting with 1eb60eea9f4ff8eebbfc71960921badc5a46788f9c55856ed6dea790469f8e6b not found: ID does not exist" containerID="1eb60eea9f4ff8eebbfc71960921badc5a46788f9c55856ed6dea790469f8e6b" Oct 30 14:50:15.773224 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.773178 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1eb60eea9f4ff8eebbfc71960921badc5a46788f9c55856ed6dea790469f8e6b"} err="failed to get container status \"1eb60eea9f4ff8eebbfc71960921badc5a46788f9c55856ed6dea790469f8e6b\": rpc error: code = NotFound desc = could not find container \"1eb60eea9f4ff8eebbfc71960921badc5a46788f9c55856ed6dea790469f8e6b\": container with ID starting with 1eb60eea9f4ff8eebbfc71960921badc5a46788f9c55856ed6dea790469f8e6b not found: ID does not exist" Oct 30 14:50:15.819753 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.819712 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b" Oct 30 14:50:15.824143 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.824106 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:15.852650 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.852615 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c99896fe-79b1-4368-aeb5-a81182e7ae13-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c99896fe-79b1-4368-aeb5-a81182e7ae13-workload-socket\") pod \"c99896fe-79b1-4368-aeb5-a81182e7ae13\" (UID: \"c99896fe-79b1-4368-aeb5-a81182e7ae13\") " Oct 30 14:50:15.852768 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.852670 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c99896fe-79b1-4368-aeb5-a81182e7ae13-kube-api-access-zkbww\" (UniqueName: \"kubernetes.io/projected/c99896fe-79b1-4368-aeb5-a81182e7ae13-kube-api-access-zkbww\") pod \"c99896fe-79b1-4368-aeb5-a81182e7ae13\" (UID: \"c99896fe-79b1-4368-aeb5-a81182e7ae13\") " Oct 30 14:50:15.852768 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.852695 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c99896fe-79b1-4368-aeb5-a81182e7ae13-istio-token\" (UniqueName: \"kubernetes.io/projected/c99896fe-79b1-4368-aeb5-a81182e7ae13-istio-token\") pod \"c99896fe-79b1-4368-aeb5-a81182e7ae13\" (UID: \"c99896fe-79b1-4368-aeb5-a81182e7ae13\") " Oct 30 14:50:15.852768 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.852763 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c99896fe-79b1-4368-aeb5-a81182e7ae13-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c99896fe-79b1-4368-aeb5-a81182e7ae13-istio-data\") pod \"c99896fe-79b1-4368-aeb5-a81182e7ae13\" (UID: \"c99896fe-79b1-4368-aeb5-a81182e7ae13\") " Oct 30 14:50:15.852964 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.852793 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c99896fe-79b1-4368-aeb5-a81182e7ae13-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c99896fe-79b1-4368-aeb5-a81182e7ae13-istio-envoy\") pod \"c99896fe-79b1-4368-aeb5-a81182e7ae13\" (UID: \"c99896fe-79b1-4368-aeb5-a81182e7ae13\") " Oct 30 14:50:15.852964 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.852819 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c99896fe-79b1-4368-aeb5-a81182e7ae13-workload-socket" pod "c99896fe-79b1-4368-aeb5-a81182e7ae13" (UID: "c99896fe-79b1-4368-aeb5-a81182e7ae13"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:15.852964 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.852835 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c99896fe-79b1-4368-aeb5-a81182e7ae13-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c99896fe-79b1-4368-aeb5-a81182e7ae13-istiod-ca-cert\") pod \"c99896fe-79b1-4368-aeb5-a81182e7ae13\" (UID: \"c99896fe-79b1-4368-aeb5-a81182e7ae13\") " Oct 30 14:50:15.852964 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.852874 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c99896fe-79b1-4368-aeb5-a81182e7ae13-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c99896fe-79b1-4368-aeb5-a81182e7ae13-credential-socket\") pod \"c99896fe-79b1-4368-aeb5-a81182e7ae13\" (UID: \"c99896fe-79b1-4368-aeb5-a81182e7ae13\") " Oct 30 14:50:15.852964 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.852899 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c99896fe-79b1-4368-aeb5-a81182e7ae13-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c99896fe-79b1-4368-aeb5-a81182e7ae13-workload-certs\") pod \"c99896fe-79b1-4368-aeb5-a81182e7ae13\" (UID: \"c99896fe-79b1-4368-aeb5-a81182e7ae13\") " Oct 30 14:50:15.852964 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.852929 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c99896fe-79b1-4368-aeb5-a81182e7ae13-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c99896fe-79b1-4368-aeb5-a81182e7ae13-istio-podinfo\") pod \"c99896fe-79b1-4368-aeb5-a81182e7ae13\" (UID: \"c99896fe-79b1-4368-aeb5-a81182e7ae13\") " Oct 30 14:50:15.853295 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.853083 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c99896fe-79b1-4368-aeb5-a81182e7ae13-istio-data" pod "c99896fe-79b1-4368-aeb5-a81182e7ae13" (UID: "c99896fe-79b1-4368-aeb5-a81182e7ae13"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:15.853295 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.853122 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c99896fe-79b1-4368-aeb5-a81182e7ae13-credential-socket" pod "c99896fe-79b1-4368-aeb5-a81182e7ae13" (UID: "c99896fe-79b1-4368-aeb5-a81182e7ae13"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:15.853295 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.853199 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c99896fe-79b1-4368-aeb5-a81182e7ae13-workload-certs" pod "c99896fe-79b1-4368-aeb5-a81182e7ae13" (UID: "c99896fe-79b1-4368-aeb5-a81182e7ae13"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:15.853295 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.853282 4982 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c99896fe-79b1-4368-aeb5-a81182e7ae13-workload-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:15.853511 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.853303 4982 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c99896fe-79b1-4368-aeb5-a81182e7ae13-istio-data\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:15.853511 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.853312 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c99896fe-79b1-4368-aeb5-a81182e7ae13-istiod-ca-cert" pod "c99896fe-79b1-4368-aeb5-a81182e7ae13" (UID: "c99896fe-79b1-4368-aeb5-a81182e7ae13"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:50:15.855186 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.855157 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c99896fe-79b1-4368-aeb5-a81182e7ae13-kube-api-access-zkbww" pod "c99896fe-79b1-4368-aeb5-a81182e7ae13" (UID: "c99896fe-79b1-4368-aeb5-a81182e7ae13"). InnerVolumeSpecName "kube-api-access-zkbww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:50:15.855302 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.855208 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c99896fe-79b1-4368-aeb5-a81182e7ae13-istio-envoy" pod "c99896fe-79b1-4368-aeb5-a81182e7ae13" (UID: "c99896fe-79b1-4368-aeb5-a81182e7ae13"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:15.855302 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.855234 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c99896fe-79b1-4368-aeb5-a81182e7ae13-istio-token" pod "c99896fe-79b1-4368-aeb5-a81182e7ae13" (UID: "c99896fe-79b1-4368-aeb5-a81182e7ae13"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:50:15.855437 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.855372 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c99896fe-79b1-4368-aeb5-a81182e7ae13-istio-podinfo" pod "c99896fe-79b1-4368-aeb5-a81182e7ae13" (UID: "c99896fe-79b1-4368-aeb5-a81182e7ae13"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:50:15.857323 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.857300 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b"] Oct 30 14:50:15.866894 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.866872 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-228454-openshift-default-6476dbf645-pwj4b"] Oct 30 14:50:15.954665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.954625 4982 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c99896fe-79b1-4368-aeb5-a81182e7ae13-istio-envoy\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:15.954665 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.954664 4982 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c99896fe-79b1-4368-aeb5-a81182e7ae13-istiod-ca-cert\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:15.954868 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.954673 4982 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c99896fe-79b1-4368-aeb5-a81182e7ae13-credential-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:15.954868 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.954682 4982 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c99896fe-79b1-4368-aeb5-a81182e7ae13-workload-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:15.954868 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.954692 4982 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c99896fe-79b1-4368-aeb5-a81182e7ae13-istio-podinfo\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:15.954868 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.954700 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkbww\" (UniqueName: \"kubernetes.io/projected/c99896fe-79b1-4368-aeb5-a81182e7ae13-kube-api-access-zkbww\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:15.954868 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:15.954709 4982 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c99896fe-79b1-4368-aeb5-a81182e7ae13-istio-token\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:16.012889 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:16.012851 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7"] Oct 30 14:50:16.021289 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:16.021260 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7"] Oct 30 14:50:16.134373 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:16.134292 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:16.400145 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:16.400070 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52b260c5-122f-4e2a-8344-56d8efbb66f6" path="/var/lib/kubelet/pods/52b260c5-122f-4e2a-8344-56d8efbb66f6/volumes" Oct 30 14:50:16.400636 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:16.400618 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="757d5732-c143-453c-9149-5744017ea635" path="/var/lib/kubelet/pods/757d5732-c143-453c-9149-5744017ea635/volumes" Oct 30 14:50:16.401161 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:16.401145 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ef323f9-5aa4-48c4-826d-4d1b9ab83126" path="/var/lib/kubelet/pods/8ef323f9-5aa4-48c4-826d-4d1b9ab83126/volumes" Oct 30 14:50:16.662272 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:16.662202 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:16.742860 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:16.742820 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b" event={"ID":"c99896fe-79b1-4368-aeb5-a81182e7ae13","Type":"ContainerDied","Data":"b55df68921fc279754136cb60172008d1f4ad1a6a137f6d3f8eb89954650a007"} Oct 30 14:50:16.742860 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:16.742844 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b" Oct 30 14:50:16.743130 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:16.742882 4982 scope.go:122] "RemoveContainer" containerID="8702c319b369afd4d5d80fad6ade9493e1cdb09f96cbf3a40b0cdb727cf5759d" Oct 30 14:50:16.815905 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:16.815880 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b"] Oct 30 14:50:16.827790 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:16.827768 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-4lg9b"] Oct 30 14:50:18.398160 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:18.398124 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c99896fe-79b1-4368-aeb5-a81182e7ae13" path="/var/lib/kubelet/pods/c99896fe-79b1-4368-aeb5-a81182e7ae13/volumes" Oct 30 14:50:18.709210 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:18.709140 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:20.501286 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:20.501249 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:21.842278 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:21.842245 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:21.985992 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:21.985955 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:22.129737 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:22.129618 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:22.477089 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:22.477001 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:22.633944 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:22.633907 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:22.801317 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:22.801214 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:22.953160 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:22.953126 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:23.567915 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:23.567879 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:23.906488 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:23.906406 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:24.156684 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:24.156598 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:24.673594 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:24.673550 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:24.998821 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:24.998739 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:26.262229 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:26.262197 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:28.011274 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:50:28.011234 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:50:29.986309 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:29.986269 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:30.141927 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:30.141896 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:30.389896 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:30.389862 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:30.699230 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:30.699142 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-00926c/distinct-2-w3-00926c-kserve-746b64d6b5-z66nw" podUID="91f37a37-70fe-4627-b2dc-77a9668d9a5f" containerName="main" containerID="cri-o://a26dd6fe249791f6e7b2984c8ac9c5790db86bd4a9af61780d45f8e9edb452ed" gracePeriod=45 Oct 30 14:50:30.712113 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:30.712083 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-00926c/distinct-w3-00926c-kserve-774df5b58-cv8vt" podUID="b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec" containerName="main" containerID="cri-o://3c238d8bd8a85494179991ea3c8193ed2423c27b8d0f7690b704e77f3a692a97" gracePeriod=45 Oct 30 14:50:30.719915 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:30.719888 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" podUID="d65dee9b-4f8b-4f09-b00b-0793d732d3ab" containerName="main" containerID="cri-o://2eb837ccd1af3f8712b30d47022687fe2c17e8013ee26e231bf5eb16986b6432" gracePeriod=45 Oct 30 14:50:30.823634 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:30.823606 4982 generic.go:356] "Generic (PLEG): container finished" podID="91f37a37-70fe-4627-b2dc-77a9668d9a5f" containerID="a26dd6fe249791f6e7b2984c8ac9c5790db86bd4a9af61780d45f8e9edb452ed" exitCode=0 Oct 30 14:50:30.823634 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:30.823672 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-00926c/distinct-2-w3-00926c-kserve-746b64d6b5-z66nw" event={"ID":"91f37a37-70fe-4627-b2dc-77a9668d9a5f","Type":"ContainerDied","Data":"a26dd6fe249791f6e7b2984c8ac9c5790db86bd4a9af61780d45f8e9edb452ed"} Oct 30 14:50:30.989771 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:30.989743 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-00926c/distinct-w3-00926c-kserve-774df5b58-cv8vt" Oct 30 14:50:30.993156 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:30.993132 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-00926c/distinct-2-w3-00926c-kserve-746b64d6b5-z66nw" Oct 30 14:50:31.004671 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.004626 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" Oct 30 14:50:31.033848 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.033819 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" podUID="4fa94d86-57ce-4d23-95d5-4efa965f6864" containerName="main" containerID="cri-o://b24f03de41febe4ea1dd73fe96cebf785a05a1decf34a4f7644408cfa18b506b" gracePeriod=45 Oct 30 14:50:31.046859 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.046821 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" podUID="ab822fe6-4349-4051-b2fd-c6e5ceb50d08" containerName="main" containerID="cri-o://5176658323e9e61440288595d8b112cebf5264072222b1a68a58ade511c01c6d" gracePeriod=45 Oct 30 14:50:31.118317 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.118282 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91f37a37-70fe-4627-b2dc-77a9668d9a5f-kube-api-access-gvkrj\" (UniqueName: \"kubernetes.io/projected/91f37a37-70fe-4627-b2dc-77a9668d9a5f-kube-api-access-gvkrj\") pod \"91f37a37-70fe-4627-b2dc-77a9668d9a5f\" (UID: \"91f37a37-70fe-4627-b2dc-77a9668d9a5f\") " Oct 30 14:50:31.118474 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.118347 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec-model-cache\") pod \"b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec\" (UID: \"b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec\") " Oct 30 14:50:31.118474 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.118382 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d65dee9b-4f8b-4f09-b00b-0793d732d3ab-tls-certs\" (UniqueName: \"kubernetes.io/secret/d65dee9b-4f8b-4f09-b00b-0793d732d3ab-tls-certs\") pod \"d65dee9b-4f8b-4f09-b00b-0793d732d3ab\" (UID: \"d65dee9b-4f8b-4f09-b00b-0793d732d3ab\") " Oct 30 14:50:31.118474 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.118411 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec-kube-api-access-mlc8w\" (UniqueName: \"kubernetes.io/projected/b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec-kube-api-access-mlc8w\") pod \"b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec\" (UID: \"b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec\") " Oct 30 14:50:31.118474 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.118455 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec-dshm\") pod \"b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec\" (UID: \"b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec\") " Oct 30 14:50:31.118741 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.118492 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d65dee9b-4f8b-4f09-b00b-0793d732d3ab-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d65dee9b-4f8b-4f09-b00b-0793d732d3ab-dshm\") pod \"d65dee9b-4f8b-4f09-b00b-0793d732d3ab\" (UID: \"d65dee9b-4f8b-4f09-b00b-0793d732d3ab\") " Oct 30 14:50:31.118741 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.118564 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d65dee9b-4f8b-4f09-b00b-0793d732d3ab-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d65dee9b-4f8b-4f09-b00b-0793d732d3ab-model-cache\") pod \"d65dee9b-4f8b-4f09-b00b-0793d732d3ab\" (UID: \"d65dee9b-4f8b-4f09-b00b-0793d732d3ab\") " Oct 30 14:50:31.118741 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.118593 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91f37a37-70fe-4627-b2dc-77a9668d9a5f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/91f37a37-70fe-4627-b2dc-77a9668d9a5f-dshm\") pod \"91f37a37-70fe-4627-b2dc-77a9668d9a5f\" (UID: \"91f37a37-70fe-4627-b2dc-77a9668d9a5f\") " Oct 30 14:50:31.118741 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.118624 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91f37a37-70fe-4627-b2dc-77a9668d9a5f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91f37a37-70fe-4627-b2dc-77a9668d9a5f-model-cache\") pod \"91f37a37-70fe-4627-b2dc-77a9668d9a5f\" (UID: \"91f37a37-70fe-4627-b2dc-77a9668d9a5f\") " Oct 30 14:50:31.118741 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.118690 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91f37a37-70fe-4627-b2dc-77a9668d9a5f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91f37a37-70fe-4627-b2dc-77a9668d9a5f-tmp-dir\") pod \"91f37a37-70fe-4627-b2dc-77a9668d9a5f\" (UID: \"91f37a37-70fe-4627-b2dc-77a9668d9a5f\") " Oct 30 14:50:31.118741 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.118736 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec-tls-certs\" (UniqueName: \"kubernetes.io/secret/b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec-tls-certs\") pod \"b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec\" (UID: \"b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec\") " Oct 30 14:50:31.119139 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.118771 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d65dee9b-4f8b-4f09-b00b-0793d732d3ab-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d65dee9b-4f8b-4f09-b00b-0793d732d3ab-tmp-dir\") pod \"d65dee9b-4f8b-4f09-b00b-0793d732d3ab\" (UID: \"d65dee9b-4f8b-4f09-b00b-0793d732d3ab\") " Oct 30 14:50:31.119139 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.118808 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d65dee9b-4f8b-4f09-b00b-0793d732d3ab-home\" (UniqueName: \"kubernetes.io/empty-dir/d65dee9b-4f8b-4f09-b00b-0793d732d3ab-home\") pod \"d65dee9b-4f8b-4f09-b00b-0793d732d3ab\" (UID: \"d65dee9b-4f8b-4f09-b00b-0793d732d3ab\") " Oct 30 14:50:31.119139 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.118863 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d65dee9b-4f8b-4f09-b00b-0793d732d3ab-kube-api-access-jth7w\" (UniqueName: \"kubernetes.io/projected/d65dee9b-4f8b-4f09-b00b-0793d732d3ab-kube-api-access-jth7w\") pod \"d65dee9b-4f8b-4f09-b00b-0793d732d3ab\" (UID: \"d65dee9b-4f8b-4f09-b00b-0793d732d3ab\") " Oct 30 14:50:31.119139 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.118902 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91f37a37-70fe-4627-b2dc-77a9668d9a5f-tls-certs\" (UniqueName: \"kubernetes.io/secret/91f37a37-70fe-4627-b2dc-77a9668d9a5f-tls-certs\") pod \"91f37a37-70fe-4627-b2dc-77a9668d9a5f\" (UID: \"91f37a37-70fe-4627-b2dc-77a9668d9a5f\") " Oct 30 14:50:31.119139 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.118934 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec-tmp-dir\") pod \"b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec\" (UID: \"b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec\") " Oct 30 14:50:31.119139 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.118959 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91f37a37-70fe-4627-b2dc-77a9668d9a5f-home\" (UniqueName: \"kubernetes.io/empty-dir/91f37a37-70fe-4627-b2dc-77a9668d9a5f-home\") pod \"91f37a37-70fe-4627-b2dc-77a9668d9a5f\" (UID: \"91f37a37-70fe-4627-b2dc-77a9668d9a5f\") " Oct 30 14:50:31.119139 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.118990 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec-home\" (UniqueName: \"kubernetes.io/empty-dir/b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec-home\") pod \"b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec\" (UID: \"b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec\") " Oct 30 14:50:31.119502 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.119383 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec-model-cache" pod "b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec" (UID: "b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:31.119502 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.119377 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d65dee9b-4f8b-4f09-b00b-0793d732d3ab-model-cache" pod "d65dee9b-4f8b-4f09-b00b-0793d732d3ab" (UID: "d65dee9b-4f8b-4f09-b00b-0793d732d3ab"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:31.119630 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.119606 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec-home" pod "b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec" (UID: "b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:31.119770 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.119725 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec-tmp-dir" pod "b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec" (UID: "b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:31.119852 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.119829 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91f37a37-70fe-4627-b2dc-77a9668d9a5f-home" pod "91f37a37-70fe-4627-b2dc-77a9668d9a5f" (UID: "91f37a37-70fe-4627-b2dc-77a9668d9a5f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:31.120789 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.120575 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d65dee9b-4f8b-4f09-b00b-0793d732d3ab-home" pod "d65dee9b-4f8b-4f09-b00b-0793d732d3ab" (UID: "d65dee9b-4f8b-4f09-b00b-0793d732d3ab"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:31.120789 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.120748 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91f37a37-70fe-4627-b2dc-77a9668d9a5f-tmp-dir" pod "91f37a37-70fe-4627-b2dc-77a9668d9a5f" (UID: "91f37a37-70fe-4627-b2dc-77a9668d9a5f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:31.120789 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.120760 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91f37a37-70fe-4627-b2dc-77a9668d9a5f-model-cache" pod "91f37a37-70fe-4627-b2dc-77a9668d9a5f" (UID: "91f37a37-70fe-4627-b2dc-77a9668d9a5f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:31.121066 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.120876 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d65dee9b-4f8b-4f09-b00b-0793d732d3ab-tmp-dir" pod "d65dee9b-4f8b-4f09-b00b-0793d732d3ab" (UID: "d65dee9b-4f8b-4f09-b00b-0793d732d3ab"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:31.122319 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.122276 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec-dshm" pod "b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec" (UID: "b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:31.122319 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.122281 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec-kube-api-access-mlc8w" pod "b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec" (UID: "b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec"). InnerVolumeSpecName "kube-api-access-mlc8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:50:31.123220 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.123194 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec-tls-certs" pod "b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec" (UID: "b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:50:31.123886 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.123793 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d65dee9b-4f8b-4f09-b00b-0793d732d3ab-tls-certs" pod "d65dee9b-4f8b-4f09-b00b-0793d732d3ab" (UID: "d65dee9b-4f8b-4f09-b00b-0793d732d3ab"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:50:31.123886 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.123850 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91f37a37-70fe-4627-b2dc-77a9668d9a5f-tls-certs" pod "91f37a37-70fe-4627-b2dc-77a9668d9a5f" (UID: "91f37a37-70fe-4627-b2dc-77a9668d9a5f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:50:31.124062 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.123920 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91f37a37-70fe-4627-b2dc-77a9668d9a5f-dshm" pod "91f37a37-70fe-4627-b2dc-77a9668d9a5f" (UID: "91f37a37-70fe-4627-b2dc-77a9668d9a5f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:31.124062 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.123898 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91f37a37-70fe-4627-b2dc-77a9668d9a5f-kube-api-access-gvkrj" pod "91f37a37-70fe-4627-b2dc-77a9668d9a5f" (UID: "91f37a37-70fe-4627-b2dc-77a9668d9a5f"). InnerVolumeSpecName "kube-api-access-gvkrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:50:31.125418 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.125396 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d65dee9b-4f8b-4f09-b00b-0793d732d3ab-kube-api-access-jth7w" pod "d65dee9b-4f8b-4f09-b00b-0793d732d3ab" (UID: "d65dee9b-4f8b-4f09-b00b-0793d732d3ab"). InnerVolumeSpecName "kube-api-access-jth7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:50:31.125494 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.125479 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d65dee9b-4f8b-4f09-b00b-0793d732d3ab-dshm" pod "d65dee9b-4f8b-4f09-b00b-0793d732d3ab" (UID: "d65dee9b-4f8b-4f09-b00b-0793d732d3ab"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:31.220333 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.220292 4982 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d65dee9b-4f8b-4f09-b00b-0793d732d3ab-tmp-dir\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:31.220558 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.220336 4982 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d65dee9b-4f8b-4f09-b00b-0793d732d3ab-home\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:31.220558 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.220350 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jth7w\" (UniqueName: \"kubernetes.io/projected/d65dee9b-4f8b-4f09-b00b-0793d732d3ab-kube-api-access-jth7w\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:31.220558 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.220361 4982 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91f37a37-70fe-4627-b2dc-77a9668d9a5f-tls-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:31.220558 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.220374 4982 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec-tmp-dir\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:31.220558 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.220385 4982 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91f37a37-70fe-4627-b2dc-77a9668d9a5f-home\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:31.220558 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.220396 4982 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec-home\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:31.220558 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.220408 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvkrj\" (UniqueName: \"kubernetes.io/projected/91f37a37-70fe-4627-b2dc-77a9668d9a5f-kube-api-access-gvkrj\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:31.220558 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.220420 4982 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec-model-cache\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:31.220558 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.220436 4982 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d65dee9b-4f8b-4f09-b00b-0793d732d3ab-tls-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:31.220558 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.220448 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlc8w\" (UniqueName: \"kubernetes.io/projected/b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec-kube-api-access-mlc8w\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:31.220558 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.220459 4982 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec-dshm\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:31.220558 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.220473 4982 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d65dee9b-4f8b-4f09-b00b-0793d732d3ab-dshm\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:31.220558 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.220483 4982 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d65dee9b-4f8b-4f09-b00b-0793d732d3ab-model-cache\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:31.220558 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.220494 4982 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91f37a37-70fe-4627-b2dc-77a9668d9a5f-dshm\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:31.220558 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.220504 4982 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91f37a37-70fe-4627-b2dc-77a9668d9a5f-model-cache\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:31.220558 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.220514 4982 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91f37a37-70fe-4627-b2dc-77a9668d9a5f-tmp-dir\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:31.220558 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.220525 4982 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec-tls-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:31.294539 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.294509 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" Oct 30 14:50:31.316263 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.316239 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" Oct 30 14:50:31.422932 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.422897 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fa94d86-57ce-4d23-95d5-4efa965f6864-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fa94d86-57ce-4d23-95d5-4efa965f6864-dshm\") pod \"4fa94d86-57ce-4d23-95d5-4efa965f6864\" (UID: \"4fa94d86-57ce-4d23-95d5-4efa965f6864\") " Oct 30 14:50:31.423104 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.422967 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab822fe6-4349-4051-b2fd-c6e5ceb50d08-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab822fe6-4349-4051-b2fd-c6e5ceb50d08-model-cache\") pod \"ab822fe6-4349-4051-b2fd-c6e5ceb50d08\" (UID: \"ab822fe6-4349-4051-b2fd-c6e5ceb50d08\") " Oct 30 14:50:31.423104 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.423011 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fa94d86-57ce-4d23-95d5-4efa965f6864-tls-certs\" (UniqueName: \"kubernetes.io/secret/4fa94d86-57ce-4d23-95d5-4efa965f6864-tls-certs\") pod \"4fa94d86-57ce-4d23-95d5-4efa965f6864\" (UID: \"4fa94d86-57ce-4d23-95d5-4efa965f6864\") " Oct 30 14:50:31.423104 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.423040 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fa94d86-57ce-4d23-95d5-4efa965f6864-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fa94d86-57ce-4d23-95d5-4efa965f6864-model-cache\") pod \"4fa94d86-57ce-4d23-95d5-4efa965f6864\" (UID: \"4fa94d86-57ce-4d23-95d5-4efa965f6864\") " Oct 30 14:50:31.423104 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.423090 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fa94d86-57ce-4d23-95d5-4efa965f6864-kube-api-access-2nm87\" (UniqueName: \"kubernetes.io/projected/4fa94d86-57ce-4d23-95d5-4efa965f6864-kube-api-access-2nm87\") pod \"4fa94d86-57ce-4d23-95d5-4efa965f6864\" (UID: \"4fa94d86-57ce-4d23-95d5-4efa965f6864\") " Oct 30 14:50:31.423356 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.423124 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab822fe6-4349-4051-b2fd-c6e5ceb50d08-home\" (UniqueName: \"kubernetes.io/empty-dir/ab822fe6-4349-4051-b2fd-c6e5ceb50d08-home\") pod \"ab822fe6-4349-4051-b2fd-c6e5ceb50d08\" (UID: \"ab822fe6-4349-4051-b2fd-c6e5ceb50d08\") " Oct 30 14:50:31.423356 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.423251 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab822fe6-4349-4051-b2fd-c6e5ceb50d08-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab822fe6-4349-4051-b2fd-c6e5ceb50d08-dshm\") pod \"ab822fe6-4349-4051-b2fd-c6e5ceb50d08\" (UID: \"ab822fe6-4349-4051-b2fd-c6e5ceb50d08\") " Oct 30 14:50:31.423356 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.423328 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab822fe6-4349-4051-b2fd-c6e5ceb50d08-kube-api-access-w4sf7\" (UniqueName: \"kubernetes.io/projected/ab822fe6-4349-4051-b2fd-c6e5ceb50d08-kube-api-access-w4sf7\") pod \"ab822fe6-4349-4051-b2fd-c6e5ceb50d08\" (UID: \"ab822fe6-4349-4051-b2fd-c6e5ceb50d08\") " Oct 30 14:50:31.423540 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.423448 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fa94d86-57ce-4d23-95d5-4efa965f6864-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fa94d86-57ce-4d23-95d5-4efa965f6864-tmp-dir\") pod \"4fa94d86-57ce-4d23-95d5-4efa965f6864\" (UID: \"4fa94d86-57ce-4d23-95d5-4efa965f6864\") " Oct 30 14:50:31.423540 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.423479 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab822fe6-4349-4051-b2fd-c6e5ceb50d08-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab822fe6-4349-4051-b2fd-c6e5ceb50d08-tmp-dir\") pod \"ab822fe6-4349-4051-b2fd-c6e5ceb50d08\" (UID: \"ab822fe6-4349-4051-b2fd-c6e5ceb50d08\") " Oct 30 14:50:31.423540 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.423495 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fa94d86-57ce-4d23-95d5-4efa965f6864-model-cache" pod "4fa94d86-57ce-4d23-95d5-4efa965f6864" (UID: "4fa94d86-57ce-4d23-95d5-4efa965f6864"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:31.423729 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.423556 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fa94d86-57ce-4d23-95d5-4efa965f6864-home\" (UniqueName: \"kubernetes.io/empty-dir/4fa94d86-57ce-4d23-95d5-4efa965f6864-home\") pod \"4fa94d86-57ce-4d23-95d5-4efa965f6864\" (UID: \"4fa94d86-57ce-4d23-95d5-4efa965f6864\") " Oct 30 14:50:31.423729 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.423593 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab822fe6-4349-4051-b2fd-c6e5ceb50d08-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab822fe6-4349-4051-b2fd-c6e5ceb50d08-tls-certs\") pod \"ab822fe6-4349-4051-b2fd-c6e5ceb50d08\" (UID: \"ab822fe6-4349-4051-b2fd-c6e5ceb50d08\") " Oct 30 14:50:31.423849 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.423769 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fa94d86-57ce-4d23-95d5-4efa965f6864-tmp-dir" pod "4fa94d86-57ce-4d23-95d5-4efa965f6864" (UID: "4fa94d86-57ce-4d23-95d5-4efa965f6864"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:31.424503 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.424000 4982 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fa94d86-57ce-4d23-95d5-4efa965f6864-tmp-dir\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:31.424503 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.424028 4982 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fa94d86-57ce-4d23-95d5-4efa965f6864-model-cache\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:31.424503 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.424090 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab822fe6-4349-4051-b2fd-c6e5ceb50d08-model-cache" pod "ab822fe6-4349-4051-b2fd-c6e5ceb50d08" (UID: "ab822fe6-4349-4051-b2fd-c6e5ceb50d08"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:31.424503 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.424255 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab822fe6-4349-4051-b2fd-c6e5ceb50d08-home" pod "ab822fe6-4349-4051-b2fd-c6e5ceb50d08" (UID: "ab822fe6-4349-4051-b2fd-c6e5ceb50d08"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:31.424503 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.424334 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab822fe6-4349-4051-b2fd-c6e5ceb50d08-tmp-dir" pod "ab822fe6-4349-4051-b2fd-c6e5ceb50d08" (UID: "ab822fe6-4349-4051-b2fd-c6e5ceb50d08"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:31.424503 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.424472 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fa94d86-57ce-4d23-95d5-4efa965f6864-home" pod "4fa94d86-57ce-4d23-95d5-4efa965f6864" (UID: "4fa94d86-57ce-4d23-95d5-4efa965f6864"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:31.425755 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.425724 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fa94d86-57ce-4d23-95d5-4efa965f6864-dshm" pod "4fa94d86-57ce-4d23-95d5-4efa965f6864" (UID: "4fa94d86-57ce-4d23-95d5-4efa965f6864"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:31.425941 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.425914 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab822fe6-4349-4051-b2fd-c6e5ceb50d08-dshm" pod "ab822fe6-4349-4051-b2fd-c6e5ceb50d08" (UID: "ab822fe6-4349-4051-b2fd-c6e5ceb50d08"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:31.426058 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.426037 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fa94d86-57ce-4d23-95d5-4efa965f6864-kube-api-access-2nm87" pod "4fa94d86-57ce-4d23-95d5-4efa965f6864" (UID: "4fa94d86-57ce-4d23-95d5-4efa965f6864"). InnerVolumeSpecName "kube-api-access-2nm87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:50:31.426128 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.426040 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fa94d86-57ce-4d23-95d5-4efa965f6864-tls-certs" pod "4fa94d86-57ce-4d23-95d5-4efa965f6864" (UID: "4fa94d86-57ce-4d23-95d5-4efa965f6864"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:50:31.426231 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.426210 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab822fe6-4349-4051-b2fd-c6e5ceb50d08-kube-api-access-w4sf7" pod "ab822fe6-4349-4051-b2fd-c6e5ceb50d08" (UID: "ab822fe6-4349-4051-b2fd-c6e5ceb50d08"). InnerVolumeSpecName "kube-api-access-w4sf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:50:31.426381 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.426366 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab822fe6-4349-4051-b2fd-c6e5ceb50d08-tls-certs" pod "ab822fe6-4349-4051-b2fd-c6e5ceb50d08" (UID: "ab822fe6-4349-4051-b2fd-c6e5ceb50d08"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:50:31.525048 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.524956 4982 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab822fe6-4349-4051-b2fd-c6e5ceb50d08-tmp-dir\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:31.525048 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.524990 4982 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fa94d86-57ce-4d23-95d5-4efa965f6864-home\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:31.525048 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.525003 4982 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab822fe6-4349-4051-b2fd-c6e5ceb50d08-tls-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:31.525048 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.525016 4982 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fa94d86-57ce-4d23-95d5-4efa965f6864-dshm\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:31.525048 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.525029 4982 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab822fe6-4349-4051-b2fd-c6e5ceb50d08-model-cache\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:31.525048 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.525044 4982 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fa94d86-57ce-4d23-95d5-4efa965f6864-tls-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:31.525418 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.525057 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nm87\" (UniqueName: \"kubernetes.io/projected/4fa94d86-57ce-4d23-95d5-4efa965f6864-kube-api-access-2nm87\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:31.525418 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.525069 4982 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab822fe6-4349-4051-b2fd-c6e5ceb50d08-home\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:31.525418 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.525082 4982 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab822fe6-4349-4051-b2fd-c6e5ceb50d08-dshm\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:31.525418 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.525094 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4sf7\" (UniqueName: \"kubernetes.io/projected/ab822fe6-4349-4051-b2fd-c6e5ceb50d08-kube-api-access-w4sf7\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:31.832136 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.832109 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-00926c/distinct-2-w3-00926c-kserve-746b64d6b5-z66nw" Oct 30 14:50:31.832315 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.832148 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-00926c/distinct-2-w3-00926c-kserve-746b64d6b5-z66nw" event={"ID":"91f37a37-70fe-4627-b2dc-77a9668d9a5f","Type":"ContainerDied","Data":"757b53c8e277384284fdd144d0dc6273f72ed86b0848f8d329da9af58ad860b7"} Oct 30 14:50:31.832315 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.832205 4982 scope.go:122] "RemoveContainer" containerID="a26dd6fe249791f6e7b2984c8ac9c5790db86bd4a9af61780d45f8e9edb452ed" Oct 30 14:50:31.843491 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.843454 4982 generic.go:356] "Generic (PLEG): container finished" podID="4fa94d86-57ce-4d23-95d5-4efa965f6864" containerID="b24f03de41febe4ea1dd73fe96cebf785a05a1decf34a4f7644408cfa18b506b" exitCode=0 Oct 30 14:50:31.843612 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.843545 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" Oct 30 14:50:31.843612 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.843558 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" event={"ID":"4fa94d86-57ce-4d23-95d5-4efa965f6864","Type":"ContainerDied","Data":"b24f03de41febe4ea1dd73fe96cebf785a05a1decf34a4f7644408cfa18b506b"} Oct 30 14:50:31.843768 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.843627 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7" event={"ID":"4fa94d86-57ce-4d23-95d5-4efa965f6864","Type":"ContainerDied","Data":"9529b2dbd152577c43fa8abd5f97ab25dfd5c838563979d7312fb109ceedcf29"} Oct 30 14:50:31.847069 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.846508 4982 generic.go:356] "Generic (PLEG): container finished" podID="ab822fe6-4349-4051-b2fd-c6e5ceb50d08" containerID="5176658323e9e61440288595d8b112cebf5264072222b1a68a58ade511c01c6d" exitCode=0 Oct 30 14:50:31.847069 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.846582 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" Oct 30 14:50:31.847069 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.846583 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" event={"ID":"ab822fe6-4349-4051-b2fd-c6e5ceb50d08","Type":"ContainerDied","Data":"5176658323e9e61440288595d8b112cebf5264072222b1a68a58ade511c01c6d"} Oct 30 14:50:31.847069 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.846899 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7" event={"ID":"ab822fe6-4349-4051-b2fd-c6e5ceb50d08","Type":"ContainerDied","Data":"7bef8037bddf6276f8cb8759524bba6744408690479b27c8d76a23e1588f51f6"} Oct 30 14:50:31.853859 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.853830 4982 generic.go:356] "Generic (PLEG): container finished" podID="b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec" containerID="3c238d8bd8a85494179991ea3c8193ed2423c27b8d0f7690b704e77f3a692a97" exitCode=0 Oct 30 14:50:31.854140 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.853857 4982 scope.go:122] "RemoveContainer" containerID="b24f03de41febe4ea1dd73fe96cebf785a05a1decf34a4f7644408cfa18b506b" Oct 30 14:50:31.854140 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.853914 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-00926c/distinct-w3-00926c-kserve-774df5b58-cv8vt" event={"ID":"b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec","Type":"ContainerDied","Data":"3c238d8bd8a85494179991ea3c8193ed2423c27b8d0f7690b704e77f3a692a97"} Oct 30 14:50:31.854140 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.853935 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-00926c/distinct-w3-00926c-kserve-774df5b58-cv8vt" Oct 30 14:50:31.854140 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.853940 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-00926c/distinct-w3-00926c-kserve-774df5b58-cv8vt" event={"ID":"b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec","Type":"ContainerDied","Data":"9733c7995f6c37d9e6ab96af4488a2d03212585f36e5e431bc422634ef2c7c95"} Oct 30 14:50:31.856428 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.856404 4982 generic.go:356] "Generic (PLEG): container finished" podID="d65dee9b-4f8b-4f09-b00b-0793d732d3ab" containerID="2eb837ccd1af3f8712b30d47022687fe2c17e8013ee26e231bf5eb16986b6432" exitCode=0 Oct 30 14:50:31.856538 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.856451 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" event={"ID":"d65dee9b-4f8b-4f09-b00b-0793d732d3ab","Type":"ContainerDied","Data":"2eb837ccd1af3f8712b30d47022687fe2c17e8013ee26e231bf5eb16986b6432"} Oct 30 14:50:31.856538 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.856474 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" event={"ID":"d65dee9b-4f8b-4f09-b00b-0793d732d3ab","Type":"ContainerDied","Data":"21a1e0312170883e575298bbd71b8e4a591113545ff9db4b39907d97fd62b61f"} Oct 30 14:50:31.856538 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.856535 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb" Oct 30 14:50:31.867825 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.867355 4982 scope.go:122] "RemoveContainer" containerID="b24f03de41febe4ea1dd73fe96cebf785a05a1decf34a4f7644408cfa18b506b" Oct 30 14:50:31.867825 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:50:31.867674 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b24f03de41febe4ea1dd73fe96cebf785a05a1decf34a4f7644408cfa18b506b\": container with ID starting with b24f03de41febe4ea1dd73fe96cebf785a05a1decf34a4f7644408cfa18b506b not found: ID does not exist" containerID="b24f03de41febe4ea1dd73fe96cebf785a05a1decf34a4f7644408cfa18b506b" Oct 30 14:50:31.867825 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.867703 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b24f03de41febe4ea1dd73fe96cebf785a05a1decf34a4f7644408cfa18b506b"} err="failed to get container status \"b24f03de41febe4ea1dd73fe96cebf785a05a1decf34a4f7644408cfa18b506b\": rpc error: code = NotFound desc = could not find container \"b24f03de41febe4ea1dd73fe96cebf785a05a1decf34a4f7644408cfa18b506b\": container with ID starting with b24f03de41febe4ea1dd73fe96cebf785a05a1decf34a4f7644408cfa18b506b not found: ID does not exist" Oct 30 14:50:31.867825 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.867725 4982 scope.go:122] "RemoveContainer" containerID="5176658323e9e61440288595d8b112cebf5264072222b1a68a58ade511c01c6d" Oct 30 14:50:31.876832 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.876816 4982 scope.go:122] "RemoveContainer" containerID="5176658323e9e61440288595d8b112cebf5264072222b1a68a58ade511c01c6d" Oct 30 14:50:31.877084 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:50:31.877063 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5176658323e9e61440288595d8b112cebf5264072222b1a68a58ade511c01c6d\": container with ID starting with 5176658323e9e61440288595d8b112cebf5264072222b1a68a58ade511c01c6d not found: ID does not exist" containerID="5176658323e9e61440288595d8b112cebf5264072222b1a68a58ade511c01c6d" Oct 30 14:50:31.877167 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.877088 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5176658323e9e61440288595d8b112cebf5264072222b1a68a58ade511c01c6d"} err="failed to get container status \"5176658323e9e61440288595d8b112cebf5264072222b1a68a58ade511c01c6d\": rpc error: code = NotFound desc = could not find container \"5176658323e9e61440288595d8b112cebf5264072222b1a68a58ade511c01c6d\": container with ID starting with 5176658323e9e61440288595d8b112cebf5264072222b1a68a58ade511c01c6d not found: ID does not exist" Oct 30 14:50:31.877167 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.877102 4982 scope.go:122] "RemoveContainer" containerID="3c238d8bd8a85494179991ea3c8193ed2423c27b8d0f7690b704e77f3a692a97" Oct 30 14:50:31.885018 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.885002 4982 scope.go:122] "RemoveContainer" containerID="3c238d8bd8a85494179991ea3c8193ed2423c27b8d0f7690b704e77f3a692a97" Oct 30 14:50:31.885271 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:50:31.885253 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c238d8bd8a85494179991ea3c8193ed2423c27b8d0f7690b704e77f3a692a97\": container with ID starting with 3c238d8bd8a85494179991ea3c8193ed2423c27b8d0f7690b704e77f3a692a97 not found: ID does not exist" containerID="3c238d8bd8a85494179991ea3c8193ed2423c27b8d0f7690b704e77f3a692a97" Oct 30 14:50:31.885312 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.885278 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c238d8bd8a85494179991ea3c8193ed2423c27b8d0f7690b704e77f3a692a97"} err="failed to get container status \"3c238d8bd8a85494179991ea3c8193ed2423c27b8d0f7690b704e77f3a692a97\": rpc error: code = NotFound desc = could not find container \"3c238d8bd8a85494179991ea3c8193ed2423c27b8d0f7690b704e77f3a692a97\": container with ID starting with 3c238d8bd8a85494179991ea3c8193ed2423c27b8d0f7690b704e77f3a692a97 not found: ID does not exist" Oct 30 14:50:31.885312 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.885293 4982 scope.go:122] "RemoveContainer" containerID="2eb837ccd1af3f8712b30d47022687fe2c17e8013ee26e231bf5eb16986b6432" Oct 30 14:50:31.892916 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.892893 4982 scope.go:122] "RemoveContainer" containerID="2eb837ccd1af3f8712b30d47022687fe2c17e8013ee26e231bf5eb16986b6432" Oct 30 14:50:31.893199 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:50:31.893179 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2eb837ccd1af3f8712b30d47022687fe2c17e8013ee26e231bf5eb16986b6432\": container with ID starting with 2eb837ccd1af3f8712b30d47022687fe2c17e8013ee26e231bf5eb16986b6432 not found: ID does not exist" containerID="2eb837ccd1af3f8712b30d47022687fe2c17e8013ee26e231bf5eb16986b6432" Oct 30 14:50:31.893260 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.893204 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2eb837ccd1af3f8712b30d47022687fe2c17e8013ee26e231bf5eb16986b6432"} err="failed to get container status \"2eb837ccd1af3f8712b30d47022687fe2c17e8013ee26e231bf5eb16986b6432\": rpc error: code = NotFound desc = could not find container \"2eb837ccd1af3f8712b30d47022687fe2c17e8013ee26e231bf5eb16986b6432\": container with ID starting with 2eb837ccd1af3f8712b30d47022687fe2c17e8013ee26e231bf5eb16986b6432 not found: ID does not exist" Oct 30 14:50:31.893400 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.893383 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-00926c/distinct-2-w3-00926c-kserve-746b64d6b5-z66nw"] Oct 30 14:50:31.901534 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.901512 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-00926c/distinct-2-w3-00926c-kserve-746b64d6b5-z66nw"] Oct 30 14:50:31.974308 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.974281 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-00926c/distinct-w3-00926c-kserve-774df5b58-cv8vt"] Oct 30 14:50:31.977122 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:31.977099 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-00926c/distinct-w3-00926c-kserve-774df5b58-cv8vt"] Oct 30 14:50:32.013743 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:32.013721 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb"] Oct 30 14:50:32.019915 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:32.019893 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-00926c/simulator-w3-00926c-kserve-d6597cf86-4f8zb"] Oct 30 14:50:32.057855 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:32.057816 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7"] Oct 30 14:50:32.060339 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:32.060316 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-39529e/premium-w6-39529e-kserve-65fccd67-q8dc7"] Oct 30 14:50:32.097315 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:32.097259 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7"] Oct 30 14:50:32.104811 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:32.104773 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-39529e/simulator-w6-39529e-kserve-77b9765c58-jntm7"] Oct 30 14:50:32.398470 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:32.398394 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fa94d86-57ce-4d23-95d5-4efa965f6864" path="/var/lib/kubelet/pods/4fa94d86-57ce-4d23-95d5-4efa965f6864/volumes" Oct 30 14:50:32.398811 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:32.398797 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91f37a37-70fe-4627-b2dc-77a9668d9a5f" path="/var/lib/kubelet/pods/91f37a37-70fe-4627-b2dc-77a9668d9a5f/volumes" Oct 30 14:50:32.399155 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:32.399144 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab822fe6-4349-4051-b2fd-c6e5ceb50d08" path="/var/lib/kubelet/pods/ab822fe6-4349-4051-b2fd-c6e5ceb50d08/volumes" Oct 30 14:50:32.399795 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:32.399782 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec" path="/var/lib/kubelet/pods/b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec/volumes" Oct 30 14:50:32.400163 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:32.400151 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d65dee9b-4f8b-4f09-b00b-0793d732d3ab" path="/var/lib/kubelet/pods/d65dee9b-4f8b-4f09-b00b-0793d732d3ab/volumes" Oct 30 14:50:33.037870 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:33.037839 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:34.852737 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:34.852699 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-00926c-b8457bc55-7f4kk"] Oct 30 14:50:34.853108 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:34.852968 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-00926c-b8457bc55-7f4kk" podUID="f547db66-885a-4c00-8451-119811fffcac" containerName="maas-api" containerID="cri-o://128eff73857a3f12988f0eacf24635b09fa70e160d4045dbe6c36eae4d276f09" gracePeriod=30 Oct 30 14:50:34.959438 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:34.959397 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:35.122068 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:35.122008 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-00926c-b8457bc55-7f4kk" Oct 30 14:50:35.256553 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:35.256519 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f547db66-885a-4c00-8451-119811fffcac-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f547db66-885a-4c00-8451-119811fffcac-openshift-service-ca\") pod \"f547db66-885a-4c00-8451-119811fffcac\" (UID: \"f547db66-885a-4c00-8451-119811fffcac\") " Oct 30 14:50:35.256751 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:35.256578 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f547db66-885a-4c00-8451-119811fffcac-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f547db66-885a-4c00-8451-119811fffcac-maas-api-tls\") pod \"f547db66-885a-4c00-8451-119811fffcac\" (UID: \"f547db66-885a-4c00-8451-119811fffcac\") " Oct 30 14:50:35.256751 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:35.256614 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f547db66-885a-4c00-8451-119811fffcac-kube-api-access-7h6gw\" (UniqueName: \"kubernetes.io/projected/f547db66-885a-4c00-8451-119811fffcac-kube-api-access-7h6gw\") pod \"f547db66-885a-4c00-8451-119811fffcac\" (UID: \"f547db66-885a-4c00-8451-119811fffcac\") " Oct 30 14:50:35.256751 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:35.256687 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f547db66-885a-4c00-8451-119811fffcac-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f547db66-885a-4c00-8451-119811fffcac-metrics-certs\") pod \"f547db66-885a-4c00-8451-119811fffcac\" (UID: \"f547db66-885a-4c00-8451-119811fffcac\") " Oct 30 14:50:35.256955 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:35.256892 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f547db66-885a-4c00-8451-119811fffcac-openshift-service-ca" pod "f547db66-885a-4c00-8451-119811fffcac" (UID: "f547db66-885a-4c00-8451-119811fffcac"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:50:35.258840 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:35.258806 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f547db66-885a-4c00-8451-119811fffcac-maas-api-tls" pod "f547db66-885a-4c00-8451-119811fffcac" (UID: "f547db66-885a-4c00-8451-119811fffcac"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:50:35.258840 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:35.258814 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f547db66-885a-4c00-8451-119811fffcac-metrics-certs" pod "f547db66-885a-4c00-8451-119811fffcac" (UID: "f547db66-885a-4c00-8451-119811fffcac"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:50:35.259000 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:35.258919 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f547db66-885a-4c00-8451-119811fffcac-kube-api-access-7h6gw" pod "f547db66-885a-4c00-8451-119811fffcac" (UID: "f547db66-885a-4c00-8451-119811fffcac"). InnerVolumeSpecName "kube-api-access-7h6gw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:50:35.270004 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:35.269980 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-39529e-64b696694c-69mqr"] Oct 30 14:50:35.270219 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:35.270197 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-39529e-64b696694c-69mqr" podUID="76be0827-b086-49a8-873d-5141c13bfc6e" containerName="maas-api" containerID="cri-o://b98448b173f66469b134dc8731f11edcb2b32d1e651e7a93a84835499ff96c0e" gracePeriod=30 Oct 30 14:50:35.357874 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:35.357834 4982 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f547db66-885a-4c00-8451-119811fffcac-openshift-service-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:35.357874 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:35.357864 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f547db66-885a-4c00-8451-119811fffcac-maas-api-tls\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:35.357874 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:35.357873 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7h6gw\" (UniqueName: \"kubernetes.io/projected/f547db66-885a-4c00-8451-119811fffcac-kube-api-access-7h6gw\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:35.357874 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:35.357882 4982 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f547db66-885a-4c00-8451-119811fffcac-metrics-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:35.418880 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:35.418842 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:35.516132 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:35.516103 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-39529e-64b696694c-69mqr" Oct 30 14:50:35.603876 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:35.603843 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:35.661232 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:35.661160 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76be0827-b086-49a8-873d-5141c13bfc6e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76be0827-b086-49a8-873d-5141c13bfc6e-maas-api-tls\") pod \"76be0827-b086-49a8-873d-5141c13bfc6e\" (UID: \"76be0827-b086-49a8-873d-5141c13bfc6e\") " Oct 30 14:50:35.661232 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:35.661222 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76be0827-b086-49a8-873d-5141c13bfc6e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/76be0827-b086-49a8-873d-5141c13bfc6e-metrics-certs\") pod \"76be0827-b086-49a8-873d-5141c13bfc6e\" (UID: \"76be0827-b086-49a8-873d-5141c13bfc6e\") " Oct 30 14:50:35.661447 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:35.661336 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76be0827-b086-49a8-873d-5141c13bfc6e-kube-api-access-hp849\" (UniqueName: \"kubernetes.io/projected/76be0827-b086-49a8-873d-5141c13bfc6e-kube-api-access-hp849\") pod \"76be0827-b086-49a8-873d-5141c13bfc6e\" (UID: \"76be0827-b086-49a8-873d-5141c13bfc6e\") " Oct 30 14:50:35.661447 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:35.661372 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76be0827-b086-49a8-873d-5141c13bfc6e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76be0827-b086-49a8-873d-5141c13bfc6e-openshift-service-ca\") pod \"76be0827-b086-49a8-873d-5141c13bfc6e\" (UID: \"76be0827-b086-49a8-873d-5141c13bfc6e\") " Oct 30 14:50:35.661761 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:35.661726 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76be0827-b086-49a8-873d-5141c13bfc6e-openshift-service-ca" pod "76be0827-b086-49a8-873d-5141c13bfc6e" (UID: "76be0827-b086-49a8-873d-5141c13bfc6e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:50:35.663522 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:35.663493 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76be0827-b086-49a8-873d-5141c13bfc6e-metrics-certs" pod "76be0827-b086-49a8-873d-5141c13bfc6e" (UID: "76be0827-b086-49a8-873d-5141c13bfc6e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:50:35.663611 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:35.663515 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76be0827-b086-49a8-873d-5141c13bfc6e-maas-api-tls" pod "76be0827-b086-49a8-873d-5141c13bfc6e" (UID: "76be0827-b086-49a8-873d-5141c13bfc6e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:50:35.663611 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:35.663546 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76be0827-b086-49a8-873d-5141c13bfc6e-kube-api-access-hp849" pod "76be0827-b086-49a8-873d-5141c13bfc6e" (UID: "76be0827-b086-49a8-873d-5141c13bfc6e"). InnerVolumeSpecName "kube-api-access-hp849". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:50:35.752252 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:35.752222 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:35.762213 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:35.762186 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hp849\" (UniqueName: \"kubernetes.io/projected/76be0827-b086-49a8-873d-5141c13bfc6e-kube-api-access-hp849\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:35.762213 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:35.762210 4982 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76be0827-b086-49a8-873d-5141c13bfc6e-openshift-service-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:35.762360 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:35.762219 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76be0827-b086-49a8-873d-5141c13bfc6e-maas-api-tls\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:35.762360 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:35.762231 4982 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76be0827-b086-49a8-873d-5141c13bfc6e-metrics-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:35.885408 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:35.885377 4982 generic.go:356] "Generic (PLEG): container finished" podID="76be0827-b086-49a8-873d-5141c13bfc6e" containerID="b98448b173f66469b134dc8731f11edcb2b32d1e651e7a93a84835499ff96c0e" exitCode=0 Oct 30 14:50:35.885782 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:35.885427 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-39529e-64b696694c-69mqr" Oct 30 14:50:35.885782 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:35.885467 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-39529e-64b696694c-69mqr" event={"ID":"76be0827-b086-49a8-873d-5141c13bfc6e","Type":"ContainerDied","Data":"b98448b173f66469b134dc8731f11edcb2b32d1e651e7a93a84835499ff96c0e"} Oct 30 14:50:35.885782 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:35.885526 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-39529e-64b696694c-69mqr" event={"ID":"76be0827-b086-49a8-873d-5141c13bfc6e","Type":"ContainerDied","Data":"434a9c80cfb58b064a8fd9008a544383ea8ed96f103a2be3b8a4e41519bfeb42"} Oct 30 14:50:35.885782 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:35.885556 4982 scope.go:122] "RemoveContainer" containerID="b98448b173f66469b134dc8731f11edcb2b32d1e651e7a93a84835499ff96c0e" Oct 30 14:50:35.886905 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:35.886758 4982 generic.go:356] "Generic (PLEG): container finished" podID="f547db66-885a-4c00-8451-119811fffcac" containerID="128eff73857a3f12988f0eacf24635b09fa70e160d4045dbe6c36eae4d276f09" exitCode=0 Oct 30 14:50:35.886905 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:35.886794 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-00926c-b8457bc55-7f4kk" Oct 30 14:50:35.886905 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:35.886839 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-00926c-b8457bc55-7f4kk" event={"ID":"f547db66-885a-4c00-8451-119811fffcac","Type":"ContainerDied","Data":"128eff73857a3f12988f0eacf24635b09fa70e160d4045dbe6c36eae4d276f09"} Oct 30 14:50:35.886905 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:35.886883 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-00926c-b8457bc55-7f4kk" event={"ID":"f547db66-885a-4c00-8451-119811fffcac","Type":"ContainerDied","Data":"911575d8d3795f6fa443515dff2b871b19522271049f59e5cbb64d344a31a218"} Oct 30 14:50:35.895177 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:35.895020 4982 scope.go:122] "RemoveContainer" containerID="b98448b173f66469b134dc8731f11edcb2b32d1e651e7a93a84835499ff96c0e" Oct 30 14:50:35.895547 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:50:35.895525 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b98448b173f66469b134dc8731f11edcb2b32d1e651e7a93a84835499ff96c0e\": container with ID starting with b98448b173f66469b134dc8731f11edcb2b32d1e651e7a93a84835499ff96c0e not found: ID does not exist" containerID="b98448b173f66469b134dc8731f11edcb2b32d1e651e7a93a84835499ff96c0e" Oct 30 14:50:35.895667 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:35.895553 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b98448b173f66469b134dc8731f11edcb2b32d1e651e7a93a84835499ff96c0e"} err="failed to get container status \"b98448b173f66469b134dc8731f11edcb2b32d1e651e7a93a84835499ff96c0e\": rpc error: code = NotFound desc = could not find container \"b98448b173f66469b134dc8731f11edcb2b32d1e651e7a93a84835499ff96c0e\": container with ID starting with b98448b173f66469b134dc8731f11edcb2b32d1e651e7a93a84835499ff96c0e not found: ID does not exist" Oct 30 14:50:35.895667 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:35.895569 4982 scope.go:122] "RemoveContainer" containerID="128eff73857a3f12988f0eacf24635b09fa70e160d4045dbe6c36eae4d276f09" Oct 30 14:50:35.904286 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:35.904270 4982 scope.go:122] "RemoveContainer" containerID="128eff73857a3f12988f0eacf24635b09fa70e160d4045dbe6c36eae4d276f09" Oct 30 14:50:35.904528 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:50:35.904505 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"128eff73857a3f12988f0eacf24635b09fa70e160d4045dbe6c36eae4d276f09\": container with ID starting with 128eff73857a3f12988f0eacf24635b09fa70e160d4045dbe6c36eae4d276f09 not found: ID does not exist" containerID="128eff73857a3f12988f0eacf24635b09fa70e160d4045dbe6c36eae4d276f09" Oct 30 14:50:35.904592 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:35.904534 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"128eff73857a3f12988f0eacf24635b09fa70e160d4045dbe6c36eae4d276f09"} err="failed to get container status \"128eff73857a3f12988f0eacf24635b09fa70e160d4045dbe6c36eae4d276f09\": rpc error: code = NotFound desc = could not find container \"128eff73857a3f12988f0eacf24635b09fa70e160d4045dbe6c36eae4d276f09\": container with ID starting with 128eff73857a3f12988f0eacf24635b09fa70e160d4045dbe6c36eae4d276f09 not found: ID does not exist" Oct 30 14:50:35.943726 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:35.943672 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-39529e-64b696694c-69mqr"] Oct 30 14:50:35.947233 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:35.947213 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-39529e-64b696694c-69mqr"] Oct 30 14:50:35.984802 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:35.984781 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-00926c-b8457bc55-7f4kk"] Oct 30 14:50:35.990282 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:35.990262 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-00926c-b8457bc55-7f4kk"] Oct 30 14:50:36.113750 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:36.113726 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:36.258590 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:36.258510 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:36.398545 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:36.398511 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76be0827-b086-49a8-873d-5141c13bfc6e" path="/var/lib/kubelet/pods/76be0827-b086-49a8-873d-5141c13bfc6e/volumes" Oct 30 14:50:36.399061 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:36.399045 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f547db66-885a-4c00-8451-119811fffcac" path="/var/lib/kubelet/pods/f547db66-885a-4c00-8451-119811fffcac/volumes" Oct 30 14:50:36.759682 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:36.759635 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:40.752540 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:40.752499 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:41.287354 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:41.287320 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:41.609120 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:41.609086 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:41.814392 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:41.814360 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:42.521096 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:42.521056 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:43.727951 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:43.727921 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:43.877475 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:43.877445 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:44.038268 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:44.038192 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf"] Oct 30 14:50:44.038419 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:44.038371 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf" podUID="952e947d-d7c5-476b-a066-e4f171894acc" containerName="istio-proxy" containerID="cri-o://108bed20f473d35b9ba53c60245d40492657338b15c2ea115fc6e016f322acf4" gracePeriod=30 Oct 30 14:50:44.088901 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:44.088870 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:44.521938 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:44.521898 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:44.674182 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:44.674149 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:44.892584 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:44.892542 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:46.586029 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:46.585991 4982 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 14:50:46.586410 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:46.586047 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf" podUID="952e947d-d7c5-476b-a066-e4f171894acc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 14:50:47.936965 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:47.936931 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:48.622237 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:48.622205 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:49.291704 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:49.291677 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf" Oct 30 14:50:49.390321 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:49.390294 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/952e947d-d7c5-476b-a066-e4f171894acc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/952e947d-d7c5-476b-a066-e4f171894acc-credential-socket\") pod \"952e947d-d7c5-476b-a066-e4f171894acc\" (UID: \"952e947d-d7c5-476b-a066-e4f171894acc\") " Oct 30 14:50:49.390478 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:49.390346 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/952e947d-d7c5-476b-a066-e4f171894acc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/952e947d-d7c5-476b-a066-e4f171894acc-istio-envoy\") pod \"952e947d-d7c5-476b-a066-e4f171894acc\" (UID: \"952e947d-d7c5-476b-a066-e4f171894acc\") " Oct 30 14:50:49.390478 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:49.390384 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/952e947d-d7c5-476b-a066-e4f171894acc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/952e947d-d7c5-476b-a066-e4f171894acc-workload-socket\") pod \"952e947d-d7c5-476b-a066-e4f171894acc\" (UID: \"952e947d-d7c5-476b-a066-e4f171894acc\") " Oct 30 14:50:49.390478 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:49.390418 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/952e947d-d7c5-476b-a066-e4f171894acc-kube-api-access-rbrm8\" (UniqueName: \"kubernetes.io/projected/952e947d-d7c5-476b-a066-e4f171894acc-kube-api-access-rbrm8\") pod \"952e947d-d7c5-476b-a066-e4f171894acc\" (UID: \"952e947d-d7c5-476b-a066-e4f171894acc\") " Oct 30 14:50:49.390478 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:49.390456 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/952e947d-d7c5-476b-a066-e4f171894acc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/952e947d-d7c5-476b-a066-e4f171894acc-istio-data\") pod \"952e947d-d7c5-476b-a066-e4f171894acc\" (UID: \"952e947d-d7c5-476b-a066-e4f171894acc\") " Oct 30 14:50:49.390740 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:49.390559 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/952e947d-d7c5-476b-a066-e4f171894acc-credential-socket" pod "952e947d-d7c5-476b-a066-e4f171894acc" (UID: "952e947d-d7c5-476b-a066-e4f171894acc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:49.390740 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:49.390579 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/952e947d-d7c5-476b-a066-e4f171894acc-istio-token\" (UniqueName: \"kubernetes.io/projected/952e947d-d7c5-476b-a066-e4f171894acc-istio-token\") pod \"952e947d-d7c5-476b-a066-e4f171894acc\" (UID: \"952e947d-d7c5-476b-a066-e4f171894acc\") " Oct 30 14:50:49.390740 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:49.390627 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/952e947d-d7c5-476b-a066-e4f171894acc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/952e947d-d7c5-476b-a066-e4f171894acc-workload-certs\") pod \"952e947d-d7c5-476b-a066-e4f171894acc\" (UID: \"952e947d-d7c5-476b-a066-e4f171894acc\") " Oct 30 14:50:49.390904 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:49.390746 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/952e947d-d7c5-476b-a066-e4f171894acc-workload-socket" pod "952e947d-d7c5-476b-a066-e4f171894acc" (UID: "952e947d-d7c5-476b-a066-e4f171894acc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:49.390904 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:49.390763 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/952e947d-d7c5-476b-a066-e4f171894acc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/952e947d-d7c5-476b-a066-e4f171894acc-istio-podinfo\") pod \"952e947d-d7c5-476b-a066-e4f171894acc\" (UID: \"952e947d-d7c5-476b-a066-e4f171894acc\") " Oct 30 14:50:49.390904 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:49.390847 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/952e947d-d7c5-476b-a066-e4f171894acc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/952e947d-d7c5-476b-a066-e4f171894acc-istiod-ca-cert\") pod \"952e947d-d7c5-476b-a066-e4f171894acc\" (UID: \"952e947d-d7c5-476b-a066-e4f171894acc\") " Oct 30 14:50:49.391120 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:49.390922 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/952e947d-d7c5-476b-a066-e4f171894acc-workload-certs" pod "952e947d-d7c5-476b-a066-e4f171894acc" (UID: "952e947d-d7c5-476b-a066-e4f171894acc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:49.391120 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:49.390947 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/952e947d-d7c5-476b-a066-e4f171894acc-istio-data" pod "952e947d-d7c5-476b-a066-e4f171894acc" (UID: "952e947d-d7c5-476b-a066-e4f171894acc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:49.391234 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:49.391180 4982 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/952e947d-d7c5-476b-a066-e4f171894acc-workload-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:49.391234 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:49.391202 4982 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/952e947d-d7c5-476b-a066-e4f171894acc-istio-data\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:49.391234 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:49.391216 4982 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/952e947d-d7c5-476b-a066-e4f171894acc-workload-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:49.391234 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:49.391229 4982 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/952e947d-d7c5-476b-a066-e4f171894acc-credential-socket\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:49.391449 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:49.391252 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/952e947d-d7c5-476b-a066-e4f171894acc-istiod-ca-cert" pod "952e947d-d7c5-476b-a066-e4f171894acc" (UID: "952e947d-d7c5-476b-a066-e4f171894acc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:50:49.392988 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:49.392962 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/952e947d-d7c5-476b-a066-e4f171894acc-kube-api-access-rbrm8" pod "952e947d-d7c5-476b-a066-e4f171894acc" (UID: "952e947d-d7c5-476b-a066-e4f171894acc"). InnerVolumeSpecName "kube-api-access-rbrm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:50:49.392988 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:49.392979 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/952e947d-d7c5-476b-a066-e4f171894acc-istio-podinfo" pod "952e947d-d7c5-476b-a066-e4f171894acc" (UID: "952e947d-d7c5-476b-a066-e4f171894acc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:50:49.393151 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:49.393073 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/952e947d-d7c5-476b-a066-e4f171894acc-istio-envoy" pod "952e947d-d7c5-476b-a066-e4f171894acc" (UID: "952e947d-d7c5-476b-a066-e4f171894acc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:49.393151 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:49.393132 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/952e947d-d7c5-476b-a066-e4f171894acc-istio-token" pod "952e947d-d7c5-476b-a066-e4f171894acc" (UID: "952e947d-d7c5-476b-a066-e4f171894acc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:50:49.492208 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:49.492183 4982 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/952e947d-d7c5-476b-a066-e4f171894acc-istio-envoy\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:49.492354 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:49.492210 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbrm8\" (UniqueName: \"kubernetes.io/projected/952e947d-d7c5-476b-a066-e4f171894acc-kube-api-access-rbrm8\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:49.492354 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:49.492232 4982 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/952e947d-d7c5-476b-a066-e4f171894acc-istio-token\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:49.492354 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:49.492246 4982 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/952e947d-d7c5-476b-a066-e4f171894acc-istio-podinfo\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:49.492354 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:49.492259 4982 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/952e947d-d7c5-476b-a066-e4f171894acc-istiod-ca-cert\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:50:49.965109 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:49.965073 4982 generic.go:356] "Generic (PLEG): container finished" podID="952e947d-d7c5-476b-a066-e4f171894acc" containerID="108bed20f473d35b9ba53c60245d40492657338b15c2ea115fc6e016f322acf4" exitCode=0 Oct 30 14:50:49.965257 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:49.965125 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf" Oct 30 14:50:49.965257 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:49.965155 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf" event={"ID":"952e947d-d7c5-476b-a066-e4f171894acc","Type":"ContainerDied","Data":"108bed20f473d35b9ba53c60245d40492657338b15c2ea115fc6e016f322acf4"} Oct 30 14:50:49.965257 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:49.965211 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf" event={"ID":"952e947d-d7c5-476b-a066-e4f171894acc","Type":"ContainerDied","Data":"917545c31707f3302f5d57d98ba312d65d195db40382d1d51e3c8ad862e091cd"} Oct 30 14:50:49.965257 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:49.965233 4982 scope.go:122] "RemoveContainer" containerID="108bed20f473d35b9ba53c60245d40492657338b15c2ea115fc6e016f322acf4" Oct 30 14:50:49.974158 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:49.974140 4982 scope.go:122] "RemoveContainer" containerID="108bed20f473d35b9ba53c60245d40492657338b15c2ea115fc6e016f322acf4" Oct 30 14:50:49.974433 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:50:49.974411 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"108bed20f473d35b9ba53c60245d40492657338b15c2ea115fc6e016f322acf4\": container with ID starting with 108bed20f473d35b9ba53c60245d40492657338b15c2ea115fc6e016f322acf4 not found: ID does not exist" containerID="108bed20f473d35b9ba53c60245d40492657338b15c2ea115fc6e016f322acf4" Oct 30 14:50:49.974485 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:49.974442 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"108bed20f473d35b9ba53c60245d40492657338b15c2ea115fc6e016f322acf4"} err="failed to get container status \"108bed20f473d35b9ba53c60245d40492657338b15c2ea115fc6e016f322acf4\": rpc error: code = NotFound desc = could not find container \"108bed20f473d35b9ba53c60245d40492657338b15c2ea115fc6e016f322acf4\": container with ID starting with 108bed20f473d35b9ba53c60245d40492657338b15c2ea115fc6e016f322acf4 not found: ID does not exist" Oct 30 14:50:50.034971 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:50.034942 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf"] Oct 30 14:50:50.048354 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:50.048334 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-7pnxf"] Oct 30 14:50:50.406006 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:50.405970 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="952e947d-d7c5-476b-a066-e4f171894acc" path="/var/lib/kubelet/pods/952e947d-d7c5-476b-a066-e4f171894acc/volumes" Oct 30 14:50:52.188962 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:52.188928 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:52.485544 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:52.485449 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:52.719955 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:52.719921 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:53.016783 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:53.016751 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:54.375237 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:54.375204 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:54.530116 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:54.530082 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:55.818449 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:55.818417 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:55.967628 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:50:55.967595 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:58.011972 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:50:58.011932 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:51:02.478060 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:51:02.478027 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:51:02.626207 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:51:02.626175 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:51:28.009509 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:51:28.009471 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:51:35.553198 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:51:35.553153 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:51:36.396463 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:51:36.396430 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:51:40.771611 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:51:40.771570 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:51:40.909050 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:51:40.909011 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:51:48.060930 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:51:48.060895 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:51:48.718437 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:51:48.718389 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:51:53.020767 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:51:53.020736 4982 scope.go:122] "RemoveContainer" containerID="d91d82838a966ad0d90ec6d3d95c548f7141768b29a80c2708d179ae60d5eb41" Oct 30 14:51:58.010206 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:51:58.010168 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:52:21.972076 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:21.972042 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:52:22.107550 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:22.107518 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:52:28.012001 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:52:28.011957 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:52:41.687668 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.687582 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-f2kp6"] Oct 30 14:52:41.688132 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.687971 4982 cpu_manager.go:414] "containerMap: removing container" podUID="42ebc63e-3f5f-4aa3-b2cc-b9d92e3f9e0c" containerName="payload-processing" Oct 30 14:52:41.688132 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.687983 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42ebc63e-3f5f-4aa3-b2cc-b9d92e3f9e0c" containerName="payload-processing" Oct 30 14:52:41.688132 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.687998 4982 cpu_manager.go:414] "containerMap: removing container" podUID="ab822fe6-4349-4051-b2fd-c6e5ceb50d08" containerName="main" Oct 30 14:52:41.688132 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.688003 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab822fe6-4349-4051-b2fd-c6e5ceb50d08" containerName="main" Oct 30 14:52:41.688132 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.688013 4982 cpu_manager.go:414] "containerMap: removing container" podUID="9b853a87-0adf-46f0-b45d-c17b69a68495" containerName="istio-proxy" Oct 30 14:52:41.688132 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.688020 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b853a87-0adf-46f0-b45d-c17b69a68495" containerName="istio-proxy" Oct 30 14:52:41.688132 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.688032 4982 cpu_manager.go:414] "containerMap: removing container" podUID="f547db66-885a-4c00-8451-119811fffcac" containerName="maas-api" Oct 30 14:52:41.688132 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.688037 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f547db66-885a-4c00-8451-119811fffcac" containerName="maas-api" Oct 30 14:52:41.688132 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.688044 4982 cpu_manager.go:414] "containerMap: removing container" podUID="757d5732-c143-453c-9149-5744017ea635" containerName="maas-api" Oct 30 14:52:41.688132 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.688049 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="757d5732-c143-453c-9149-5744017ea635" containerName="maas-api" Oct 30 14:52:41.688132 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.688055 4982 cpu_manager.go:414] "containerMap: removing container" podUID="02ab3f3b-3aac-4acd-8340-3f63c789ec8c" containerName="payload-pre-processing" Oct 30 14:52:41.688132 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.688060 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02ab3f3b-3aac-4acd-8340-3f63c789ec8c" containerName="payload-pre-processing" Oct 30 14:52:41.688132 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.688068 4982 cpu_manager.go:414] "containerMap: removing container" podUID="52b260c5-122f-4e2a-8344-56d8efbb66f6" containerName="istio-proxy" Oct 30 14:52:41.688132 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.688073 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52b260c5-122f-4e2a-8344-56d8efbb66f6" containerName="istio-proxy" Oct 30 14:52:41.688132 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.688081 4982 cpu_manager.go:414] "containerMap: removing container" podUID="4fa94d86-57ce-4d23-95d5-4efa965f6864" containerName="main" Oct 30 14:52:41.688132 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.688086 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fa94d86-57ce-4d23-95d5-4efa965f6864" containerName="main" Oct 30 14:52:41.688132 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.688092 4982 cpu_manager.go:414] "containerMap: removing container" podUID="c99896fe-79b1-4368-aeb5-a81182e7ae13" containerName="istio-proxy" Oct 30 14:52:41.688132 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.688098 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c99896fe-79b1-4368-aeb5-a81182e7ae13" containerName="istio-proxy" Oct 30 14:52:41.688132 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.688109 4982 cpu_manager.go:414] "containerMap: removing container" podUID="6c3e3a0f-43a7-41d3-acf7-c6299b1f5668" containerName="revoke-keys" Oct 30 14:52:41.688132 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.688115 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c3e3a0f-43a7-41d3-acf7-c6299b1f5668" containerName="revoke-keys" Oct 30 14:52:41.688132 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.688124 4982 cpu_manager.go:414] "containerMap: removing container" podUID="946696b9-91bd-437a-9f23-c16b73a5a9a6" containerName="main" Oct 30 14:52:41.688132 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.688129 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="946696b9-91bd-437a-9f23-c16b73a5a9a6" containerName="main" Oct 30 14:52:41.688132 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.688138 4982 cpu_manager.go:414] "containerMap: removing container" podUID="76be0827-b086-49a8-873d-5141c13bfc6e" containerName="maas-api" Oct 30 14:52:41.688132 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.688143 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76be0827-b086-49a8-873d-5141c13bfc6e" containerName="maas-api" Oct 30 14:52:41.688912 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.688157 4982 cpu_manager.go:414] "containerMap: removing container" podUID="d65dee9b-4f8b-4f09-b00b-0793d732d3ab" containerName="main" Oct 30 14:52:41.688912 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.688161 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d65dee9b-4f8b-4f09-b00b-0793d732d3ab" containerName="main" Oct 30 14:52:41.688912 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.688168 4982 cpu_manager.go:414] "containerMap: removing container" podUID="952e947d-d7c5-476b-a066-e4f171894acc" containerName="istio-proxy" Oct 30 14:52:41.688912 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.688172 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="952e947d-d7c5-476b-a066-e4f171894acc" containerName="istio-proxy" Oct 30 14:52:41.688912 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.688180 4982 cpu_manager.go:414] "containerMap: removing container" podUID="b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec" containerName="main" Oct 30 14:52:41.688912 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.688186 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec" containerName="main" Oct 30 14:52:41.688912 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.688195 4982 cpu_manager.go:414] "containerMap: removing container" podUID="c99896fe-79b1-4368-aeb5-a81182e7ae13" containerName="istio-proxy" Oct 30 14:52:41.688912 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.688199 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c99896fe-79b1-4368-aeb5-a81182e7ae13" containerName="istio-proxy" Oct 30 14:52:41.688912 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.688208 4982 cpu_manager.go:414] "containerMap: removing container" podUID="91f37a37-70fe-4627-b2dc-77a9668d9a5f" containerName="main" Oct 30 14:52:41.688912 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.688212 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91f37a37-70fe-4627-b2dc-77a9668d9a5f" containerName="main" Oct 30 14:52:41.688912 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.688269 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="946696b9-91bd-437a-9f23-c16b73a5a9a6" containerName="main" Oct 30 14:52:41.688912 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.688279 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="76be0827-b086-49a8-873d-5141c13bfc6e" containerName="maas-api" Oct 30 14:52:41.688912 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.688287 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="52b260c5-122f-4e2a-8344-56d8efbb66f6" containerName="istio-proxy" Oct 30 14:52:41.688912 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.688294 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="d65dee9b-4f8b-4f09-b00b-0793d732d3ab" containerName="main" Oct 30 14:52:41.688912 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.688299 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="952e947d-d7c5-476b-a066-e4f171894acc" containerName="istio-proxy" Oct 30 14:52:41.688912 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.688312 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6b9d7e6-4eeb-4798-9cd5-0c1f669cd6ec" containerName="main" Oct 30 14:52:41.688912 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.688317 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="c99896fe-79b1-4368-aeb5-a81182e7ae13" containerName="istio-proxy" Oct 30 14:52:41.688912 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.688324 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="91f37a37-70fe-4627-b2dc-77a9668d9a5f" containerName="main" Oct 30 14:52:41.688912 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.688329 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="42ebc63e-3f5f-4aa3-b2cc-b9d92e3f9e0c" containerName="payload-processing" Oct 30 14:52:41.688912 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.688337 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab822fe6-4349-4051-b2fd-c6e5ceb50d08" containerName="main" Oct 30 14:52:41.688912 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.688346 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="f547db66-885a-4c00-8451-119811fffcac" containerName="maas-api" Oct 30 14:52:41.688912 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.688353 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="757d5732-c143-453c-9149-5744017ea635" containerName="maas-api" Oct 30 14:52:41.688912 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.688357 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="02ab3f3b-3aac-4acd-8340-3f63c789ec8c" containerName="payload-pre-processing" Oct 30 14:52:41.688912 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.688363 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fa94d86-57ce-4d23-95d5-4efa965f6864" containerName="main" Oct 30 14:52:41.688912 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.688367 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="c99896fe-79b1-4368-aeb5-a81182e7ae13" containerName="istio-proxy" Oct 30 14:52:41.688912 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.688372 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c3e3a0f-43a7-41d3-acf7-c6299b1f5668" containerName="revoke-keys" Oct 30 14:52:41.691633 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.691613 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-f2kp6" Oct 30 14:52:41.726064 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.726034 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-f2kp6"] Oct 30 14:52:41.727894 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.727870 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kxfp8"] Oct 30 14:52:41.728197 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.728183 4982 cpu_manager.go:414] "containerMap: removing container" podUID="52b260c5-122f-4e2a-8344-56d8efbb66f6" containerName="istio-proxy" Oct 30 14:52:41.728197 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.728196 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52b260c5-122f-4e2a-8344-56d8efbb66f6" containerName="istio-proxy" Oct 30 14:52:41.728351 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.728279 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="52b260c5-122f-4e2a-8344-56d8efbb66f6" containerName="istio-proxy" Oct 30 14:52:41.731203 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.731178 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kxfp8" Oct 30 14:52:41.746575 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.746548 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-f2kp6\" (UID: \"f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-f2kp6" Oct 30 14:52:41.746726 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.746589 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-f2kp6\" (UID: \"f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-f2kp6" Oct 30 14:52:41.746726 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.746623 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-f2kp6\" (UID: \"f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-f2kp6" Oct 30 14:52:41.746881 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.746769 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-f2kp6\" (UID: \"f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-f2kp6" Oct 30 14:52:41.746881 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.746813 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzl7k\" (UniqueName: \"kubernetes.io/projected/f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc-kube-api-access-tzl7k\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-f2kp6\" (UID: \"f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-f2kp6" Oct 30 14:52:41.746881 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.746845 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-f2kp6\" (UID: \"f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-f2kp6" Oct 30 14:52:41.746881 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.746861 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-f2kp6\" (UID: \"f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-f2kp6" Oct 30 14:52:41.747025 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.746929 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-f2kp6\" (UID: \"f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-f2kp6" Oct 30 14:52:41.747025 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.746947 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-f2kp6\" (UID: \"f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-f2kp6" Oct 30 14:52:41.752478 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.752448 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kxfp8"] Oct 30 14:52:41.847590 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.847558 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-f2kp6\" (UID: \"f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-f2kp6" Oct 30 14:52:41.847590 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.847591 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-f2kp6\" (UID: \"f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-f2kp6" Oct 30 14:52:41.847766 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.847612 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/556451f5-3d37-40a5-bf8f-f470d05ef183-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kxfp8\" (UID: \"556451f5-3d37-40a5-bf8f-f470d05ef183\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kxfp8" Oct 30 14:52:41.847766 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.847706 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/556451f5-3d37-40a5-bf8f-f470d05ef183-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kxfp8\" (UID: \"556451f5-3d37-40a5-bf8f-f470d05ef183\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kxfp8" Oct 30 14:52:41.847836 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.847808 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/556451f5-3d37-40a5-bf8f-f470d05ef183-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kxfp8\" (UID: \"556451f5-3d37-40a5-bf8f-f470d05ef183\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kxfp8" Oct 30 14:52:41.847876 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.847849 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/556451f5-3d37-40a5-bf8f-f470d05ef183-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kxfp8\" (UID: \"556451f5-3d37-40a5-bf8f-f470d05ef183\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kxfp8" Oct 30 14:52:41.847921 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.847893 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/556451f5-3d37-40a5-bf8f-f470d05ef183-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kxfp8\" (UID: \"556451f5-3d37-40a5-bf8f-f470d05ef183\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kxfp8" Oct 30 14:52:41.848028 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.848005 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffq28\" (UniqueName: \"kubernetes.io/projected/556451f5-3d37-40a5-bf8f-f470d05ef183-kube-api-access-ffq28\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kxfp8\" (UID: \"556451f5-3d37-40a5-bf8f-f470d05ef183\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kxfp8" Oct 30 14:52:41.848091 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.848050 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-f2kp6\" (UID: \"f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-f2kp6" Oct 30 14:52:41.848091 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.848051 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-f2kp6\" (UID: \"f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-f2kp6" Oct 30 14:52:41.848179 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.848110 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-f2kp6\" (UID: \"f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-f2kp6" Oct 30 14:52:41.848179 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.848134 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-f2kp6\" (UID: \"f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-f2kp6" Oct 30 14:52:41.848179 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.848148 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-f2kp6\" (UID: \"f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-f2kp6" Oct 30 14:52:41.848179 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.848169 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-f2kp6\" (UID: \"f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-f2kp6" Oct 30 14:52:41.848401 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.848188 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/556451f5-3d37-40a5-bf8f-f470d05ef183-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kxfp8\" (UID: \"556451f5-3d37-40a5-bf8f-f470d05ef183\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kxfp8" Oct 30 14:52:41.848401 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.848213 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-f2kp6\" (UID: \"f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-f2kp6" Oct 30 14:52:41.848401 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.848244 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzl7k\" (UniqueName: \"kubernetes.io/projected/f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc-kube-api-access-tzl7k\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-f2kp6\" (UID: \"f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-f2kp6" Oct 30 14:52:41.848401 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.848276 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/556451f5-3d37-40a5-bf8f-f470d05ef183-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kxfp8\" (UID: \"556451f5-3d37-40a5-bf8f-f470d05ef183\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kxfp8" Oct 30 14:52:41.848401 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.848307 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/556451f5-3d37-40a5-bf8f-f470d05ef183-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kxfp8\" (UID: \"556451f5-3d37-40a5-bf8f-f470d05ef183\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kxfp8" Oct 30 14:52:41.848401 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.848346 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-f2kp6\" (UID: \"f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-f2kp6" Oct 30 14:52:41.848704 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.848404 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-f2kp6\" (UID: \"f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-f2kp6" Oct 30 14:52:41.848704 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.848499 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-f2kp6\" (UID: \"f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-f2kp6" Oct 30 14:52:41.848704 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.848593 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-f2kp6\" (UID: \"f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-f2kp6" Oct 30 14:52:41.850241 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.850214 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-f2kp6\" (UID: \"f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-f2kp6" Oct 30 14:52:41.850540 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.850517 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-f2kp6\" (UID: \"f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-f2kp6" Oct 30 14:52:41.873810 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.873790 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-f2kp6\" (UID: \"f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-f2kp6" Oct 30 14:52:41.873902 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.873876 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzl7k\" (UniqueName: \"kubernetes.io/projected/f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc-kube-api-access-tzl7k\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-f2kp6\" (UID: \"f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-f2kp6" Oct 30 14:52:41.949714 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.949619 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/556451f5-3d37-40a5-bf8f-f470d05ef183-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kxfp8\" (UID: \"556451f5-3d37-40a5-bf8f-f470d05ef183\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kxfp8" Oct 30 14:52:41.949714 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.949690 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/556451f5-3d37-40a5-bf8f-f470d05ef183-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kxfp8\" (UID: \"556451f5-3d37-40a5-bf8f-f470d05ef183\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kxfp8" Oct 30 14:52:41.949886 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.949721 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/556451f5-3d37-40a5-bf8f-f470d05ef183-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kxfp8\" (UID: \"556451f5-3d37-40a5-bf8f-f470d05ef183\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kxfp8" Oct 30 14:52:41.949886 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.949749 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/556451f5-3d37-40a5-bf8f-f470d05ef183-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kxfp8\" (UID: \"556451f5-3d37-40a5-bf8f-f470d05ef183\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kxfp8" Oct 30 14:52:41.949886 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.949804 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/556451f5-3d37-40a5-bf8f-f470d05ef183-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kxfp8\" (UID: \"556451f5-3d37-40a5-bf8f-f470d05ef183\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kxfp8" Oct 30 14:52:41.949886 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.949845 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffq28\" (UniqueName: \"kubernetes.io/projected/556451f5-3d37-40a5-bf8f-f470d05ef183-kube-api-access-ffq28\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kxfp8\" (UID: \"556451f5-3d37-40a5-bf8f-f470d05ef183\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kxfp8" Oct 30 14:52:41.950103 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.949900 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/556451f5-3d37-40a5-bf8f-f470d05ef183-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kxfp8\" (UID: \"556451f5-3d37-40a5-bf8f-f470d05ef183\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kxfp8" Oct 30 14:52:41.950103 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.949946 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/556451f5-3d37-40a5-bf8f-f470d05ef183-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kxfp8\" (UID: \"556451f5-3d37-40a5-bf8f-f470d05ef183\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kxfp8" Oct 30 14:52:41.950103 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.949972 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/556451f5-3d37-40a5-bf8f-f470d05ef183-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kxfp8\" (UID: \"556451f5-3d37-40a5-bf8f-f470d05ef183\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kxfp8" Oct 30 14:52:41.950275 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.950130 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/556451f5-3d37-40a5-bf8f-f470d05ef183-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kxfp8\" (UID: \"556451f5-3d37-40a5-bf8f-f470d05ef183\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kxfp8" Oct 30 14:52:41.950275 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.950156 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/556451f5-3d37-40a5-bf8f-f470d05ef183-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kxfp8\" (UID: \"556451f5-3d37-40a5-bf8f-f470d05ef183\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kxfp8" Oct 30 14:52:41.950384 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.950354 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/556451f5-3d37-40a5-bf8f-f470d05ef183-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kxfp8\" (UID: \"556451f5-3d37-40a5-bf8f-f470d05ef183\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kxfp8" Oct 30 14:52:41.950384 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.950369 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/556451f5-3d37-40a5-bf8f-f470d05ef183-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kxfp8\" (UID: \"556451f5-3d37-40a5-bf8f-f470d05ef183\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kxfp8" Oct 30 14:52:41.950578 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.950554 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/556451f5-3d37-40a5-bf8f-f470d05ef183-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kxfp8\" (UID: \"556451f5-3d37-40a5-bf8f-f470d05ef183\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kxfp8" Oct 30 14:52:41.951851 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.951825 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/556451f5-3d37-40a5-bf8f-f470d05ef183-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kxfp8\" (UID: \"556451f5-3d37-40a5-bf8f-f470d05ef183\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kxfp8" Oct 30 14:52:41.952023 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.952000 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/556451f5-3d37-40a5-bf8f-f470d05ef183-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kxfp8\" (UID: \"556451f5-3d37-40a5-bf8f-f470d05ef183\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kxfp8" Oct 30 14:52:41.975387 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.975361 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/556451f5-3d37-40a5-bf8f-f470d05ef183-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kxfp8\" (UID: \"556451f5-3d37-40a5-bf8f-f470d05ef183\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kxfp8" Oct 30 14:52:41.975471 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:41.975426 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffq28\" (UniqueName: \"kubernetes.io/projected/556451f5-3d37-40a5-bf8f-f470d05ef183-kube-api-access-ffq28\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kxfp8\" (UID: \"556451f5-3d37-40a5-bf8f-f470d05ef183\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kxfp8" Oct 30 14:52:42.004225 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:42.004204 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-f2kp6" Oct 30 14:52:42.044783 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:42.044747 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kxfp8" Oct 30 14:52:42.319424 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:42.319398 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-f2kp6"] Oct 30 14:52:42.321969 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:52:42.321943 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4dd3cf1_c4f4_45ee_8ef7_294af175cbdc.slice/crio-c320d79c5a523521913173436a8c39c192d9986bad45edde9a0dc672bbb6076b WatchSource:0}: Error finding container c320d79c5a523521913173436a8c39c192d9986bad45edde9a0dc672bbb6076b: Status 404 returned error can't find the container with id c320d79c5a523521913173436a8c39c192d9986bad45edde9a0dc672bbb6076b Oct 30 14:52:42.323933 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:42.323905 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:52:42.323993 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:42.323964 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:52:42.324030 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:42.323991 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:52:42.363333 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:42.363312 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kxfp8"] Oct 30 14:52:42.365464 ip-10-0-140-105 kubenswrapper[4982]: W1030 14:52:42.365439 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod556451f5_3d37_40a5_bf8f_f470d05ef183.slice/crio-82b4f843c3c55420591f12d079d9c9ed98f33cf03504e7846087871b38ffeaf5 WatchSource:0}: Error finding container 82b4f843c3c55420591f12d079d9c9ed98f33cf03504e7846087871b38ffeaf5: Status 404 returned error can't find the container with id 82b4f843c3c55420591f12d079d9c9ed98f33cf03504e7846087871b38ffeaf5 Oct 30 14:52:42.367356 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:42.367327 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:52:42.367439 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:42.367383 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:52:42.367439 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:42.367414 4982 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Oct 30 14:52:42.481448 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:42.481410 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-f2kp6" event={"ID":"f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc","Type":"ContainerStarted","Data":"e27126e8c5c65577a49beb9779c885a6c5ad56a78ec65336272c8b038c7fc6ea"} Oct 30 14:52:42.481669 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:42.481473 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-f2kp6" event={"ID":"f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc","Type":"ContainerStarted","Data":"c320d79c5a523521913173436a8c39c192d9986bad45edde9a0dc672bbb6076b"} Oct 30 14:52:42.483082 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:42.483051 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kxfp8" event={"ID":"556451f5-3d37-40a5-bf8f-f470d05ef183","Type":"ContainerStarted","Data":"1b1438a6ecac6b3aeb05c037b33abf8b802186e657ad732d5ac25fa8736b720b"} Oct 30 14:52:42.483192 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:42.483104 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kxfp8" event={"ID":"556451f5-3d37-40a5-bf8f-f470d05ef183","Type":"ContainerStarted","Data":"82b4f843c3c55420591f12d079d9c9ed98f33cf03504e7846087871b38ffeaf5"} Oct 30 14:52:42.530529 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:42.530478 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-f2kp6" podStartSLOduration=1.5304649540000002 podStartE2EDuration="1.530464954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:52:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:52:42.528950329 +0000 UTC m=+3832.273084370" watchObservedRunningTime="2026-10-30 14:52:42.530464954 +0000 UTC m=+3832.274598991" Oct 30 14:52:42.577188 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:42.577097 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kxfp8" podStartSLOduration=1.577083675 podStartE2EDuration="1.577083675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:52:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:52:42.576392078 +0000 UTC m=+3832.320526117" watchObservedRunningTime="2026-10-30 14:52:42.577083675 +0000 UTC m=+3832.321217788" Oct 30 14:52:43.013742 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:43.013300 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-f2kp6" Oct 30 14:52:43.013742 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:43.013447 4982 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-f2kp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body= Oct 30 14:52:43.013742 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:43.013493 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-f2kp6" podUID="f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" Oct 30 14:52:43.046012 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:43.045927 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kxfp8" Oct 30 14:52:43.047275 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:43.047189 4982 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-kxfp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body= Oct 30 14:52:43.047275 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:43.047233 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kxfp8" podUID="556451f5-3d37-40a5-bf8f-f470d05ef183" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" Oct 30 14:52:44.014664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:44.006981 4982 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-f2kp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body= Oct 30 14:52:44.014664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:44.007043 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-f2kp6" podUID="f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" Oct 30 14:52:44.046095 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:44.046067 4982 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-kxfp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body= Oct 30 14:52:44.046283 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:44.046266 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kxfp8" podUID="556451f5-3d37-40a5-bf8f-f470d05ef183" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" Oct 30 14:52:45.006002 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:45.005955 4982 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-f2kp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body= Oct 30 14:52:45.006264 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:45.006236 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-f2kp6" podUID="f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" Oct 30 14:52:45.045525 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:45.045493 4982 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-kxfp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body= Oct 30 14:52:45.045960 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:45.045739 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kxfp8" podUID="556451f5-3d37-40a5-bf8f-f470d05ef183" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" Oct 30 14:52:46.009664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:46.007993 4982 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-f2kp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body= Oct 30 14:52:46.009664 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:46.008056 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-f2kp6" podUID="f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" Oct 30 14:52:46.045940 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:46.045906 4982 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-kxfp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body= Oct 30 14:52:46.046285 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:46.045964 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kxfp8" podUID="556451f5-3d37-40a5-bf8f-f470d05ef183" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" Oct 30 14:52:47.007037 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:47.006995 4982 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-f2kp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body= Oct 30 14:52:47.007373 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:47.007332 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-f2kp6" podUID="f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" Oct 30 14:52:47.046105 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:47.046074 4982 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-kxfp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body= Oct 30 14:52:47.046455 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:47.046116 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kxfp8" podUID="556451f5-3d37-40a5-bf8f-f470d05ef183" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" Oct 30 14:52:48.005000 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:48.004963 4982 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-f2kp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body= Oct 30 14:52:48.005287 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:48.005258 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-f2kp6" podUID="f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" Oct 30 14:52:48.046689 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:48.046661 4982 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-kxfp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body= Oct 30 14:52:48.047101 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:48.047079 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kxfp8" podUID="556451f5-3d37-40a5-bf8f-f470d05ef183" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" Oct 30 14:52:48.201774 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:48.201730 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:52:49.005392 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:49.005350 4982 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-f2kp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body= Oct 30 14:52:49.005616 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:49.005423 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-f2kp6" podUID="f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" Oct 30 14:52:49.045875 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:49.045835 4982 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-kxfp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body= Oct 30 14:52:49.046053 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:49.045894 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kxfp8" podUID="556451f5-3d37-40a5-bf8f-f470d05ef183" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" Oct 30 14:52:51.011301 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:51.011259 4982 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-f2kp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:52:51.011950 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:51.011915 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-f2kp6" podUID="f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:52:51.046116 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:51.046074 4982 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-kxfp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:52:51.046416 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:51.046387 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kxfp8" podUID="556451f5-3d37-40a5-bf8f-f470d05ef183" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:52:52.053774 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:52.053740 4982 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-f2kp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:52:52.054141 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:52.053791 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-f2kp6" podUID="f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:52:52.088150 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:52.087805 4982 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-kxfp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:52:52.088150 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:52.087854 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kxfp8" podUID="556451f5-3d37-40a5-bf8f-f470d05ef183" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:52:53.095861 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:53.095813 4982 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-f2kp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:52:53.096461 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:53.096422 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-f2kp6" podUID="f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:52:53.129790 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:53.129757 4982 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-kxfp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:52:53.129936 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:53.129813 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kxfp8" podUID="556451f5-3d37-40a5-bf8f-f470d05ef183" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:52:54.097301 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:54.097256 4982 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-f2kp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:52:54.097817 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:54.097326 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-f2kp6" podUID="f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:52:54.171814 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:54.171778 4982 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-kxfp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:52:54.171969 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:54.171842 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kxfp8" podUID="556451f5-3d37-40a5-bf8f-f470d05ef183" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:52:55.138819 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:55.138771 4982 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-f2kp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:52:55.139324 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:55.138834 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-f2kp6" podUID="f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:52:55.213843 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:55.213781 4982 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-kxfp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:52:55.214105 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:55.214067 4982 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kxfp8" podUID="556451f5-3d37-40a5-bf8f-f470d05ef183" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:52:55.550176 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:55.550099 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kxfp8" Oct 30 14:52:55.550419 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:55.550391 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kxfp8" Oct 30 14:52:55.551163 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:55.551143 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kxfp8" Oct 30 14:52:55.731811 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:55.731779 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-f2kp6" Oct 30 14:52:55.732149 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:55.732070 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-f2kp6" Oct 30 14:52:55.732733 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:52:55.732712 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-f2kp6" Oct 30 14:52:58.010439 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:52:58.010404 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:53:28.009687 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:53:28.009617 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:53:42.031980 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:53:42.031948 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50f76dcc-fngln"] Oct 30 14:53:42.044953 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:53:42.044907 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50f76dcc-fngln"] Oct 30 14:53:42.396939 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:53:42.396902 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31ed90a7-a67c-4ece-b5b0-0f88f6860ec9" path="/var/lib/kubelet/pods/31ed90a7-a67c-4ece-b5b0-0f88f6860ec9/volumes" Oct 30 14:53:48.315151 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:53:48.315117 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:53:51.026221 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:53:51.026188 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ntlqq_dc967875-1546-40bb-8376-fde09416c204/console-operator/2.log" Oct 30 14:53:51.031064 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:53:51.031043 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhxjf_aca47fe8-7068-4743-b6ab-283ee0874f66/ovn-acl-logging/0.log" Oct 30 14:53:51.041337 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:53:51.041318 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ntlqq_dc967875-1546-40bb-8376-fde09416c204/console-operator/2.log" Oct 30 14:53:51.045453 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:53:51.045436 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhxjf_aca47fe8-7068-4743-b6ab-283ee0874f66/ovn-acl-logging/0.log" Oct 30 14:53:58.089971 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:53:58.089919 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:54:15.416162 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:54:15.416125 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:54:15.553093 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:54:15.553063 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:54:28.010234 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:54:28.010202 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:54:43.537528 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:54:43.537491 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:54:48.444382 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:54:48.444345 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:54:53.056336 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:54:53.056291 4982 scope.go:122] "RemoveContainer" containerID="a4283f1bf83083d0bc18ac2cb920d901a76b53449add986e9a9941a550c6d097" Oct 30 14:54:53.066371 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:54:53.066348 4982 scope.go:122] "RemoveContainer" containerID="d3c2119525ebc3e3c977a5f4c7b35033eda59f343c52dfeeee616111597afe89" Oct 30 14:54:53.074193 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:54:53.074170 4982 scope.go:122] "RemoveContainer" containerID="86d268d59d78d8923362c622e428d7fac3c9c0a5caf1eab2db638c732cda1880" Oct 30 14:54:53.081780 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:54:53.081750 4982 scope.go:122] "RemoveContainer" containerID="b6e2587053490171965f4d54570c62cc10dca4cfc29d50dabbfc8751c10d9ff6" Oct 30 14:54:53.093845 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:54:53.093817 4982 scope.go:122] "RemoveContainer" containerID="f5b48ff42807153a83eb8dc1d90fc3c9e36a26a90877017134bead26bec531e5" Oct 30 14:54:58.010868 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:54:58.010828 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:55:28.010244 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:55:28.010206 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:55:48.509130 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:55:48.509057 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:55:53.129967 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:55:53.129931 4982 scope.go:122] "RemoveContainer" containerID="0ca0030799625f39defcd6871a7cdf06e3f2c6361168594c474aa860231a0a51" Oct 30 14:55:58.010350 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:55:58.010311 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:56:28.010346 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:56:28.010313 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:56:48.673369 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:56:48.673331 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:56:58.009830 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:56:58.009782 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:57:14.077693 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:57:14.077662 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:57:14.578530 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:57:14.578494 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:57:14.721415 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:57:14.721378 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:57:18.559777 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:57:18.559742 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:57:18.715765 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:57:18.715721 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:57:20.149528 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:57:20.149492 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:57:20.308856 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:57:20.308814 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:57:20.479494 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:57:20.479419 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:57:22.407225 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:57:22.407134 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:57:24.267838 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:57:24.267791 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:57:26.149392 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:57:26.149353 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:57:28.009170 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:57:28.009136 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:57:28.042873 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:57:28.042843 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:57:29.897739 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:57:29.897705 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:57:32.066535 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:57:32.066504 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:57:33.953652 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:57:33.953601 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:57:35.928685 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:57:35.928622 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:57:37.846278 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:57:37.846242 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:57:38.362014 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:57:38.361980 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:57:39.039984 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:57:39.039929 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:57:39.196419 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:57:39.196384 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:57:39.371450 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:57:39.371416 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:57:39.835949 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:57:39.835903 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:57:40.860525 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:57:40.860487 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:57:41.693491 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:57:41.693452 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:57:43.626624 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:57:43.626584 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:57:45.549527 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:57:45.549490 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:57:47.468906 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:57:47.468865 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:57:48.784430 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:57:48.784392 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:57:49.283699 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:57:49.283661 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:57:51.139369 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:57:51.139327 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:57:53.069447 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:57:53.069411 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:57:54.989714 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:57:54.989682 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:57:56.839628 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:57:56.839592 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:57:58.011742 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:57:58.011702 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:57:58.878568 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:57:58.878533 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:58:00.729024 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:58:00.728984 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:58:02.576692 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:58:02.576624 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:58:04.572430 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:58:04.572393 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:58:06.444612 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:58:06.444580 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:58:08.296508 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:58:08.296470 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:58:10.190756 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:58:10.190724 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:58:12.064025 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:58:12.063963 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:58:14.067038 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:58:14.067003 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:58:15.928165 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:58:15.928132 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:58:17.972425 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:58:17.972332 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:58:19.800099 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:58:19.800060 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:58:20.165369 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:58:20.165335 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:58:21.648586 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:58:21.648552 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:58:23.581963 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:58:23.581931 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:58:25.411781 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:58:25.411747 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:58:25.556125 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:58:25.556093 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:58:27.672006 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:58:27.671971 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:58:28.011156 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:58:28.011074 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:58:29.507348 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:58:29.507305 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:58:30.454770 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:58:30.454723 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:58:31.353455 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:58:31.353409 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:58:33.375612 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:58:33.375499 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:58:35.223662 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:58:35.223618 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:58:36.206279 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:58:36.206246 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:58:36.352188 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:58:36.352136 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:58:36.873806 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:58:36.873765 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:58:37.435841 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:58:37.435803 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:58:37.599671 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:58:37.599605 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:58:38.115779 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:58:38.115742 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:58:44.644872 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:58:44.644834 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:58:44.790329 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:58:44.790296 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:58:45.034417 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:58:45.034321 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:58:48.898871 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:58:48.898836 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:58:51.055775 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:58:51.055740 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ntlqq_dc967875-1546-40bb-8376-fde09416c204/console-operator/2.log" Oct 30 14:58:51.060961 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:58:51.060936 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhxjf_aca47fe8-7068-4743-b6ab-283ee0874f66/ovn-acl-logging/0.log" Oct 30 14:58:51.077100 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:58:51.077077 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ntlqq_dc967875-1546-40bb-8376-fde09416c204/console-operator/2.log" Oct 30 14:58:51.081414 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:58:51.081396 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhxjf_aca47fe8-7068-4743-b6ab-283ee0874f66/ovn-acl-logging/0.log" Oct 30 14:58:58.010473 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:58:58.010437 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:59:04.289126 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:59:04.289087 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-p5rrv"] Oct 30 14:59:04.289702 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:59:04.289290 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-p5rrv" podUID="a3210e1a-fd2b-4aea-9e2a-acecc54a7144" containerName="manager" containerID="cri-o://475a35cfd1e1a9906890478c8110763ceb6ba89035ebacba244561747235c3ec" gracePeriod=10 Oct 30 14:59:04.540498 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:59:04.540447 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-p5rrv" Oct 30 14:59:04.675454 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:59:04.675424 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3210e1a-fd2b-4aea-9e2a-acecc54a7144-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a3210e1a-fd2b-4aea-9e2a-acecc54a7144-extensions-socket-volume\") pod \"a3210e1a-fd2b-4aea-9e2a-acecc54a7144\" (UID: \"a3210e1a-fd2b-4aea-9e2a-acecc54a7144\") " Oct 30 14:59:04.675676 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:59:04.675487 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3210e1a-fd2b-4aea-9e2a-acecc54a7144-kube-api-access-zrltl\" (UniqueName: \"kubernetes.io/projected/a3210e1a-fd2b-4aea-9e2a-acecc54a7144-kube-api-access-zrltl\") pod \"a3210e1a-fd2b-4aea-9e2a-acecc54a7144\" (UID: \"a3210e1a-fd2b-4aea-9e2a-acecc54a7144\") " Oct 30 14:59:04.675882 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:59:04.675855 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3210e1a-fd2b-4aea-9e2a-acecc54a7144-extensions-socket-volume" pod "a3210e1a-fd2b-4aea-9e2a-acecc54a7144" (UID: "a3210e1a-fd2b-4aea-9e2a-acecc54a7144"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:59:04.677591 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:59:04.677565 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3210e1a-fd2b-4aea-9e2a-acecc54a7144-kube-api-access-zrltl" pod "a3210e1a-fd2b-4aea-9e2a-acecc54a7144" (UID: "a3210e1a-fd2b-4aea-9e2a-acecc54a7144"). InnerVolumeSpecName "kube-api-access-zrltl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:59:04.776522 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:59:04.776483 4982 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a3210e1a-fd2b-4aea-9e2a-acecc54a7144-extensions-socket-volume\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:59:04.776522 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:59:04.776519 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrltl\" (UniqueName: \"kubernetes.io/projected/a3210e1a-fd2b-4aea-9e2a-acecc54a7144-kube-api-access-zrltl\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 14:59:05.390781 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:59:05.390746 4982 generic.go:356] "Generic (PLEG): container finished" podID="a3210e1a-fd2b-4aea-9e2a-acecc54a7144" containerID="475a35cfd1e1a9906890478c8110763ceb6ba89035ebacba244561747235c3ec" exitCode=0 Oct 30 14:59:05.391173 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:59:05.390786 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-p5rrv" Oct 30 14:59:05.391173 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:59:05.390829 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-p5rrv" event={"ID":"a3210e1a-fd2b-4aea-9e2a-acecc54a7144","Type":"ContainerDied","Data":"475a35cfd1e1a9906890478c8110763ceb6ba89035ebacba244561747235c3ec"} Oct 30 14:59:05.391173 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:59:05.390876 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-p5rrv" event={"ID":"a3210e1a-fd2b-4aea-9e2a-acecc54a7144","Type":"ContainerDied","Data":"4c399ca1c32d19b4a249b976af944e419ea267c5e93b679d144e7a6f27320e49"} Oct 30 14:59:05.391173 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:59:05.390896 4982 scope.go:122] "RemoveContainer" containerID="475a35cfd1e1a9906890478c8110763ceb6ba89035ebacba244561747235c3ec" Oct 30 14:59:05.400478 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:59:05.400460 4982 scope.go:122] "RemoveContainer" containerID="475a35cfd1e1a9906890478c8110763ceb6ba89035ebacba244561747235c3ec" Oct 30 14:59:05.400738 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:59:05.400719 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"475a35cfd1e1a9906890478c8110763ceb6ba89035ebacba244561747235c3ec\": container with ID starting with 475a35cfd1e1a9906890478c8110763ceb6ba89035ebacba244561747235c3ec not found: ID does not exist" containerID="475a35cfd1e1a9906890478c8110763ceb6ba89035ebacba244561747235c3ec" Oct 30 14:59:05.400797 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:59:05.400746 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"475a35cfd1e1a9906890478c8110763ceb6ba89035ebacba244561747235c3ec"} err="failed to get container status \"475a35cfd1e1a9906890478c8110763ceb6ba89035ebacba244561747235c3ec\": rpc error: code = NotFound desc = could not find container \"475a35cfd1e1a9906890478c8110763ceb6ba89035ebacba244561747235c3ec\": container with ID starting with 475a35cfd1e1a9906890478c8110763ceb6ba89035ebacba244561747235c3ec not found: ID does not exist" Oct 30 14:59:05.461889 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:59:05.461866 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-p5rrv"] Oct 30 14:59:05.475150 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:59:05.475112 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-p5rrv"] Oct 30 14:59:06.397125 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:59:06.397092 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3210e1a-fd2b-4aea-9e2a-acecc54a7144" path="/var/lib/kubelet/pods/a3210e1a-fd2b-4aea-9e2a-acecc54a7144/volumes" Oct 30 14:59:07.573622 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:59:07.573589 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:08.449265 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:59:08.449229 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:14.629818 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:59:14.629783 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:28.009746 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:59:28.009708 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 14:59:49.092847 ip-10-0-140-105 kubenswrapper[4982]: I1030 14:59:49.092809 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:58.010391 ip-10-0-140-105 kubenswrapper[4982]: E1030 14:59:58.010355 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 15:00:28.072026 ip-10-0-140-105 kubenswrapper[4982]: E1030 15:00:28.071987 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 15:00:28.445712 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:00:28.445335 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:00:30.250756 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:00:30.250721 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:00:44.754780 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:00:44.754747 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:00:44.901200 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:00:44.901171 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:00:49.234198 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:00:49.234161 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:00:53.648355 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:00:53.648323 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:00:54.691158 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:00:54.691124 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:00:58.011059 ip-10-0-140-105 kubenswrapper[4982]: E1030 15:00:58.011025 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 15:01:13.106145 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:01:13.106108 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:01:13.637143 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:01:13.637105 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:01:13.956704 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:01:13.956611 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:01:27.087194 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:01:27.087157 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:01:27.413175 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:01:27.413095 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:01:28.010279 ip-10-0-140-105 kubenswrapper[4982]: E1030 15:01:28.010239 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 15:01:28.776909 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:01:28.776875 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:01:28.935898 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:01:28.935837 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:01:29.115928 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:01:29.115887 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:01:30.874035 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:01:30.873998 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:01:32.881209 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:01:32.881175 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:01:34.845896 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:01:34.845843 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:01:36.867432 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:01:36.867398 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:01:38.827056 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:01:38.827017 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:01:40.793729 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:01:40.793670 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:01:41.200086 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:01:41.200042 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:01:41.897740 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:01:41.897689 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:01:42.568243 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:01:42.568201 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:01:42.714742 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:01:42.714710 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:01:42.874169 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:01:42.874090 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:01:44.638319 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:01:44.638283 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-2ef0c7/simulator-single-main-2ef0c7-kserve-7dc78468b7-tzmr5"] Oct 30 15:01:44.638832 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:01:44.638683 4982 cpu_manager.go:414] "containerMap: removing container" podUID="a3210e1a-fd2b-4aea-9e2a-acecc54a7144" containerName="manager" Oct 30 15:01:44.638832 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:01:44.638698 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3210e1a-fd2b-4aea-9e2a-acecc54a7144" containerName="manager" Oct 30 15:01:44.638832 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:01:44.638775 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3210e1a-fd2b-4aea-9e2a-acecc54a7144" containerName="manager" Oct 30 15:01:44.641855 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:01:44.641832 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-2ef0c7/simulator-single-main-2ef0c7-kserve-7dc78468b7-tzmr5" Oct 30 15:01:44.660904 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:01:44.660879 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-2ef0c7\"/\"openshift-service-ca.crt\"" Oct 30 15:01:44.673326 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:01:44.673294 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-2ef0c7\"/\"kube-root-ca.crt\"" Oct 30 15:01:44.674151 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:01:44.674127 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-2ef0c7\"/\"default-dockercfg-b8cgm\"" Oct 30 15:01:44.674292 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:01:44.674197 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-2ef0c7\"/\"simulator-single-main-2ef0c7-kserve-self-signed-certs\"" Oct 30 15:01:44.674872 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:01:44.674852 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-2ef0c7/simulator-single-main-2ef0c7-kserve-7dc78468b7-tzmr5"] Oct 30 15:01:44.681800 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:01:44.681769 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54857d2d-ebf3-414d-9ec2-f48894955994-model-cache\") pod \"simulator-single-main-2ef0c7-kserve-7dc78468b7-tzmr5\" (UID: \"54857d2d-ebf3-414d-9ec2-f48894955994\") " pod="e2e-models-e2e-worker-single-main-2ef0c7/simulator-single-main-2ef0c7-kserve-7dc78468b7-tzmr5" Oct 30 15:01:44.681921 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:01:44.681814 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54857d2d-ebf3-414d-9ec2-f48894955994-tls-certs\") pod \"simulator-single-main-2ef0c7-kserve-7dc78468b7-tzmr5\" (UID: \"54857d2d-ebf3-414d-9ec2-f48894955994\") " pod="e2e-models-e2e-worker-single-main-2ef0c7/simulator-single-main-2ef0c7-kserve-7dc78468b7-tzmr5" Oct 30 15:01:44.681921 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:01:44.681884 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54857d2d-ebf3-414d-9ec2-f48894955994-home\") pod \"simulator-single-main-2ef0c7-kserve-7dc78468b7-tzmr5\" (UID: \"54857d2d-ebf3-414d-9ec2-f48894955994\") " pod="e2e-models-e2e-worker-single-main-2ef0c7/simulator-single-main-2ef0c7-kserve-7dc78468b7-tzmr5" Oct 30 15:01:44.682052 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:01:44.681919 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54857d2d-ebf3-414d-9ec2-f48894955994-tmp-dir\") pod \"simulator-single-main-2ef0c7-kserve-7dc78468b7-tzmr5\" (UID: \"54857d2d-ebf3-414d-9ec2-f48894955994\") " pod="e2e-models-e2e-worker-single-main-2ef0c7/simulator-single-main-2ef0c7-kserve-7dc78468b7-tzmr5" Oct 30 15:01:44.682052 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:01:44.681954 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnxn5\" (UniqueName: \"kubernetes.io/projected/54857d2d-ebf3-414d-9ec2-f48894955994-kube-api-access-rnxn5\") pod \"simulator-single-main-2ef0c7-kserve-7dc78468b7-tzmr5\" (UID: \"54857d2d-ebf3-414d-9ec2-f48894955994\") " pod="e2e-models-e2e-worker-single-main-2ef0c7/simulator-single-main-2ef0c7-kserve-7dc78468b7-tzmr5" Oct 30 15:01:44.682175 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:01:44.682067 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54857d2d-ebf3-414d-9ec2-f48894955994-dshm\") pod \"simulator-single-main-2ef0c7-kserve-7dc78468b7-tzmr5\" (UID: \"54857d2d-ebf3-414d-9ec2-f48894955994\") " pod="e2e-models-e2e-worker-single-main-2ef0c7/simulator-single-main-2ef0c7-kserve-7dc78468b7-tzmr5" Oct 30 15:01:44.783286 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:01:44.783262 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54857d2d-ebf3-414d-9ec2-f48894955994-model-cache\") pod \"simulator-single-main-2ef0c7-kserve-7dc78468b7-tzmr5\" (UID: \"54857d2d-ebf3-414d-9ec2-f48894955994\") " pod="e2e-models-e2e-worker-single-main-2ef0c7/simulator-single-main-2ef0c7-kserve-7dc78468b7-tzmr5" Oct 30 15:01:44.783413 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:01:44.783291 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54857d2d-ebf3-414d-9ec2-f48894955994-tls-certs\") pod \"simulator-single-main-2ef0c7-kserve-7dc78468b7-tzmr5\" (UID: \"54857d2d-ebf3-414d-9ec2-f48894955994\") " pod="e2e-models-e2e-worker-single-main-2ef0c7/simulator-single-main-2ef0c7-kserve-7dc78468b7-tzmr5" Oct 30 15:01:44.783413 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:01:44.783311 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54857d2d-ebf3-414d-9ec2-f48894955994-home\") pod \"simulator-single-main-2ef0c7-kserve-7dc78468b7-tzmr5\" (UID: \"54857d2d-ebf3-414d-9ec2-f48894955994\") " pod="e2e-models-e2e-worker-single-main-2ef0c7/simulator-single-main-2ef0c7-kserve-7dc78468b7-tzmr5" Oct 30 15:01:44.783413 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:01:44.783327 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54857d2d-ebf3-414d-9ec2-f48894955994-tmp-dir\") pod \"simulator-single-main-2ef0c7-kserve-7dc78468b7-tzmr5\" (UID: \"54857d2d-ebf3-414d-9ec2-f48894955994\") " pod="e2e-models-e2e-worker-single-main-2ef0c7/simulator-single-main-2ef0c7-kserve-7dc78468b7-tzmr5" Oct 30 15:01:44.783593 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:01:44.783465 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnxn5\" (UniqueName: \"kubernetes.io/projected/54857d2d-ebf3-414d-9ec2-f48894955994-kube-api-access-rnxn5\") pod \"simulator-single-main-2ef0c7-kserve-7dc78468b7-tzmr5\" (UID: \"54857d2d-ebf3-414d-9ec2-f48894955994\") " pod="e2e-models-e2e-worker-single-main-2ef0c7/simulator-single-main-2ef0c7-kserve-7dc78468b7-tzmr5" Oct 30 15:01:44.783675 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:01:44.783601 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54857d2d-ebf3-414d-9ec2-f48894955994-dshm\") pod \"simulator-single-main-2ef0c7-kserve-7dc78468b7-tzmr5\" (UID: \"54857d2d-ebf3-414d-9ec2-f48894955994\") " pod="e2e-models-e2e-worker-single-main-2ef0c7/simulator-single-main-2ef0c7-kserve-7dc78468b7-tzmr5" Oct 30 15:01:44.783738 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:01:44.783692 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54857d2d-ebf3-414d-9ec2-f48894955994-model-cache\") pod \"simulator-single-main-2ef0c7-kserve-7dc78468b7-tzmr5\" (UID: \"54857d2d-ebf3-414d-9ec2-f48894955994\") " pod="e2e-models-e2e-worker-single-main-2ef0c7/simulator-single-main-2ef0c7-kserve-7dc78468b7-tzmr5" Oct 30 15:01:44.783788 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:01:44.783733 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54857d2d-ebf3-414d-9ec2-f48894955994-tmp-dir\") pod \"simulator-single-main-2ef0c7-kserve-7dc78468b7-tzmr5\" (UID: \"54857d2d-ebf3-414d-9ec2-f48894955994\") " pod="e2e-models-e2e-worker-single-main-2ef0c7/simulator-single-main-2ef0c7-kserve-7dc78468b7-tzmr5" Oct 30 15:01:44.783827 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:01:44.783785 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54857d2d-ebf3-414d-9ec2-f48894955994-home\") pod \"simulator-single-main-2ef0c7-kserve-7dc78468b7-tzmr5\" (UID: \"54857d2d-ebf3-414d-9ec2-f48894955994\") " pod="e2e-models-e2e-worker-single-main-2ef0c7/simulator-single-main-2ef0c7-kserve-7dc78468b7-tzmr5" Oct 30 15:01:44.785811 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:01:44.785791 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54857d2d-ebf3-414d-9ec2-f48894955994-dshm\") pod \"simulator-single-main-2ef0c7-kserve-7dc78468b7-tzmr5\" (UID: \"54857d2d-ebf3-414d-9ec2-f48894955994\") " pod="e2e-models-e2e-worker-single-main-2ef0c7/simulator-single-main-2ef0c7-kserve-7dc78468b7-tzmr5" Oct 30 15:01:44.785900 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:01:44.785850 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54857d2d-ebf3-414d-9ec2-f48894955994-tls-certs\") pod \"simulator-single-main-2ef0c7-kserve-7dc78468b7-tzmr5\" (UID: \"54857d2d-ebf3-414d-9ec2-f48894955994\") " pod="e2e-models-e2e-worker-single-main-2ef0c7/simulator-single-main-2ef0c7-kserve-7dc78468b7-tzmr5" Oct 30 15:01:44.786077 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:01:44.786057 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:01:44.809099 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:01:44.809074 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnxn5\" (UniqueName: \"kubernetes.io/projected/54857d2d-ebf3-414d-9ec2-f48894955994-kube-api-access-rnxn5\") pod \"simulator-single-main-2ef0c7-kserve-7dc78468b7-tzmr5\" (UID: \"54857d2d-ebf3-414d-9ec2-f48894955994\") " pod="e2e-models-e2e-worker-single-main-2ef0c7/simulator-single-main-2ef0c7-kserve-7dc78468b7-tzmr5" Oct 30 15:01:44.949436 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:01:44.949360 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:01:44.954320 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:01:44.954295 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-2ef0c7/simulator-single-main-2ef0c7-kserve-7dc78468b7-tzmr5" Oct 30 15:01:45.272079 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:01:45.272051 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-2ef0c7/simulator-single-main-2ef0c7-kserve-7dc78468b7-tzmr5"] Oct 30 15:01:45.273139 ip-10-0-140-105 kubenswrapper[4982]: W1030 15:01:45.273112 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod54857d2d_ebf3_414d_9ec2_f48894955994.slice/crio-8b5077d5e655e090df7646925a397a4e9f7d20886baca2838d37338c0ee34404 WatchSource:0}: Error finding container 8b5077d5e655e090df7646925a397a4e9f7d20886baca2838d37338c0ee34404: Status 404 returned error can't find the container with id 8b5077d5e655e090df7646925a397a4e9f7d20886baca2838d37338c0ee34404 Oct 30 15:01:46.112916 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:01:46.112880 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-2ef0c7/simulator-single-main-2ef0c7-kserve-7dc78468b7-tzmr5" event={"ID":"54857d2d-ebf3-414d-9ec2-f48894955994","Type":"ContainerStarted","Data":"15333f4b36a1729641752302284108a02f64cf00f7989338658ee610850dc258"} Oct 30 15:01:46.113349 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:01:46.112930 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-2ef0c7/simulator-single-main-2ef0c7-kserve-7dc78468b7-tzmr5" event={"ID":"54857d2d-ebf3-414d-9ec2-f48894955994","Type":"ContainerStarted","Data":"8b5077d5e655e090df7646925a397a4e9f7d20886baca2838d37338c0ee34404"} Oct 30 15:01:46.161714 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:01:46.161657 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-2ef0c7/simulator-single-main-2ef0c7-kserve-7dc78468b7-tzmr5" podStartSLOduration=2.161628983 podStartE2EDuration="2.161628983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:01:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:01:46.159838605 +0000 UTC m=+4375.903972642" watchObservedRunningTime="2026-10-30 15:01:46.161628983 +0000 UTC m=+4375.905763027" Oct 30 15:01:46.757087 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:01:46.757049 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:01:46.954705 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:01:46.954673 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-2ef0c7/simulator-single-main-2ef0c7-kserve-7dc78468b7-tzmr5" Oct 30 15:01:48.864071 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:01:48.864039 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:01:49.276773 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:01:49.276697 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:01:50.683773 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:01:50.683740 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:01:50.831319 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:01:50.831285 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:01:52.579510 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:01:52.579474 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:01:54.439451 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:01:54.439391 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:01:54.955212 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:01:54.955178 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-2ef0c7/simulator-single-main-2ef0c7-kserve-7dc78468b7-tzmr5" Oct 30 15:01:54.966911 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:01:54.966887 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-2ef0c7/simulator-single-main-2ef0c7-kserve-7dc78468b7-tzmr5" Oct 30 15:01:55.175197 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:01:55.175165 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-2ef0c7/simulator-single-main-2ef0c7-kserve-7dc78468b7-tzmr5" Oct 30 15:01:56.413290 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:01:56.413258 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:01:58.016685 ip-10-0-140-105 kubenswrapper[4982]: E1030 15:01:58.016622 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 15:01:58.257975 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:01:58.257945 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:00.249208 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:00.249172 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:02.106627 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:02.106588 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:04.186030 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:04.185998 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:06.035556 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:06.035513 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:07.687940 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:07.687904 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:07.830891 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:07.830856 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:08.072969 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:08.072941 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:09.910286 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:09.910253 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:11.780126 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:11.780091 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:13.676287 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:13.676253 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:15.525858 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:15.525822 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:17.438172 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:17.438135 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:19.291992 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:19.291959 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:21.312998 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:21.312963 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:23.169063 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:23.169020 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:24.691453 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:24.691415 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:25.182755 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:25.182724 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:27.295499 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:27.295464 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:28.011973 ip-10-0-140-105 kubenswrapper[4982]: E1030 15:02:28.011937 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 15:02:29.113445 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:29.113406 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:29.618327 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:29.618297 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-2ef0c7/simulator-single-main-2ef0c7-kserve-7dc78468b7-tzmr5"] Oct 30 15:02:29.728310 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:29.728272 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:30.967509 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:30.967478 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:32.872279 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:32.872249 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:34.753336 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:34.753300 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:35.440632 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:35.440592 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:35.584042 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:35.584006 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:36.774806 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:36.774768 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:38.399376 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:38.399342 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:38.953324 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:38.953287 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:39.090829 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:39.090794 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:39.615332 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:39.615298 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:44.637853 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:44.637797 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-2ef0c7/simulator-single-main-2ef0c7-kserve-7dc78468b7-tzmr5" podUID="54857d2d-ebf3-414d-9ec2-f48894955994" containerName="main" containerID="cri-o://15333f4b36a1729641752302284108a02f64cf00f7989338658ee610850dc258" gracePeriod=45 Oct 30 15:02:44.884891 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:44.884866 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-2ef0c7/simulator-single-main-2ef0c7-kserve-7dc78468b7-tzmr5" Oct 30 15:02:44.976247 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:44.976165 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54857d2d-ebf3-414d-9ec2-f48894955994-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54857d2d-ebf3-414d-9ec2-f48894955994-model-cache\") pod \"54857d2d-ebf3-414d-9ec2-f48894955994\" (UID: \"54857d2d-ebf3-414d-9ec2-f48894955994\") " Oct 30 15:02:44.976247 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:44.976218 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54857d2d-ebf3-414d-9ec2-f48894955994-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54857d2d-ebf3-414d-9ec2-f48894955994-tmp-dir\") pod \"54857d2d-ebf3-414d-9ec2-f48894955994\" (UID: \"54857d2d-ebf3-414d-9ec2-f48894955994\") " Oct 30 15:02:44.976489 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:44.976340 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54857d2d-ebf3-414d-9ec2-f48894955994-kube-api-access-rnxn5\" (UniqueName: \"kubernetes.io/projected/54857d2d-ebf3-414d-9ec2-f48894955994-kube-api-access-rnxn5\") pod \"54857d2d-ebf3-414d-9ec2-f48894955994\" (UID: \"54857d2d-ebf3-414d-9ec2-f48894955994\") " Oct 30 15:02:44.976489 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:44.976372 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54857d2d-ebf3-414d-9ec2-f48894955994-home\" (UniqueName: \"kubernetes.io/empty-dir/54857d2d-ebf3-414d-9ec2-f48894955994-home\") pod \"54857d2d-ebf3-414d-9ec2-f48894955994\" (UID: \"54857d2d-ebf3-414d-9ec2-f48894955994\") " Oct 30 15:02:44.976489 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:44.976404 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54857d2d-ebf3-414d-9ec2-f48894955994-tls-certs\" (UniqueName: \"kubernetes.io/secret/54857d2d-ebf3-414d-9ec2-f48894955994-tls-certs\") pod \"54857d2d-ebf3-414d-9ec2-f48894955994\" (UID: \"54857d2d-ebf3-414d-9ec2-f48894955994\") " Oct 30 15:02:44.976489 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:44.976435 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54857d2d-ebf3-414d-9ec2-f48894955994-dshm\" (UniqueName: \"kubernetes.io/empty-dir/54857d2d-ebf3-414d-9ec2-f48894955994-dshm\") pod \"54857d2d-ebf3-414d-9ec2-f48894955994\" (UID: \"54857d2d-ebf3-414d-9ec2-f48894955994\") " Oct 30 15:02:44.976730 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:44.976490 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54857d2d-ebf3-414d-9ec2-f48894955994-tmp-dir" pod "54857d2d-ebf3-414d-9ec2-f48894955994" (UID: "54857d2d-ebf3-414d-9ec2-f48894955994"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:02:44.976730 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:44.976503 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54857d2d-ebf3-414d-9ec2-f48894955994-model-cache" pod "54857d2d-ebf3-414d-9ec2-f48894955994" (UID: "54857d2d-ebf3-414d-9ec2-f48894955994"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:02:44.976730 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:44.976711 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54857d2d-ebf3-414d-9ec2-f48894955994-home" pod "54857d2d-ebf3-414d-9ec2-f48894955994" (UID: "54857d2d-ebf3-414d-9ec2-f48894955994"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:02:44.976906 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:44.976779 4982 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54857d2d-ebf3-414d-9ec2-f48894955994-home\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 15:02:44.976906 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:44.976796 4982 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54857d2d-ebf3-414d-9ec2-f48894955994-model-cache\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 15:02:44.976906 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:44.976808 4982 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54857d2d-ebf3-414d-9ec2-f48894955994-tmp-dir\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 15:02:44.978742 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:44.978705 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54857d2d-ebf3-414d-9ec2-f48894955994-dshm" pod "54857d2d-ebf3-414d-9ec2-f48894955994" (UID: "54857d2d-ebf3-414d-9ec2-f48894955994"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:02:44.978742 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:44.978718 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54857d2d-ebf3-414d-9ec2-f48894955994-tls-certs" pod "54857d2d-ebf3-414d-9ec2-f48894955994" (UID: "54857d2d-ebf3-414d-9ec2-f48894955994"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:02:44.978892 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:44.978773 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54857d2d-ebf3-414d-9ec2-f48894955994-kube-api-access-rnxn5" pod "54857d2d-ebf3-414d-9ec2-f48894955994" (UID: "54857d2d-ebf3-414d-9ec2-f48894955994"). InnerVolumeSpecName "kube-api-access-rnxn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:02:45.078053 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:45.078021 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnxn5\" (UniqueName: \"kubernetes.io/projected/54857d2d-ebf3-414d-9ec2-f48894955994-kube-api-access-rnxn5\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 15:02:45.078053 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:45.078050 4982 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54857d2d-ebf3-414d-9ec2-f48894955994-tls-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 15:02:45.078053 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:45.078060 4982 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54857d2d-ebf3-414d-9ec2-f48894955994-dshm\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 15:02:45.383144 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:45.383117 4982 generic.go:356] "Generic (PLEG): container finished" podID="54857d2d-ebf3-414d-9ec2-f48894955994" containerID="15333f4b36a1729641752302284108a02f64cf00f7989338658ee610850dc258" exitCode=0 Oct 30 15:02:45.383299 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:45.383163 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-2ef0c7/simulator-single-main-2ef0c7-kserve-7dc78468b7-tzmr5" Oct 30 15:02:45.383299 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:45.383177 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-2ef0c7/simulator-single-main-2ef0c7-kserve-7dc78468b7-tzmr5" event={"ID":"54857d2d-ebf3-414d-9ec2-f48894955994","Type":"ContainerDied","Data":"15333f4b36a1729641752302284108a02f64cf00f7989338658ee610850dc258"} Oct 30 15:02:45.383299 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:45.383209 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-2ef0c7/simulator-single-main-2ef0c7-kserve-7dc78468b7-tzmr5" event={"ID":"54857d2d-ebf3-414d-9ec2-f48894955994","Type":"ContainerDied","Data":"8b5077d5e655e090df7646925a397a4e9f7d20886baca2838d37338c0ee34404"} Oct 30 15:02:45.383299 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:45.383226 4982 scope.go:122] "RemoveContainer" containerID="15333f4b36a1729641752302284108a02f64cf00f7989338658ee610850dc258" Oct 30 15:02:45.392703 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:45.392681 4982 scope.go:122] "RemoveContainer" containerID="15333f4b36a1729641752302284108a02f64cf00f7989338658ee610850dc258" Oct 30 15:02:45.392943 ip-10-0-140-105 kubenswrapper[4982]: E1030 15:02:45.392923 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15333f4b36a1729641752302284108a02f64cf00f7989338658ee610850dc258\": container with ID starting with 15333f4b36a1729641752302284108a02f64cf00f7989338658ee610850dc258 not found: ID does not exist" containerID="15333f4b36a1729641752302284108a02f64cf00f7989338658ee610850dc258" Oct 30 15:02:45.392999 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:45.392950 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15333f4b36a1729641752302284108a02f64cf00f7989338658ee610850dc258"} err="failed to get container status \"15333f4b36a1729641752302284108a02f64cf00f7989338658ee610850dc258\": rpc error: code = NotFound desc = could not find container \"15333f4b36a1729641752302284108a02f64cf00f7989338658ee610850dc258\": container with ID starting with 15333f4b36a1729641752302284108a02f64cf00f7989338658ee610850dc258 not found: ID does not exist" Oct 30 15:02:45.439532 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:45.439509 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-2ef0c7/simulator-single-main-2ef0c7-kserve-7dc78468b7-tzmr5"] Oct 30 15:02:45.452569 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:45.452548 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-2ef0c7/simulator-single-main-2ef0c7-kserve-7dc78468b7-tzmr5"] Oct 30 15:02:46.398123 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:46.398086 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54857d2d-ebf3-414d-9ec2-f48894955994" path="/var/lib/kubelet/pods/54857d2d-ebf3-414d-9ec2-f48894955994/volumes" Oct 30 15:02:46.447270 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:46.447244 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:46.611934 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:46.611903 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:46.921880 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:46.921843 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:49.420411 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:49.420380 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:50.898895 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:50.898857 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-7f7d1aa4-kserve-5758c45989-npr8j"] Oct 30 15:02:50.899436 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:50.899421 4982 cpu_manager.go:414] "containerMap: removing container" podUID="54857d2d-ebf3-414d-9ec2-f48894955994" containerName="main" Oct 30 15:02:50.899485 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:50.899439 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54857d2d-ebf3-414d-9ec2-f48894955994" containerName="main" Oct 30 15:02:50.899573 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:50.899561 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="54857d2d-ebf3-414d-9ec2-f48894955994" containerName="main" Oct 30 15:02:50.904211 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:50.904190 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-7f7d1aa4-kserve-5758c45989-npr8j" Oct 30 15:02:50.925043 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:50.925014 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-7f7d1aa4-kserve-5758c45989-npr8j"] Oct 30 15:02:50.936618 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:50.936571 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 15:02:50.936618 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:50.936571 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-7f7d1aa4-kserve-self-signed-certs\"" Oct 30 15:02:50.936618 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:50.936609 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 15:02:50.936835 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:50.936670 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-lmhz8\"" Oct 30 15:02:50.985375 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:50.985347 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:51.025924 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:51.025895 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3180f572-aca4-49ad-adb2-4ac152008f46-model-cache\") pod \"e2e-same-id-a-7f7d1aa4-kserve-5758c45989-npr8j\" (UID: \"3180f572-aca4-49ad-adb2-4ac152008f46\") " pod="llm/e2e-same-id-a-7f7d1aa4-kserve-5758c45989-npr8j" Oct 30 15:02:51.026045 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:51.025934 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xsrt\" (UniqueName: \"kubernetes.io/projected/3180f572-aca4-49ad-adb2-4ac152008f46-kube-api-access-4xsrt\") pod \"e2e-same-id-a-7f7d1aa4-kserve-5758c45989-npr8j\" (UID: \"3180f572-aca4-49ad-adb2-4ac152008f46\") " pod="llm/e2e-same-id-a-7f7d1aa4-kserve-5758c45989-npr8j" Oct 30 15:02:51.026045 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:51.025961 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3180f572-aca4-49ad-adb2-4ac152008f46-tmp-dir\") pod \"e2e-same-id-a-7f7d1aa4-kserve-5758c45989-npr8j\" (UID: \"3180f572-aca4-49ad-adb2-4ac152008f46\") " pod="llm/e2e-same-id-a-7f7d1aa4-kserve-5758c45989-npr8j" Oct 30 15:02:51.026141 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:51.026118 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3180f572-aca4-49ad-adb2-4ac152008f46-home\") pod \"e2e-same-id-a-7f7d1aa4-kserve-5758c45989-npr8j\" (UID: \"3180f572-aca4-49ad-adb2-4ac152008f46\") " pod="llm/e2e-same-id-a-7f7d1aa4-kserve-5758c45989-npr8j" Oct 30 15:02:51.026238 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:51.026218 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3180f572-aca4-49ad-adb2-4ac152008f46-dshm\") pod \"e2e-same-id-a-7f7d1aa4-kserve-5758c45989-npr8j\" (UID: \"3180f572-aca4-49ad-adb2-4ac152008f46\") " pod="llm/e2e-same-id-a-7f7d1aa4-kserve-5758c45989-npr8j" Oct 30 15:02:51.026301 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:51.026274 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3180f572-aca4-49ad-adb2-4ac152008f46-tls-certs\") pod \"e2e-same-id-a-7f7d1aa4-kserve-5758c45989-npr8j\" (UID: \"3180f572-aca4-49ad-adb2-4ac152008f46\") " pod="llm/e2e-same-id-a-7f7d1aa4-kserve-5758c45989-npr8j" Oct 30 15:02:51.127590 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:51.127553 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3180f572-aca4-49ad-adb2-4ac152008f46-dshm\") pod \"e2e-same-id-a-7f7d1aa4-kserve-5758c45989-npr8j\" (UID: \"3180f572-aca4-49ad-adb2-4ac152008f46\") " pod="llm/e2e-same-id-a-7f7d1aa4-kserve-5758c45989-npr8j" Oct 30 15:02:51.127810 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:51.127608 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3180f572-aca4-49ad-adb2-4ac152008f46-tls-certs\") pod \"e2e-same-id-a-7f7d1aa4-kserve-5758c45989-npr8j\" (UID: \"3180f572-aca4-49ad-adb2-4ac152008f46\") " pod="llm/e2e-same-id-a-7f7d1aa4-kserve-5758c45989-npr8j" Oct 30 15:02:51.127810 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:51.127636 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3180f572-aca4-49ad-adb2-4ac152008f46-model-cache\") pod \"e2e-same-id-a-7f7d1aa4-kserve-5758c45989-npr8j\" (UID: \"3180f572-aca4-49ad-adb2-4ac152008f46\") " pod="llm/e2e-same-id-a-7f7d1aa4-kserve-5758c45989-npr8j" Oct 30 15:02:51.127810 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:51.127699 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xsrt\" (UniqueName: \"kubernetes.io/projected/3180f572-aca4-49ad-adb2-4ac152008f46-kube-api-access-4xsrt\") pod \"e2e-same-id-a-7f7d1aa4-kserve-5758c45989-npr8j\" (UID: \"3180f572-aca4-49ad-adb2-4ac152008f46\") " pod="llm/e2e-same-id-a-7f7d1aa4-kserve-5758c45989-npr8j" Oct 30 15:02:51.127810 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:51.127740 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3180f572-aca4-49ad-adb2-4ac152008f46-tmp-dir\") pod \"e2e-same-id-a-7f7d1aa4-kserve-5758c45989-npr8j\" (UID: \"3180f572-aca4-49ad-adb2-4ac152008f46\") " pod="llm/e2e-same-id-a-7f7d1aa4-kserve-5758c45989-npr8j" Oct 30 15:02:51.127810 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:51.127802 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3180f572-aca4-49ad-adb2-4ac152008f46-home\") pod \"e2e-same-id-a-7f7d1aa4-kserve-5758c45989-npr8j\" (UID: \"3180f572-aca4-49ad-adb2-4ac152008f46\") " pod="llm/e2e-same-id-a-7f7d1aa4-kserve-5758c45989-npr8j" Oct 30 15:02:51.128108 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:51.128062 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3180f572-aca4-49ad-adb2-4ac152008f46-model-cache\") pod \"e2e-same-id-a-7f7d1aa4-kserve-5758c45989-npr8j\" (UID: \"3180f572-aca4-49ad-adb2-4ac152008f46\") " pod="llm/e2e-same-id-a-7f7d1aa4-kserve-5758c45989-npr8j" Oct 30 15:02:51.128174 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:51.128130 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3180f572-aca4-49ad-adb2-4ac152008f46-home\") pod \"e2e-same-id-a-7f7d1aa4-kserve-5758c45989-npr8j\" (UID: \"3180f572-aca4-49ad-adb2-4ac152008f46\") " pod="llm/e2e-same-id-a-7f7d1aa4-kserve-5758c45989-npr8j" Oct 30 15:02:51.128174 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:51.128159 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3180f572-aca4-49ad-adb2-4ac152008f46-tmp-dir\") pod \"e2e-same-id-a-7f7d1aa4-kserve-5758c45989-npr8j\" (UID: \"3180f572-aca4-49ad-adb2-4ac152008f46\") " pod="llm/e2e-same-id-a-7f7d1aa4-kserve-5758c45989-npr8j" Oct 30 15:02:51.129794 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:51.129772 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3180f572-aca4-49ad-adb2-4ac152008f46-dshm\") pod \"e2e-same-id-a-7f7d1aa4-kserve-5758c45989-npr8j\" (UID: \"3180f572-aca4-49ad-adb2-4ac152008f46\") " pod="llm/e2e-same-id-a-7f7d1aa4-kserve-5758c45989-npr8j" Oct 30 15:02:51.130133 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:51.130113 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3180f572-aca4-49ad-adb2-4ac152008f46-tls-certs\") pod \"e2e-same-id-a-7f7d1aa4-kserve-5758c45989-npr8j\" (UID: \"3180f572-aca4-49ad-adb2-4ac152008f46\") " pod="llm/e2e-same-id-a-7f7d1aa4-kserve-5758c45989-npr8j" Oct 30 15:02:51.152979 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:51.152926 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xsrt\" (UniqueName: \"kubernetes.io/projected/3180f572-aca4-49ad-adb2-4ac152008f46-kube-api-access-4xsrt\") pod \"e2e-same-id-a-7f7d1aa4-kserve-5758c45989-npr8j\" (UID: \"3180f572-aca4-49ad-adb2-4ac152008f46\") " pod="llm/e2e-same-id-a-7f7d1aa4-kserve-5758c45989-npr8j" Oct 30 15:02:51.215245 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:51.215225 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-7f7d1aa4-kserve-5758c45989-npr8j" Oct 30 15:02:51.514447 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:51.514417 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-7f7d1aa4-kserve-5758c45989-npr8j"] Oct 30 15:02:51.515607 ip-10-0-140-105 kubenswrapper[4982]: W1030 15:02:51.515579 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3180f572_aca4_49ad_adb2_4ac152008f46.slice/crio-18d2e9b658c9f38c2eecbedd6761e803776ff572c6e541d07d283c415723a7b1 WatchSource:0}: Error finding container 18d2e9b658c9f38c2eecbedd6761e803776ff572c6e541d07d283c415723a7b1: Status 404 returned error can't find the container with id 18d2e9b658c9f38c2eecbedd6761e803776ff572c6e541d07d283c415723a7b1 Oct 30 15:02:52.417573 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:52.417538 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-7f7d1aa4-kserve-5758c45989-npr8j" event={"ID":"3180f572-aca4-49ad-adb2-4ac152008f46","Type":"ContainerStarted","Data":"ffe9a99fb7e99dd44418d4c989453aed6ca646067e24fa98ff11eade91253aa1"} Oct 30 15:02:52.417973 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:52.417579 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-7f7d1aa4-kserve-5758c45989-npr8j" event={"ID":"3180f572-aca4-49ad-adb2-4ac152008f46","Type":"ContainerStarted","Data":"18d2e9b658c9f38c2eecbedd6761e803776ff572c6e541d07d283c415723a7b1"} Oct 30 15:02:52.462022 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:52.461980 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-7f7d1aa4-kserve-5758c45989-npr8j" podStartSLOduration=2.461968937 podStartE2EDuration="2.461968937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:02:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:02:52.461735056 +0000 UTC m=+4442.205869096" watchObservedRunningTime="2026-10-30 15:02:52.461968937 +0000 UTC m=+4442.206102995" Oct 30 15:02:53.215583 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:02:53.215556 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-7f7d1aa4-kserve-5758c45989-npr8j" Oct 30 15:02:58.009865 ip-10-0-140-105 kubenswrapper[4982]: E1030 15:02:58.009828 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 15:03:01.216257 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:03:01.216222 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-7f7d1aa4-kserve-5758c45989-npr8j" Oct 30 15:03:01.228006 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:03:01.227977 4982 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-7f7d1aa4-kserve-5758c45989-npr8j" Oct 30 15:03:01.483091 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:03:01.483018 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-7f7d1aa4-kserve-5758c45989-npr8j" Oct 30 15:03:06.584601 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:03:06.584543 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:03:08.108324 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:03:08.108272 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:03:14.586677 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:03:14.586625 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:03:18.333962 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:03:18.333922 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:03:22.134808 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:03:22.134731 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:03:23.132859 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:03:23.132820 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:03:28.010393 ip-10-0-140-105 kubenswrapper[4982]: E1030 15:03:28.010356 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 15:03:49.627607 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:03:49.627567 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:03:51.094727 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:03:51.094691 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ntlqq_dc967875-1546-40bb-8376-fde09416c204/console-operator/2.log" Oct 30 15:03:51.103805 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:03:51.103775 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhxjf_aca47fe8-7068-4743-b6ab-283ee0874f66/ovn-acl-logging/0.log" Oct 30 15:03:51.109319 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:03:51.109300 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ntlqq_dc967875-1546-40bb-8376-fde09416c204/console-operator/2.log" Oct 30 15:03:51.113672 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:03:51.113635 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhxjf_aca47fe8-7068-4743-b6ab-283ee0874f66/ovn-acl-logging/0.log" Oct 30 15:03:56.271598 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:03:56.271561 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-7f7d1aa4-kserve-5758c45989-npr8j"] Oct 30 15:03:56.355127 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:03:56.355087 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:03:58.009360 ip-10-0-140-105 kubenswrapper[4982]: E1030 15:03:58.009325 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 15:04:00.664589 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:04:00.664552 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:04:11.291804 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:04:11.291768 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-7f7d1aa4-kserve-5758c45989-npr8j" podUID="3180f572-aca4-49ad-adb2-4ac152008f46" containerName="main" containerID="cri-o://ffe9a99fb7e99dd44418d4c989453aed6ca646067e24fa98ff11eade91253aa1" gracePeriod=45 Oct 30 15:04:11.541360 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:04:11.541334 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-7f7d1aa4-kserve-5758c45989-npr8j" Oct 30 15:04:11.642274 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:04:11.642246 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3180f572-aca4-49ad-adb2-4ac152008f46-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3180f572-aca4-49ad-adb2-4ac152008f46-tmp-dir\") pod \"3180f572-aca4-49ad-adb2-4ac152008f46\" (UID: \"3180f572-aca4-49ad-adb2-4ac152008f46\") " Oct 30 15:04:11.642410 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:04:11.642300 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3180f572-aca4-49ad-adb2-4ac152008f46-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3180f572-aca4-49ad-adb2-4ac152008f46-model-cache\") pod \"3180f572-aca4-49ad-adb2-4ac152008f46\" (UID: \"3180f572-aca4-49ad-adb2-4ac152008f46\") " Oct 30 15:04:11.642410 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:04:11.642357 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3180f572-aca4-49ad-adb2-4ac152008f46-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3180f572-aca4-49ad-adb2-4ac152008f46-dshm\") pod \"3180f572-aca4-49ad-adb2-4ac152008f46\" (UID: \"3180f572-aca4-49ad-adb2-4ac152008f46\") " Oct 30 15:04:11.642410 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:04:11.642378 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3180f572-aca4-49ad-adb2-4ac152008f46-home\" (UniqueName: \"kubernetes.io/empty-dir/3180f572-aca4-49ad-adb2-4ac152008f46-home\") pod \"3180f572-aca4-49ad-adb2-4ac152008f46\" (UID: \"3180f572-aca4-49ad-adb2-4ac152008f46\") " Oct 30 15:04:11.642583 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:04:11.642410 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3180f572-aca4-49ad-adb2-4ac152008f46-tls-certs\" (UniqueName: \"kubernetes.io/secret/3180f572-aca4-49ad-adb2-4ac152008f46-tls-certs\") pod \"3180f572-aca4-49ad-adb2-4ac152008f46\" (UID: \"3180f572-aca4-49ad-adb2-4ac152008f46\") " Oct 30 15:04:11.642583 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:04:11.642442 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3180f572-aca4-49ad-adb2-4ac152008f46-kube-api-access-4xsrt\" (UniqueName: \"kubernetes.io/projected/3180f572-aca4-49ad-adb2-4ac152008f46-kube-api-access-4xsrt\") pod \"3180f572-aca4-49ad-adb2-4ac152008f46\" (UID: \"3180f572-aca4-49ad-adb2-4ac152008f46\") " Oct 30 15:04:11.642583 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:04:11.642489 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3180f572-aca4-49ad-adb2-4ac152008f46-tmp-dir" pod "3180f572-aca4-49ad-adb2-4ac152008f46" (UID: "3180f572-aca4-49ad-adb2-4ac152008f46"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:04:11.642583 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:04:11.642540 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3180f572-aca4-49ad-adb2-4ac152008f46-model-cache" pod "3180f572-aca4-49ad-adb2-4ac152008f46" (UID: "3180f572-aca4-49ad-adb2-4ac152008f46"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:04:11.642821 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:04:11.642709 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3180f572-aca4-49ad-adb2-4ac152008f46-home" pod "3180f572-aca4-49ad-adb2-4ac152008f46" (UID: "3180f572-aca4-49ad-adb2-4ac152008f46"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:04:11.642821 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:04:11.642774 4982 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3180f572-aca4-49ad-adb2-4ac152008f46-tmp-dir\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 15:04:11.642821 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:04:11.642802 4982 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3180f572-aca4-49ad-adb2-4ac152008f46-model-cache\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 15:04:11.644579 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:04:11.644561 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3180f572-aca4-49ad-adb2-4ac152008f46-dshm" pod "3180f572-aca4-49ad-adb2-4ac152008f46" (UID: "3180f572-aca4-49ad-adb2-4ac152008f46"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:04:11.644684 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:04:11.644664 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3180f572-aca4-49ad-adb2-4ac152008f46-kube-api-access-4xsrt" pod "3180f572-aca4-49ad-adb2-4ac152008f46" (UID: "3180f572-aca4-49ad-adb2-4ac152008f46"). InnerVolumeSpecName "kube-api-access-4xsrt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:04:11.644761 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:04:11.644738 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3180f572-aca4-49ad-adb2-4ac152008f46-tls-certs" pod "3180f572-aca4-49ad-adb2-4ac152008f46" (UID: "3180f572-aca4-49ad-adb2-4ac152008f46"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:04:11.743461 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:04:11.743440 4982 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3180f572-aca4-49ad-adb2-4ac152008f46-dshm\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 15:04:11.743461 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:04:11.743461 4982 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3180f572-aca4-49ad-adb2-4ac152008f46-home\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 15:04:11.743601 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:04:11.743470 4982 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3180f572-aca4-49ad-adb2-4ac152008f46-tls-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 15:04:11.743601 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:04:11.743479 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xsrt\" (UniqueName: \"kubernetes.io/projected/3180f572-aca4-49ad-adb2-4ac152008f46-kube-api-access-4xsrt\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 15:04:11.799251 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:04:11.799221 4982 generic.go:356] "Generic (PLEG): container finished" podID="3180f572-aca4-49ad-adb2-4ac152008f46" containerID="ffe9a99fb7e99dd44418d4c989453aed6ca646067e24fa98ff11eade91253aa1" exitCode=0 Oct 30 15:04:11.799356 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:04:11.799262 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-7f7d1aa4-kserve-5758c45989-npr8j" Oct 30 15:04:11.799356 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:04:11.799293 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-7f7d1aa4-kserve-5758c45989-npr8j" event={"ID":"3180f572-aca4-49ad-adb2-4ac152008f46","Type":"ContainerDied","Data":"ffe9a99fb7e99dd44418d4c989453aed6ca646067e24fa98ff11eade91253aa1"} Oct 30 15:04:11.799356 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:04:11.799336 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-7f7d1aa4-kserve-5758c45989-npr8j" event={"ID":"3180f572-aca4-49ad-adb2-4ac152008f46","Type":"ContainerDied","Data":"18d2e9b658c9f38c2eecbedd6761e803776ff572c6e541d07d283c415723a7b1"} Oct 30 15:04:11.799523 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:04:11.799357 4982 scope.go:122] "RemoveContainer" containerID="ffe9a99fb7e99dd44418d4c989453aed6ca646067e24fa98ff11eade91253aa1" Oct 30 15:04:11.808384 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:04:11.808365 4982 scope.go:122] "RemoveContainer" containerID="ffe9a99fb7e99dd44418d4c989453aed6ca646067e24fa98ff11eade91253aa1" Oct 30 15:04:11.808692 ip-10-0-140-105 kubenswrapper[4982]: E1030 15:04:11.808661 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffe9a99fb7e99dd44418d4c989453aed6ca646067e24fa98ff11eade91253aa1\": container with ID starting with ffe9a99fb7e99dd44418d4c989453aed6ca646067e24fa98ff11eade91253aa1 not found: ID does not exist" containerID="ffe9a99fb7e99dd44418d4c989453aed6ca646067e24fa98ff11eade91253aa1" Oct 30 15:04:11.808757 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:04:11.808704 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffe9a99fb7e99dd44418d4c989453aed6ca646067e24fa98ff11eade91253aa1"} err="failed to get container status \"ffe9a99fb7e99dd44418d4c989453aed6ca646067e24fa98ff11eade91253aa1\": rpc error: code = NotFound desc = could not find container \"ffe9a99fb7e99dd44418d4c989453aed6ca646067e24fa98ff11eade91253aa1\": container with ID starting with ffe9a99fb7e99dd44418d4c989453aed6ca646067e24fa98ff11eade91253aa1 not found: ID does not exist" Oct 30 15:04:11.856509 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:04:11.856487 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-7f7d1aa4-kserve-5758c45989-npr8j"] Oct 30 15:04:11.862028 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:04:11.862005 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-7f7d1aa4-kserve-5758c45989-npr8j"] Oct 30 15:04:12.397392 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:04:12.397364 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3180f572-aca4-49ad-adb2-4ac152008f46" path="/var/lib/kubelet/pods/3180f572-aca4-49ad-adb2-4ac152008f46/volumes" Oct 30 15:04:12.473841 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:04:12.473808 4982 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-a-7f7d1aa4-kserve-5758c45989-npr8j" podUID="3180f572-aca4-49ad-adb2-4ac152008f46" containerName="main" probeResult="failure" output="Get \"https://10.132.1.16:8000/ready\": context deadline exceeded" Oct 30 15:04:19.524988 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:04:19.524954 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:04:28.010257 ip-10-0-140-105 kubenswrapper[4982]: E1030 15:04:28.010222 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 15:04:49.769971 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:04:49.769856 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:04:55.894367 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:04:55.894329 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-hplkd"] Oct 30 15:04:55.894904 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:04:55.894539 4982 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5c48877c75-hplkd" podUID="5fe51702-54a7-4f97-8b08-ffdbdd6fc93a" containerName="maas-api" containerID="cri-o://53c29989b094344bdcf8a4e7e6d33b3aada96b1bedf8509e40c8215e4f2cdb4e" gracePeriod=30 Oct 30 15:04:57.138705 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:04:57.138678 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-hplkd" Oct 30 15:04:57.242959 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:04:57.242884 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fe51702-54a7-4f97-8b08-ffdbdd6fc93a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fe51702-54a7-4f97-8b08-ffdbdd6fc93a-openshift-service-ca\") pod \"5fe51702-54a7-4f97-8b08-ffdbdd6fc93a\" (UID: \"5fe51702-54a7-4f97-8b08-ffdbdd6fc93a\") " Oct 30 15:04:57.242959 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:04:57.242944 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fe51702-54a7-4f97-8b08-ffdbdd6fc93a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fe51702-54a7-4f97-8b08-ffdbdd6fc93a-maas-api-tls\") pod \"5fe51702-54a7-4f97-8b08-ffdbdd6fc93a\" (UID: \"5fe51702-54a7-4f97-8b08-ffdbdd6fc93a\") " Oct 30 15:04:57.243131 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:04:57.242981 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fe51702-54a7-4f97-8b08-ffdbdd6fc93a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fe51702-54a7-4f97-8b08-ffdbdd6fc93a-metrics-certs\") pod \"5fe51702-54a7-4f97-8b08-ffdbdd6fc93a\" (UID: \"5fe51702-54a7-4f97-8b08-ffdbdd6fc93a\") " Oct 30 15:04:57.243131 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:04:57.243032 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fe51702-54a7-4f97-8b08-ffdbdd6fc93a-kube-api-access-p4jjx\" (UniqueName: \"kubernetes.io/projected/5fe51702-54a7-4f97-8b08-ffdbdd6fc93a-kube-api-access-p4jjx\") pod \"5fe51702-54a7-4f97-8b08-ffdbdd6fc93a\" (UID: \"5fe51702-54a7-4f97-8b08-ffdbdd6fc93a\") " Oct 30 15:04:57.243304 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:04:57.243275 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fe51702-54a7-4f97-8b08-ffdbdd6fc93a-openshift-service-ca" pod "5fe51702-54a7-4f97-8b08-ffdbdd6fc93a" (UID: "5fe51702-54a7-4f97-8b08-ffdbdd6fc93a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:04:57.245055 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:04:57.245029 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fe51702-54a7-4f97-8b08-ffdbdd6fc93a-metrics-certs" pod "5fe51702-54a7-4f97-8b08-ffdbdd6fc93a" (UID: "5fe51702-54a7-4f97-8b08-ffdbdd6fc93a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:04:57.245165 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:04:57.245072 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fe51702-54a7-4f97-8b08-ffdbdd6fc93a-maas-api-tls" pod "5fe51702-54a7-4f97-8b08-ffdbdd6fc93a" (UID: "5fe51702-54a7-4f97-8b08-ffdbdd6fc93a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:04:57.245165 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:04:57.245093 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fe51702-54a7-4f97-8b08-ffdbdd6fc93a-kube-api-access-p4jjx" pod "5fe51702-54a7-4f97-8b08-ffdbdd6fc93a" (UID: "5fe51702-54a7-4f97-8b08-ffdbdd6fc93a"). InnerVolumeSpecName "kube-api-access-p4jjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:04:57.343894 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:04:57.343861 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fe51702-54a7-4f97-8b08-ffdbdd6fc93a-maas-api-tls\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 15:04:57.343894 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:04:57.343893 4982 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fe51702-54a7-4f97-8b08-ffdbdd6fc93a-metrics-certs\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 15:04:57.343894 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:04:57.343903 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4jjx\" (UniqueName: \"kubernetes.io/projected/5fe51702-54a7-4f97-8b08-ffdbdd6fc93a-kube-api-access-p4jjx\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 15:04:57.344093 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:04:57.343915 4982 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fe51702-54a7-4f97-8b08-ffdbdd6fc93a-openshift-service-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 15:04:58.008297 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:04:58.008264 4982 generic.go:356] "Generic (PLEG): container finished" podID="5fe51702-54a7-4f97-8b08-ffdbdd6fc93a" containerID="53c29989b094344bdcf8a4e7e6d33b3aada96b1bedf8509e40c8215e4f2cdb4e" exitCode=0 Oct 30 15:04:58.008513 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:04:58.008331 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-hplkd" Oct 30 15:04:58.008513 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:04:58.008356 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-hplkd" event={"ID":"5fe51702-54a7-4f97-8b08-ffdbdd6fc93a","Type":"ContainerDied","Data":"53c29989b094344bdcf8a4e7e6d33b3aada96b1bedf8509e40c8215e4f2cdb4e"} Oct 30 15:04:58.008513 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:04:58.008413 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-hplkd" event={"ID":"5fe51702-54a7-4f97-8b08-ffdbdd6fc93a","Type":"ContainerDied","Data":"c4deee74f0ff479f705413944ffa71af7a5e49a0cc12645a22e8cda317f73e87"} Oct 30 15:04:58.008513 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:04:58.008434 4982 scope.go:122] "RemoveContainer" containerID="53c29989b094344bdcf8a4e7e6d33b3aada96b1bedf8509e40c8215e4f2cdb4e" Oct 30 15:04:58.010258 ip-10-0-140-105 kubenswrapper[4982]: E1030 15:04:58.010231 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 15:04:58.018914 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:04:58.018698 4982 scope.go:122] "RemoveContainer" containerID="53c29989b094344bdcf8a4e7e6d33b3aada96b1bedf8509e40c8215e4f2cdb4e" Oct 30 15:04:58.019004 ip-10-0-140-105 kubenswrapper[4982]: E1030 15:04:58.018986 4982 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53c29989b094344bdcf8a4e7e6d33b3aada96b1bedf8509e40c8215e4f2cdb4e\": container with ID starting with 53c29989b094344bdcf8a4e7e6d33b3aada96b1bedf8509e40c8215e4f2cdb4e not found: ID does not exist" containerID="53c29989b094344bdcf8a4e7e6d33b3aada96b1bedf8509e40c8215e4f2cdb4e" Oct 30 15:04:58.019065 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:04:58.019015 4982 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53c29989b094344bdcf8a4e7e6d33b3aada96b1bedf8509e40c8215e4f2cdb4e"} err="failed to get container status \"53c29989b094344bdcf8a4e7e6d33b3aada96b1bedf8509e40c8215e4f2cdb4e\": rpc error: code = NotFound desc = could not find container \"53c29989b094344bdcf8a4e7e6d33b3aada96b1bedf8509e40c8215e4f2cdb4e\": container with ID starting with 53c29989b094344bdcf8a4e7e6d33b3aada96b1bedf8509e40c8215e4f2cdb4e not found: ID does not exist" Oct 30 15:04:58.064648 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:04:58.064622 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-hplkd"] Oct 30 15:04:58.078022 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:04:58.077997 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-hplkd"] Oct 30 15:04:58.397464 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:04:58.397433 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fe51702-54a7-4f97-8b08-ffdbdd6fc93a" path="/var/lib/kubelet/pods/5fe51702-54a7-4f97-8b08-ffdbdd6fc93a/volumes" Oct 30 15:05:28.010660 ip-10-0-140-105 kubenswrapper[4982]: E1030 15:05:28.010599 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 15:05:40.768568 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:05:40.768525 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:05:42.166664 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:05:42.166608 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:05:49.856366 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:05:49.856328 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:05:58.010374 ip-10-0-140-105 kubenswrapper[4982]: E1030 15:05:58.010329 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 15:06:06.542534 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:06.542494 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jhtht"] Oct 30 15:06:06.542949 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:06.542892 4982 cpu_manager.go:414] "containerMap: removing container" podUID="5fe51702-54a7-4f97-8b08-ffdbdd6fc93a" containerName="maas-api" Oct 30 15:06:06.542949 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:06.542904 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fe51702-54a7-4f97-8b08-ffdbdd6fc93a" containerName="maas-api" Oct 30 15:06:06.542949 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:06.542921 4982 cpu_manager.go:414] "containerMap: removing container" podUID="3180f572-aca4-49ad-adb2-4ac152008f46" containerName="main" Oct 30 15:06:06.542949 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:06.542926 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3180f572-aca4-49ad-adb2-4ac152008f46" containerName="main" Oct 30 15:06:06.543097 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:06.542998 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fe51702-54a7-4f97-8b08-ffdbdd6fc93a" containerName="maas-api" Oct 30 15:06:06.543097 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:06.543008 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="3180f572-aca4-49ad-adb2-4ac152008f46" containerName="main" Oct 30 15:06:06.546059 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:06.546038 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jhtht" Oct 30 15:06:06.563029 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:06.563000 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:06:06.575483 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:06.575447 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:06:06.575483 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:06.575473 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 15:06:06.575483 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:06.575466 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:06:06.577203 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:06.577180 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jhtht"] Oct 30 15:06:06.685535 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:06.685493 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17c065e3-9db1-4d11-95f6-0fb55a170289-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-jhtht\" (UID: \"17c065e3-9db1-4d11-95f6-0fb55a170289\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jhtht" Oct 30 15:06:06.685808 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:06.685558 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kztr\" (UniqueName: \"kubernetes.io/projected/17c065e3-9db1-4d11-95f6-0fb55a170289-kube-api-access-5kztr\") pod \"maas-api-revoke-sub-6b19338f-jhtht\" (UID: \"17c065e3-9db1-4d11-95f6-0fb55a170289\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jhtht" Oct 30 15:06:06.685808 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:06.685716 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17c065e3-9db1-4d11-95f6-0fb55a170289-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-jhtht\" (UID: \"17c065e3-9db1-4d11-95f6-0fb55a170289\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jhtht" Oct 30 15:06:06.787030 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:06.786983 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17c065e3-9db1-4d11-95f6-0fb55a170289-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-jhtht\" (UID: \"17c065e3-9db1-4d11-95f6-0fb55a170289\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jhtht" Oct 30 15:06:06.787220 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:06.787086 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17c065e3-9db1-4d11-95f6-0fb55a170289-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-jhtht\" (UID: \"17c065e3-9db1-4d11-95f6-0fb55a170289\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jhtht" Oct 30 15:06:06.787220 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:06.787133 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kztr\" (UniqueName: \"kubernetes.io/projected/17c065e3-9db1-4d11-95f6-0fb55a170289-kube-api-access-5kztr\") pod \"maas-api-revoke-sub-6b19338f-jhtht\" (UID: \"17c065e3-9db1-4d11-95f6-0fb55a170289\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jhtht" Oct 30 15:06:06.787828 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:06.787807 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17c065e3-9db1-4d11-95f6-0fb55a170289-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-jhtht\" (UID: \"17c065e3-9db1-4d11-95f6-0fb55a170289\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jhtht" Oct 30 15:06:06.789520 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:06.789501 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17c065e3-9db1-4d11-95f6-0fb55a170289-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-jhtht\" (UID: \"17c065e3-9db1-4d11-95f6-0fb55a170289\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jhtht" Oct 30 15:06:06.813378 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:06.813344 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kztr\" (UniqueName: \"kubernetes.io/projected/17c065e3-9db1-4d11-95f6-0fb55a170289-kube-api-access-5kztr\") pod \"maas-api-revoke-sub-6b19338f-jhtht\" (UID: \"17c065e3-9db1-4d11-95f6-0fb55a170289\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jhtht" Oct 30 15:06:06.858634 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:06.858591 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jhtht" Oct 30 15:06:07.156935 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:07.156903 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jhtht"] Oct 30 15:06:07.158558 ip-10-0-140-105 kubenswrapper[4982]: W1030 15:06:07.158525 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17c065e3_9db1_4d11_95f6_0fb55a170289.slice/crio-a4adef7cbf62a8bdfa9207cdd140d358af15d25b6563a2ece21ce5a6b0988580 WatchSource:0}: Error finding container a4adef7cbf62a8bdfa9207cdd140d358af15d25b6563a2ece21ce5a6b0988580: Status 404 returned error can't find the container with id a4adef7cbf62a8bdfa9207cdd140d358af15d25b6563a2ece21ce5a6b0988580 Oct 30 15:06:07.329143 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:07.329102 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jhtht" event={"ID":"17c065e3-9db1-4d11-95f6-0fb55a170289","Type":"ContainerStarted","Data":"a4adef7cbf62a8bdfa9207cdd140d358af15d25b6563a2ece21ce5a6b0988580"} Oct 30 15:06:08.334835 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:08.334800 4982 generic.go:356] "Generic (PLEG): container finished" podID="17c065e3-9db1-4d11-95f6-0fb55a170289" containerID="640b2e029e7b2b918914f78ccfd382a8141139efd24aa241ee5e19f45eefdeeb" exitCode=0 Oct 30 15:06:08.335241 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:08.334867 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jhtht" event={"ID":"17c065e3-9db1-4d11-95f6-0fb55a170289","Type":"ContainerDied","Data":"640b2e029e7b2b918914f78ccfd382a8141139efd24aa241ee5e19f45eefdeeb"} Oct 30 15:06:09.479361 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:09.479333 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jhtht" Oct 30 15:06:09.614202 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:09.614117 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17c065e3-9db1-4d11-95f6-0fb55a170289-kube-api-access-5kztr\" (UniqueName: \"kubernetes.io/projected/17c065e3-9db1-4d11-95f6-0fb55a170289-kube-api-access-5kztr\") pod \"17c065e3-9db1-4d11-95f6-0fb55a170289\" (UID: \"17c065e3-9db1-4d11-95f6-0fb55a170289\") " Oct 30 15:06:09.614360 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:09.614224 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17c065e3-9db1-4d11-95f6-0fb55a170289-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17c065e3-9db1-4d11-95f6-0fb55a170289-maas-api-xks-ca\") pod \"17c065e3-9db1-4d11-95f6-0fb55a170289\" (UID: \"17c065e3-9db1-4d11-95f6-0fb55a170289\") " Oct 30 15:06:09.614360 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:09.614264 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17c065e3-9db1-4d11-95f6-0fb55a170289-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17c065e3-9db1-4d11-95f6-0fb55a170289-maas-api-openshift-ca\") pod \"17c065e3-9db1-4d11-95f6-0fb55a170289\" (UID: \"17c065e3-9db1-4d11-95f6-0fb55a170289\") " Oct 30 15:06:09.614718 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:09.614688 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17c065e3-9db1-4d11-95f6-0fb55a170289-maas-api-openshift-ca" pod "17c065e3-9db1-4d11-95f6-0fb55a170289" (UID: "17c065e3-9db1-4d11-95f6-0fb55a170289"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:06:09.616327 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:09.616297 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17c065e3-9db1-4d11-95f6-0fb55a170289-maas-api-xks-ca" pod "17c065e3-9db1-4d11-95f6-0fb55a170289" (UID: "17c065e3-9db1-4d11-95f6-0fb55a170289"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:06:09.616432 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:09.616382 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17c065e3-9db1-4d11-95f6-0fb55a170289-kube-api-access-5kztr" pod "17c065e3-9db1-4d11-95f6-0fb55a170289" (UID: "17c065e3-9db1-4d11-95f6-0fb55a170289"). InnerVolumeSpecName "kube-api-access-5kztr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:06:09.715460 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:09.715419 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17c065e3-9db1-4d11-95f6-0fb55a170289-maas-api-xks-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 15:06:09.715460 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:09.715453 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17c065e3-9db1-4d11-95f6-0fb55a170289-maas-api-openshift-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 15:06:09.715460 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:09.715463 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kztr\" (UniqueName: \"kubernetes.io/projected/17c065e3-9db1-4d11-95f6-0fb55a170289-kube-api-access-5kztr\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 15:06:10.345729 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:10.345686 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jhtht" event={"ID":"17c065e3-9db1-4d11-95f6-0fb55a170289","Type":"ContainerDied","Data":"a4adef7cbf62a8bdfa9207cdd140d358af15d25b6563a2ece21ce5a6b0988580"} Oct 30 15:06:10.345729 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:10.345717 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jhtht" Oct 30 15:06:10.345966 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:10.345743 4982 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4adef7cbf62a8bdfa9207cdd140d358af15d25b6563a2ece21ce5a6b0988580" Oct 30 15:06:28.011448 ip-10-0-140-105 kubenswrapper[4982]: E1030 15:06:28.011402 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 15:06:29.004601 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:29.004558 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tnjbh"] Oct 30 15:06:29.005029 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:29.005010 4982 cpu_manager.go:414] "containerMap: removing container" podUID="17c065e3-9db1-4d11-95f6-0fb55a170289" containerName="revoke-keys" Oct 30 15:06:29.005075 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:29.005030 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17c065e3-9db1-4d11-95f6-0fb55a170289" containerName="revoke-keys" Oct 30 15:06:29.005111 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:29.005103 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="17c065e3-9db1-4d11-95f6-0fb55a170289" containerName="revoke-keys" Oct 30 15:06:29.008427 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:29.008401 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tnjbh" Oct 30 15:06:29.008883 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:29.008858 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82b47295-2835-4ff3-8a70-3cda9dd85710-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-tnjbh\" (UID: \"82b47295-2835-4ff3-8a70-3cda9dd85710\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tnjbh" Oct 30 15:06:29.009007 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:29.008917 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82b47295-2835-4ff3-8a70-3cda9dd85710-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-tnjbh\" (UID: \"82b47295-2835-4ff3-8a70-3cda9dd85710\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tnjbh" Oct 30 15:06:29.009060 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:29.009039 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj8j8\" (UniqueName: \"kubernetes.io/projected/82b47295-2835-4ff3-8a70-3cda9dd85710-kube-api-access-rj8j8\") pod \"maas-api-revoke-sub-f03629db-tnjbh\" (UID: \"82b47295-2835-4ff3-8a70-3cda9dd85710\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tnjbh" Oct 30 15:06:29.025485 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:29.025456 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:06:29.038140 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:29.038106 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 15:06:29.038293 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:29.038106 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:06:29.038293 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:29.038106 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:06:29.039900 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:29.039875 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tnjbh"] Oct 30 15:06:29.109859 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:29.109814 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82b47295-2835-4ff3-8a70-3cda9dd85710-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-tnjbh\" (UID: \"82b47295-2835-4ff3-8a70-3cda9dd85710\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tnjbh" Oct 30 15:06:29.109859 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:29.109865 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82b47295-2835-4ff3-8a70-3cda9dd85710-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-tnjbh\" (UID: \"82b47295-2835-4ff3-8a70-3cda9dd85710\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tnjbh" Oct 30 15:06:29.110152 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:29.109891 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj8j8\" (UniqueName: \"kubernetes.io/projected/82b47295-2835-4ff3-8a70-3cda9dd85710-kube-api-access-rj8j8\") pod \"maas-api-revoke-sub-f03629db-tnjbh\" (UID: \"82b47295-2835-4ff3-8a70-3cda9dd85710\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tnjbh" Oct 30 15:06:29.110443 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:29.110416 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82b47295-2835-4ff3-8a70-3cda9dd85710-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-tnjbh\" (UID: \"82b47295-2835-4ff3-8a70-3cda9dd85710\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tnjbh" Oct 30 15:06:29.112222 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:29.112191 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82b47295-2835-4ff3-8a70-3cda9dd85710-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-tnjbh\" (UID: \"82b47295-2835-4ff3-8a70-3cda9dd85710\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tnjbh" Oct 30 15:06:29.135952 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:29.135920 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj8j8\" (UniqueName: \"kubernetes.io/projected/82b47295-2835-4ff3-8a70-3cda9dd85710-kube-api-access-rj8j8\") pod \"maas-api-revoke-sub-f03629db-tnjbh\" (UID: \"82b47295-2835-4ff3-8a70-3cda9dd85710\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tnjbh" Oct 30 15:06:29.320882 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:29.320835 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tnjbh" Oct 30 15:06:29.620367 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:29.620332 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tnjbh"] Oct 30 15:06:29.621752 ip-10-0-140-105 kubenswrapper[4982]: W1030 15:06:29.621723 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82b47295_2835_4ff3_8a70_3cda9dd85710.slice/crio-20ba5cbbd9178a3c91d141ab76ace5599891ba7633e0f2ba438f17d97ff764da WatchSource:0}: Error finding container 20ba5cbbd9178a3c91d141ab76ace5599891ba7633e0f2ba438f17d97ff764da: Status 404 returned error can't find the container with id 20ba5cbbd9178a3c91d141ab76ace5599891ba7633e0f2ba438f17d97ff764da Oct 30 15:06:30.271565 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:30.271475 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:30.409837 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:30.409778 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:30.478441 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:30.478411 4982 generic.go:356] "Generic (PLEG): container finished" podID="82b47295-2835-4ff3-8a70-3cda9dd85710" containerID="8db8a65648c86a0abd0ae0a585312f75d7c226ab03d94c0de67e8afe2ebb3d38" exitCode=0 Oct 30 15:06:30.478606 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:30.478497 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tnjbh" event={"ID":"82b47295-2835-4ff3-8a70-3cda9dd85710","Type":"ContainerDied","Data":"8db8a65648c86a0abd0ae0a585312f75d7c226ab03d94c0de67e8afe2ebb3d38"} Oct 30 15:06:30.478606 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:30.478552 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tnjbh" event={"ID":"82b47295-2835-4ff3-8a70-3cda9dd85710","Type":"ContainerStarted","Data":"20ba5cbbd9178a3c91d141ab76ace5599891ba7633e0f2ba438f17d97ff764da"} Oct 30 15:06:31.619921 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:31.619896 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tnjbh" Oct 30 15:06:31.632013 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:31.631977 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82b47295-2835-4ff3-8a70-3cda9dd85710-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82b47295-2835-4ff3-8a70-3cda9dd85710-maas-api-openshift-ca\") pod \"82b47295-2835-4ff3-8a70-3cda9dd85710\" (UID: \"82b47295-2835-4ff3-8a70-3cda9dd85710\") " Oct 30 15:06:31.632189 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:31.632022 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82b47295-2835-4ff3-8a70-3cda9dd85710-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82b47295-2835-4ff3-8a70-3cda9dd85710-maas-api-xks-ca\") pod \"82b47295-2835-4ff3-8a70-3cda9dd85710\" (UID: \"82b47295-2835-4ff3-8a70-3cda9dd85710\") " Oct 30 15:06:31.632189 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:31.632089 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82b47295-2835-4ff3-8a70-3cda9dd85710-kube-api-access-rj8j8\" (UniqueName: \"kubernetes.io/projected/82b47295-2835-4ff3-8a70-3cda9dd85710-kube-api-access-rj8j8\") pod \"82b47295-2835-4ff3-8a70-3cda9dd85710\" (UID: \"82b47295-2835-4ff3-8a70-3cda9dd85710\") " Oct 30 15:06:31.632336 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:31.632300 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82b47295-2835-4ff3-8a70-3cda9dd85710-maas-api-openshift-ca" pod "82b47295-2835-4ff3-8a70-3cda9dd85710" (UID: "82b47295-2835-4ff3-8a70-3cda9dd85710"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:06:31.632448 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:31.632435 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82b47295-2835-4ff3-8a70-3cda9dd85710-maas-api-openshift-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 15:06:31.634178 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:31.634155 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82b47295-2835-4ff3-8a70-3cda9dd85710-maas-api-xks-ca" pod "82b47295-2835-4ff3-8a70-3cda9dd85710" (UID: "82b47295-2835-4ff3-8a70-3cda9dd85710"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:06:31.634269 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:31.634254 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82b47295-2835-4ff3-8a70-3cda9dd85710-kube-api-access-rj8j8" pod "82b47295-2835-4ff3-8a70-3cda9dd85710" (UID: "82b47295-2835-4ff3-8a70-3cda9dd85710"). InnerVolumeSpecName "kube-api-access-rj8j8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:06:31.733083 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:31.732991 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82b47295-2835-4ff3-8a70-3cda9dd85710-maas-api-xks-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 15:06:31.733083 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:31.733026 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rj8j8\" (UniqueName: \"kubernetes.io/projected/82b47295-2835-4ff3-8a70-3cda9dd85710-kube-api-access-rj8j8\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 15:06:32.487901 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:32.487861 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tnjbh" event={"ID":"82b47295-2835-4ff3-8a70-3cda9dd85710","Type":"ContainerDied","Data":"20ba5cbbd9178a3c91d141ab76ace5599891ba7633e0f2ba438f17d97ff764da"} Oct 30 15:06:32.488064 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:32.487918 4982 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20ba5cbbd9178a3c91d141ab76ace5599891ba7633e0f2ba438f17d97ff764da" Oct 30 15:06:32.488064 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:32.487879 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tnjbh" Oct 30 15:06:49.982289 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:06:49.982251 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:58.010746 ip-10-0-140-105 kubenswrapper[4982]: E1030 15:06:58.010702 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 15:07:07.907357 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:07:07.907320 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dpjc6"] Oct 30 15:07:07.907896 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:07:07.907732 4982 cpu_manager.go:414] "containerMap: removing container" podUID="82b47295-2835-4ff3-8a70-3cda9dd85710" containerName="revoke-keys" Oct 30 15:07:07.907896 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:07:07.907745 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82b47295-2835-4ff3-8a70-3cda9dd85710" containerName="revoke-keys" Oct 30 15:07:07.907896 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:07:07.907811 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="82b47295-2835-4ff3-8a70-3cda9dd85710" containerName="revoke-keys" Oct 30 15:07:07.910873 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:07:07.910855 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dpjc6" Oct 30 15:07:07.928289 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:07:07.928255 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:07:07.941009 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:07:07.940971 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:07:07.941009 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:07:07.940974 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:07:07.941243 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:07:07.941127 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 15:07:07.943461 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:07:07.943437 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dpjc6"] Oct 30 15:07:07.994559 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:07:07.994521 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/135829ee-0f75-460d-a3da-cea671894b08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-dpjc6\" (UID: \"135829ee-0f75-460d-a3da-cea671894b08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dpjc6" Oct 30 15:07:07.994559 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:07:07.994559 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcs6c\" (UniqueName: \"kubernetes.io/projected/135829ee-0f75-460d-a3da-cea671894b08-kube-api-access-mcs6c\") pod \"maas-api-revoke-sub-0165d7d2-dpjc6\" (UID: \"135829ee-0f75-460d-a3da-cea671894b08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dpjc6" Oct 30 15:07:07.994808 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:07:07.994581 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/135829ee-0f75-460d-a3da-cea671894b08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-dpjc6\" (UID: \"135829ee-0f75-460d-a3da-cea671894b08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dpjc6" Oct 30 15:07:08.095595 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:07:08.095548 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/135829ee-0f75-460d-a3da-cea671894b08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-dpjc6\" (UID: \"135829ee-0f75-460d-a3da-cea671894b08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dpjc6" Oct 30 15:07:08.095595 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:07:08.095593 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcs6c\" (UniqueName: \"kubernetes.io/projected/135829ee-0f75-460d-a3da-cea671894b08-kube-api-access-mcs6c\") pod \"maas-api-revoke-sub-0165d7d2-dpjc6\" (UID: \"135829ee-0f75-460d-a3da-cea671894b08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dpjc6" Oct 30 15:07:08.095855 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:07:08.095617 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/135829ee-0f75-460d-a3da-cea671894b08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-dpjc6\" (UID: \"135829ee-0f75-460d-a3da-cea671894b08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dpjc6" Oct 30 15:07:08.096344 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:07:08.096301 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/135829ee-0f75-460d-a3da-cea671894b08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-dpjc6\" (UID: \"135829ee-0f75-460d-a3da-cea671894b08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dpjc6" Oct 30 15:07:08.097964 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:07:08.097936 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/135829ee-0f75-460d-a3da-cea671894b08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-dpjc6\" (UID: \"135829ee-0f75-460d-a3da-cea671894b08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dpjc6" Oct 30 15:07:08.121689 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:07:08.121654 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcs6c\" (UniqueName: \"kubernetes.io/projected/135829ee-0f75-460d-a3da-cea671894b08-kube-api-access-mcs6c\") pod \"maas-api-revoke-sub-0165d7d2-dpjc6\" (UID: \"135829ee-0f75-460d-a3da-cea671894b08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dpjc6" Oct 30 15:07:08.224891 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:07:08.224794 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dpjc6" Oct 30 15:07:08.522233 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:07:08.522201 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dpjc6"] Oct 30 15:07:08.523933 ip-10-0-140-105 kubenswrapper[4982]: W1030 15:07:08.523906 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod135829ee_0f75_460d_a3da_cea671894b08.slice/crio-b8eb1175fd0e38e59912f360c9c204443b3978a5035be8f21be392e1bbabe999 WatchSource:0}: Error finding container b8eb1175fd0e38e59912f360c9c204443b3978a5035be8f21be392e1bbabe999: Status 404 returned error can't find the container with id b8eb1175fd0e38e59912f360c9c204443b3978a5035be8f21be392e1bbabe999 Oct 30 15:07:08.656663 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:07:08.656604 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dpjc6" event={"ID":"135829ee-0f75-460d-a3da-cea671894b08","Type":"ContainerStarted","Data":"b8eb1175fd0e38e59912f360c9c204443b3978a5035be8f21be392e1bbabe999"} Oct 30 15:07:09.664609 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:07:09.664520 4982 generic.go:356] "Generic (PLEG): container finished" podID="135829ee-0f75-460d-a3da-cea671894b08" containerID="9c899ba668e3bc590cd97ab67989aa4bd54dbffe7e251efc902a5b2fe26d0431" exitCode=0 Oct 30 15:07:09.665010 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:07:09.664605 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dpjc6" event={"ID":"135829ee-0f75-460d-a3da-cea671894b08","Type":"ContainerDied","Data":"9c899ba668e3bc590cd97ab67989aa4bd54dbffe7e251efc902a5b2fe26d0431"} Oct 30 15:07:10.799611 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:07:10.799584 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dpjc6" Oct 30 15:07:10.821075 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:07:10.821038 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/135829ee-0f75-460d-a3da-cea671894b08-kube-api-access-mcs6c\" (UniqueName: \"kubernetes.io/projected/135829ee-0f75-460d-a3da-cea671894b08-kube-api-access-mcs6c\") pod \"135829ee-0f75-460d-a3da-cea671894b08\" (UID: \"135829ee-0f75-460d-a3da-cea671894b08\") " Oct 30 15:07:10.821225 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:07:10.821100 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/135829ee-0f75-460d-a3da-cea671894b08-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/135829ee-0f75-460d-a3da-cea671894b08-maas-api-xks-ca\") pod \"135829ee-0f75-460d-a3da-cea671894b08\" (UID: \"135829ee-0f75-460d-a3da-cea671894b08\") " Oct 30 15:07:10.821225 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:07:10.821151 4982 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/135829ee-0f75-460d-a3da-cea671894b08-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/135829ee-0f75-460d-a3da-cea671894b08-maas-api-openshift-ca\") pod \"135829ee-0f75-460d-a3da-cea671894b08\" (UID: \"135829ee-0f75-460d-a3da-cea671894b08\") " Oct 30 15:07:10.821568 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:07:10.821539 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/135829ee-0f75-460d-a3da-cea671894b08-maas-api-openshift-ca" pod "135829ee-0f75-460d-a3da-cea671894b08" (UID: "135829ee-0f75-460d-a3da-cea671894b08"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:07:10.823322 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:07:10.823299 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/135829ee-0f75-460d-a3da-cea671894b08-kube-api-access-mcs6c" pod "135829ee-0f75-460d-a3da-cea671894b08" (UID: "135829ee-0f75-460d-a3da-cea671894b08"). InnerVolumeSpecName "kube-api-access-mcs6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:07:10.823407 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:07:10.823331 4982 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/135829ee-0f75-460d-a3da-cea671894b08-maas-api-xks-ca" pod "135829ee-0f75-460d-a3da-cea671894b08" (UID: "135829ee-0f75-460d-a3da-cea671894b08"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:07:10.922108 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:07:10.922016 4982 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcs6c\" (UniqueName: \"kubernetes.io/projected/135829ee-0f75-460d-a3da-cea671894b08-kube-api-access-mcs6c\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 15:07:10.922108 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:07:10.922048 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/135829ee-0f75-460d-a3da-cea671894b08-maas-api-xks-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 15:07:10.922108 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:07:10.922059 4982 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/135829ee-0f75-460d-a3da-cea671894b08-maas-api-openshift-ca\") on node \"ip-10-0-140-105.ec2.internal\" DevicePath \"\"" Oct 30 15:07:11.674775 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:07:11.674743 4982 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dpjc6" Oct 30 15:07:11.674975 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:07:11.674778 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dpjc6" event={"ID":"135829ee-0f75-460d-a3da-cea671894b08","Type":"ContainerDied","Data":"b8eb1175fd0e38e59912f360c9c204443b3978a5035be8f21be392e1bbabe999"} Oct 30 15:07:11.674975 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:07:11.674836 4982 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8eb1175fd0e38e59912f360c9c204443b3978a5035be8f21be392e1bbabe999" Oct 30 15:07:24.322580 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:07:24.322538 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:24.468767 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:07:24.468734 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:28.011357 ip-10-0-140-105 kubenswrapper[4982]: E1030 15:07:28.011322 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 15:07:50.102463 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:07:50.102382 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:50.274511 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:07:50.274477 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-bxsb8_1be45a6e-572d-4d76-a68b-08464bac8146/manager/0.log" Oct 30 15:07:50.468843 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:07:50.468763 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-5cfb77bf8c-ct5zj_ad1f34b6-efe9-4981-8d30-845e2accbc06/manager/0.log" Oct 30 15:07:50.698549 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:07:50.698510 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-j5x4l_3c2a2ce1-4b10-45db-846b-574da819ac25/manager/0.log" Oct 30 15:07:51.220090 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:07:51.220059 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-64r4n_4a67cbd2-b867-4cf9-9cfd-c35ae950a761/server/0.log" Oct 30 15:07:52.102934 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:07:52.102898 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d55fbd5c9-47k9k_cbfe1a62-d260-44c1-9eda-ac16409bb69b/copy-manifests/0.log" Oct 30 15:07:52.142000 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:07:52.141968 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d55fbd5c9-47k9k_cbfe1a62-d260-44c1-9eda-ac16409bb69b/manager/0.log" Oct 30 15:07:53.677909 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:07:53.677865 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:54.742573 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:07:54.742535 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:58.010552 ip-10-0-140-105 kubenswrapper[4982]: E1030 15:07:58.010516 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 15:08:03.857430 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:08:03.857393 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-dpjc6_135829ee-0f75-460d-a3da-cea671894b08/revoke-keys/0.log" Oct 30 15:08:04.613751 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:08:04.613717 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-jhtht_17c065e3-9db1-4d11-95f6-0fb55a170289/revoke-keys/0.log" Oct 30 15:08:04.792236 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:08:04.792205 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-tnjbh_82b47295-2835-4ff3-8a70-3cda9dd85710/revoke-keys/0.log" Oct 30 15:08:07.175858 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:08:07.175822 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-5m4s9_303ce1a6-cb22-49ce-b424-be1f64e5f950/kuadrant-console-plugin/0.log" Oct 30 15:08:08.578499 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:08:08.578456 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-98f7774f-vb9c4_84f3cb27-ca2f-4a9c-8f30-53fe968ec15d/istio-proxy/0.log" Oct 30 15:08:10.157407 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:08:10.157367 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-f2kp6_f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc/istio-proxy/0.log" Oct 30 15:08:10.758251 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:08:10.758215 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-kxfp8_556451f5-3d37-40a5-bf8f-f470d05ef183/istio-proxy/0.log" Oct 30 15:08:10.949080 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:08:10.949046 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-mqc7s_f784c488-9f50-4924-a54b-811e5be8661f/istio-proxy/0.log" Oct 30 15:08:12.626455 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:08:12.626419 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:21.735663 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:08:21.735616 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-659879fb99-pjvcq_6d73c6c3-044d-4b88-a105-72bd5e4491cd/router/0.log" Oct 30 15:08:28.011173 ip-10-0-140-105 kubenswrapper[4982]: E1030 15:08:28.011133 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 15:08:44.842237 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:08:44.842199 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-bxsb8_1be45a6e-572d-4d76-a68b-08464bac8146/manager/0.log" Oct 30 15:08:45.048830 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:08:45.048798 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-5cfb77bf8c-ct5zj_ad1f34b6-efe9-4981-8d30-845e2accbc06/manager/0.log" Oct 30 15:08:45.267503 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:08:45.267410 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-j5x4l_3c2a2ce1-4b10-45db-846b-574da819ac25/manager/0.log" Oct 30 15:08:45.731258 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:08:45.731227 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-64r4n_4a67cbd2-b867-4cf9-9cfd-c35ae950a761/server/0.log" Oct 30 15:08:46.647863 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:08:46.647832 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d55fbd5c9-47k9k_cbfe1a62-d260-44c1-9eda-ac16409bb69b/copy-manifests/0.log" Oct 30 15:08:46.683969 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:08:46.683938 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d55fbd5c9-47k9k_cbfe1a62-d260-44c1-9eda-ac16409bb69b/manager/0.log" Oct 30 15:08:48.158426 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:08:48.158397 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-dpjc6_135829ee-0f75-460d-a3da-cea671894b08/revoke-keys/0.log" Oct 30 15:08:48.948715 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:08:48.948676 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-jhtht_17c065e3-9db1-4d11-95f6-0fb55a170289/revoke-keys/0.log" Oct 30 15:08:49.152147 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:08:49.152109 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-tnjbh_82b47295-2835-4ff3-8a70-3cda9dd85710/revoke-keys/0.log" Oct 30 15:08:50.216942 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:08:50.216906 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:51.127310 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:08:51.127274 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ntlqq_dc967875-1546-40bb-8376-fde09416c204/console-operator/2.log" Oct 30 15:08:51.132590 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:08:51.132564 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhxjf_aca47fe8-7068-4743-b6ab-283ee0874f66/ovn-acl-logging/0.log" Oct 30 15:08:51.138814 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:08:51.138793 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ntlqq_dc967875-1546-40bb-8376-fde09416c204/console-operator/2.log" Oct 30 15:08:51.143822 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:08:51.143801 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhxjf_aca47fe8-7068-4743-b6ab-283ee0874f66/ovn-acl-logging/0.log" Oct 30 15:08:58.010352 ip-10-0-140-105 kubenswrapper[4982]: E1030 15:08:58.010314 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 15:09:01.190845 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:09:01.190816 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-5m4s9_303ce1a6-cb22-49ce-b424-be1f64e5f950/kuadrant-console-plugin/0.log" Oct 30 15:09:12.479612 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:09:12.479566 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-98f7774f-vb9c4_84f3cb27-ca2f-4a9c-8f30-53fe968ec15d/istio-proxy/0.log" Oct 30 15:09:14.189482 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:09:14.189447 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-f2kp6_f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc/istio-proxy/0.log" Oct 30 15:09:14.809488 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:09:14.809453 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-kxfp8_556451f5-3d37-40a5-bf8f-f470d05ef183/istio-proxy/0.log" Oct 30 15:09:15.023024 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:09:15.022995 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-mqc7s_f784c488-9f50-4924-a54b-811e5be8661f/istio-proxy/0.log" Oct 30 15:09:15.838551 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:09:15.838522 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-659879fb99-pjvcq_6d73c6c3-044d-4b88-a105-72bd5e4491cd/router/0.log" Oct 30 15:09:28.010550 ip-10-0-140-105 kubenswrapper[4982]: E1030 15:09:28.010511 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 15:09:50.350521 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:09:50.350486 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:09:58.010677 ip-10-0-140-105 kubenswrapper[4982]: E1030 15:09:58.010619 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 15:10:28.010859 ip-10-0-140-105 kubenswrapper[4982]: E1030 15:10:28.010823 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 15:10:50.449703 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:10:50.449663 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:10:58.009565 ip-10-0-140-105 kubenswrapper[4982]: E1030 15:10:58.009525 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 15:11:03.507288 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:03.507254 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:11:10.032895 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:10.032860 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jhtht"] Oct 30 15:11:10.045995 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:10.045964 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jhtht"] Oct 30 15:11:10.397619 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:10.397590 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17c065e3-9db1-4d11-95f6-0fb55a170289" path="/var/lib/kubelet/pods/17c065e3-9db1-4d11-95f6-0fb55a170289/volumes" Oct 30 15:11:28.010256 ip-10-0-140-105 kubenswrapper[4982]: E1030 15:11:28.010220 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 15:11:28.626231 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:28.626202 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-48dpl/must-gather-hmlrz"] Oct 30 15:11:28.626551 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:28.626540 4982 cpu_manager.go:414] "containerMap: removing container" podUID="135829ee-0f75-460d-a3da-cea671894b08" containerName="revoke-keys" Oct 30 15:11:28.626597 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:28.626553 4982 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="135829ee-0f75-460d-a3da-cea671894b08" containerName="revoke-keys" Oct 30 15:11:28.626671 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:28.626661 4982 memory_manager.go:365] "RemoveStaleState removing state" podUID="135829ee-0f75-460d-a3da-cea671894b08" containerName="revoke-keys" Oct 30 15:11:28.629864 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:28.629845 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-48dpl/must-gather-hmlrz" Oct 30 15:11:28.642685 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:28.642655 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:11:28.659273 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:28.659249 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-48dpl\"/\"openshift-service-ca.crt\"" Oct 30 15:11:28.659273 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:28.659263 4982 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-48dpl\"/\"default-dockercfg-dvdxx\"" Oct 30 15:11:28.659399 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:28.659249 4982 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-48dpl\"/\"kube-root-ca.crt\"" Oct 30 15:11:28.664460 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:28.664436 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-48dpl/must-gather-hmlrz"] Oct 30 15:11:28.690588 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:28.690558 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/423ecc11-f38f-4946-94b3-2068207db108-must-gather-output\") pod \"must-gather-hmlrz\" (UID: \"423ecc11-f38f-4946-94b3-2068207db108\") " pod="openshift-must-gather-48dpl/must-gather-hmlrz" Oct 30 15:11:28.690738 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:28.690628 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zl5p\" (UniqueName: \"kubernetes.io/projected/423ecc11-f38f-4946-94b3-2068207db108-kube-api-access-6zl5p\") pod \"must-gather-hmlrz\" (UID: \"423ecc11-f38f-4946-94b3-2068207db108\") " pod="openshift-must-gather-48dpl/must-gather-hmlrz" Oct 30 15:11:28.791623 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:28.791592 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zl5p\" (UniqueName: \"kubernetes.io/projected/423ecc11-f38f-4946-94b3-2068207db108-kube-api-access-6zl5p\") pod \"must-gather-hmlrz\" (UID: \"423ecc11-f38f-4946-94b3-2068207db108\") " pod="openshift-must-gather-48dpl/must-gather-hmlrz" Oct 30 15:11:28.791768 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:28.791693 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/423ecc11-f38f-4946-94b3-2068207db108-must-gather-output\") pod \"must-gather-hmlrz\" (UID: \"423ecc11-f38f-4946-94b3-2068207db108\") " pod="openshift-must-gather-48dpl/must-gather-hmlrz" Oct 30 15:11:28.792018 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:28.791998 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/423ecc11-f38f-4946-94b3-2068207db108-must-gather-output\") pod \"must-gather-hmlrz\" (UID: \"423ecc11-f38f-4946-94b3-2068207db108\") " pod="openshift-must-gather-48dpl/must-gather-hmlrz" Oct 30 15:11:28.799878 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:28.799853 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:11:28.816920 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:28.816897 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zl5p\" (UniqueName: \"kubernetes.io/projected/423ecc11-f38f-4946-94b3-2068207db108-kube-api-access-6zl5p\") pod \"must-gather-hmlrz\" (UID: \"423ecc11-f38f-4946-94b3-2068207db108\") " pod="openshift-must-gather-48dpl/must-gather-hmlrz" Oct 30 15:11:28.943469 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:28.943412 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-48dpl/must-gather-hmlrz" Oct 30 15:11:29.240128 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:29.240093 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-48dpl/must-gather-hmlrz"] Oct 30 15:11:29.241309 ip-10-0-140-105 kubenswrapper[4982]: W1030 15:11:29.241276 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod423ecc11_f38f_4946_94b3_2068207db108.slice/crio-19c98118ff00ca8ff747394867959dd808ab53b9db54e4e2d00bea3a06ea5a5c WatchSource:0}: Error finding container 19c98118ff00ca8ff747394867959dd808ab53b9db54e4e2d00bea3a06ea5a5c: Status 404 returned error can't find the container with id 19c98118ff00ca8ff747394867959dd808ab53b9db54e4e2d00bea3a06ea5a5c Oct 30 15:11:29.243233 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:29.243204 4982 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 15:11:29.818165 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:29.818130 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-48dpl/must-gather-hmlrz" event={"ID":"423ecc11-f38f-4946-94b3-2068207db108","Type":"ContainerStarted","Data":"19c98118ff00ca8ff747394867959dd808ab53b9db54e4e2d00bea3a06ea5a5c"} Oct 30 15:11:30.825385 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:30.825346 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-48dpl/must-gather-hmlrz" event={"ID":"423ecc11-f38f-4946-94b3-2068207db108","Type":"ContainerStarted","Data":"a1170396632474cffeb0739ea7684a15d2644361c37c9afc5f3e25d7cddf6f50"} Oct 30 15:11:30.825907 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:30.825395 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-48dpl/must-gather-hmlrz" event={"ID":"423ecc11-f38f-4946-94b3-2068207db108","Type":"ContainerStarted","Data":"682c5385e644423bc76b2594a8434c7fa1031ab264974fee2545b20b602f1b88"} Oct 30 15:11:30.869031 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:30.868984 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-48dpl/must-gather-hmlrz" podStartSLOduration=2.049978393 podStartE2EDuration="2.868970163s" totalImagesPullingTime="818.99177ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:11:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:11:30.868477278 +0000 UTC m=+4960.612611297" watchObservedRunningTime="2026-10-30 15:11:30.868970163 +0000 UTC m=+4960.613104268" Oct 30 15:11:32.039912 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:32.039866 4982 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tnjbh"] Oct 30 15:11:32.051570 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:32.051518 4982 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tnjbh"] Oct 30 15:11:32.397268 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:32.397234 4982 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82b47295-2835-4ff3-8a70-3cda9dd85710" path="/var/lib/kubelet/pods/82b47295-2835-4ff3-8a70-3cda9dd85710/volumes" Oct 30 15:11:33.225478 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:33.225445 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-bdmgn_6b5e74da-5460-44ed-a599-b01620b64c92/global-pull-secret-syncer/0.log" Oct 30 15:11:33.636597 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:33.636567 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-4h2sq_dad9381c-aebb-448d-80b1-e2b65dd8ac80/connection-checker/0.log" Oct 30 15:11:33.771663 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:33.765801 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-plvgd_d7a64da6-7b20-4f20-90d7-bc23a71290e5/connection-checker/0.log" Oct 30 15:11:33.912291 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:33.912220 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-smvmg_cb90396e-d4e9-4eb0-888f-64c59d492fd4/connection-checker/0.log" Oct 30 15:11:34.106891 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:34.106852 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" Oct 30 15:11:34.789193 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:34.789161 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-105.ec2.internal_7f6d989ed16482ea69070d9acfe2e5f0/haproxy/0.log" Oct 30 15:11:41.107812 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:41.107775 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-5m4s9_303ce1a6-cb22-49ce-b424-be1f64e5f950/kuadrant-console-plugin/0.log" Oct 30 15:11:42.192805 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:42.192751 4982 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-48dpl/perf-node-gather-daemonset-74dxq"] Oct 30 15:11:42.201863 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:42.201831 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-48dpl/perf-node-gather-daemonset-74dxq" Oct 30 15:11:42.218790 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:42.216383 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-48dpl/perf-node-gather-daemonset-74dxq"] Oct 30 15:11:42.254577 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:42.254533 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/640eb352-c81c-4190-b10a-b18c1ff4495c-sys\") pod \"perf-node-gather-daemonset-74dxq\" (UID: \"640eb352-c81c-4190-b10a-b18c1ff4495c\") " pod="openshift-must-gather-48dpl/perf-node-gather-daemonset-74dxq" Oct 30 15:11:42.254857 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:42.254831 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phx5v\" (UniqueName: \"kubernetes.io/projected/640eb352-c81c-4190-b10a-b18c1ff4495c-kube-api-access-phx5v\") pod \"perf-node-gather-daemonset-74dxq\" (UID: \"640eb352-c81c-4190-b10a-b18c1ff4495c\") " pod="openshift-must-gather-48dpl/perf-node-gather-daemonset-74dxq" Oct 30 15:11:42.255041 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:42.255025 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/640eb352-c81c-4190-b10a-b18c1ff4495c-podres\") pod \"perf-node-gather-daemonset-74dxq\" (UID: \"640eb352-c81c-4190-b10a-b18c1ff4495c\") " pod="openshift-must-gather-48dpl/perf-node-gather-daemonset-74dxq" Oct 30 15:11:42.255204 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:42.255190 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/640eb352-c81c-4190-b10a-b18c1ff4495c-proc\") pod \"perf-node-gather-daemonset-74dxq\" (UID: \"640eb352-c81c-4190-b10a-b18c1ff4495c\") " pod="openshift-must-gather-48dpl/perf-node-gather-daemonset-74dxq" Oct 30 15:11:42.255322 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:42.255309 4982 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/640eb352-c81c-4190-b10a-b18c1ff4495c-lib-modules\") pod \"perf-node-gather-daemonset-74dxq\" (UID: \"640eb352-c81c-4190-b10a-b18c1ff4495c\") " pod="openshift-must-gather-48dpl/perf-node-gather-daemonset-74dxq" Oct 30 15:11:42.356667 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:42.356613 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/640eb352-c81c-4190-b10a-b18c1ff4495c-podres\") pod \"perf-node-gather-daemonset-74dxq\" (UID: \"640eb352-c81c-4190-b10a-b18c1ff4495c\") " pod="openshift-must-gather-48dpl/perf-node-gather-daemonset-74dxq" Oct 30 15:11:42.356871 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:42.356725 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/640eb352-c81c-4190-b10a-b18c1ff4495c-proc\") pod \"perf-node-gather-daemonset-74dxq\" (UID: \"640eb352-c81c-4190-b10a-b18c1ff4495c\") " pod="openshift-must-gather-48dpl/perf-node-gather-daemonset-74dxq" Oct 30 15:11:42.356871 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:42.356741 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/640eb352-c81c-4190-b10a-b18c1ff4495c-podres\") pod \"perf-node-gather-daemonset-74dxq\" (UID: \"640eb352-c81c-4190-b10a-b18c1ff4495c\") " pod="openshift-must-gather-48dpl/perf-node-gather-daemonset-74dxq" Oct 30 15:11:42.356871 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:42.356757 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/640eb352-c81c-4190-b10a-b18c1ff4495c-lib-modules\") pod \"perf-node-gather-daemonset-74dxq\" (UID: \"640eb352-c81c-4190-b10a-b18c1ff4495c\") " pod="openshift-must-gather-48dpl/perf-node-gather-daemonset-74dxq" Oct 30 15:11:42.356871 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:42.356830 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/640eb352-c81c-4190-b10a-b18c1ff4495c-sys\") pod \"perf-node-gather-daemonset-74dxq\" (UID: \"640eb352-c81c-4190-b10a-b18c1ff4495c\") " pod="openshift-must-gather-48dpl/perf-node-gather-daemonset-74dxq" Oct 30 15:11:42.356871 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:42.356853 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/640eb352-c81c-4190-b10a-b18c1ff4495c-proc\") pod \"perf-node-gather-daemonset-74dxq\" (UID: \"640eb352-c81c-4190-b10a-b18c1ff4495c\") " pod="openshift-must-gather-48dpl/perf-node-gather-daemonset-74dxq" Oct 30 15:11:42.356871 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:42.356866 4982 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phx5v\" (UniqueName: \"kubernetes.io/projected/640eb352-c81c-4190-b10a-b18c1ff4495c-kube-api-access-phx5v\") pod \"perf-node-gather-daemonset-74dxq\" (UID: \"640eb352-c81c-4190-b10a-b18c1ff4495c\") " pod="openshift-must-gather-48dpl/perf-node-gather-daemonset-74dxq" Oct 30 15:11:42.357142 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:42.356830 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/640eb352-c81c-4190-b10a-b18c1ff4495c-lib-modules\") pod \"perf-node-gather-daemonset-74dxq\" (UID: \"640eb352-c81c-4190-b10a-b18c1ff4495c\") " pod="openshift-must-gather-48dpl/perf-node-gather-daemonset-74dxq" Oct 30 15:11:42.357142 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:42.356867 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/640eb352-c81c-4190-b10a-b18c1ff4495c-sys\") pod \"perf-node-gather-daemonset-74dxq\" (UID: \"640eb352-c81c-4190-b10a-b18c1ff4495c\") " pod="openshift-must-gather-48dpl/perf-node-gather-daemonset-74dxq" Oct 30 15:11:42.384353 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:42.383554 4982 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phx5v\" (UniqueName: \"kubernetes.io/projected/640eb352-c81c-4190-b10a-b18c1ff4495c-kube-api-access-phx5v\") pod \"perf-node-gather-daemonset-74dxq\" (UID: \"640eb352-c81c-4190-b10a-b18c1ff4495c\") " pod="openshift-must-gather-48dpl/perf-node-gather-daemonset-74dxq" Oct 30 15:11:42.526034 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:42.525949 4982 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-48dpl/perf-node-gather-daemonset-74dxq" Oct 30 15:11:42.857222 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:42.856674 4982 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-48dpl/perf-node-gather-daemonset-74dxq"] Oct 30 15:11:42.865148 ip-10-0-140-105 kubenswrapper[4982]: W1030 15:11:42.863704 4982 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod640eb352_c81c_4190_b10a_b18c1ff4495c.slice/crio-4b30dfc527728983d861405ca530e61cebc85712e733f175eb37b620ab7628b1 WatchSource:0}: Error finding container 4b30dfc527728983d861405ca530e61cebc85712e733f175eb37b620ab7628b1: Status 404 returned error can't find the container with id 4b30dfc527728983d861405ca530e61cebc85712e733f175eb37b620ab7628b1 Oct 30 15:11:42.930219 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:42.930181 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-48dpl/perf-node-gather-daemonset-74dxq" event={"ID":"640eb352-c81c-4190-b10a-b18c1ff4495c","Type":"ContainerStarted","Data":"4b30dfc527728983d861405ca530e61cebc85712e733f175eb37b620ab7628b1"} Oct 30 15:11:43.902827 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:43.902795 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-9r9zr_b0b2d906-7995-4769-8222-fa8ba2a30f3d/cluster-monitoring-operator/0.log" Oct 30 15:11:43.936216 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:43.936184 4982 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-48dpl/perf-node-gather-daemonset-74dxq" event={"ID":"640eb352-c81c-4190-b10a-b18c1ff4495c","Type":"ContainerStarted","Data":"5bf344efb576a5f7245f4f6e16bc66cdc6fde0b51c21e5536e879a719c12f03a"} Oct 30 15:11:43.936400 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:43.936264 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-48dpl/perf-node-gather-daemonset-74dxq" Oct 30 15:11:43.981269 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:43.981215 4982 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-48dpl/perf-node-gather-daemonset-74dxq" podStartSLOduration=1.981203423 podStartE2EDuration="1.981203423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 15:11:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:11:43.979666872 +0000 UTC m=+4973.723800907" watchObservedRunningTime="2026-10-30 15:11:43.981203423 +0000 UTC m=+4973.725337464" Oct 30 15:11:45.020749 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:45.020715 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8qgb5_518f84a1-ae8d-4fcd-8c5b-1eeb820fac19/node-exporter/0.log" Oct 30 15:11:45.138444 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:45.138415 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8qgb5_518f84a1-ae8d-4fcd-8c5b-1eeb820fac19/kube-rbac-proxy/0.log" Oct 30 15:11:45.268441 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:45.268411 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8qgb5_518f84a1-ae8d-4fcd-8c5b-1eeb820fac19/init-textfile/0.log" Oct 30 15:11:49.949314 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:49.949284 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-2bnbl_8309f073-dcb0-4971-9eb1-8a9ee0eb68c6/networking-console-plugin/0.log" Oct 30 15:11:49.952847 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:49.952823 4982 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-48dpl/perf-node-gather-daemonset-74dxq" Oct 30 15:11:50.578887 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:50.578843 4982 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b622r_ad85cf2a-6c44-4e85-8677-8b07cb9f78a4/konnectivity-agent/0.log" limit=1024 Oct 30 15:11:51.049771 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:51.049697 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ntlqq_dc967875-1546-40bb-8376-fde09416c204/console-operator/2.log" Oct 30 15:11:51.054060 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:51.054033 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ntlqq_dc967875-1546-40bb-8376-fde09416c204/console-operator/3.log" Oct 30 15:11:53.455786 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:53.455753 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-brd96_8d9908f6-aefa-4bde-9182-2c199f08a24a/volume-data-source-validator/0.log" Oct 30 15:11:55.064623 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:55.064592 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-98f7774f-vb9c4_84f3cb27-ca2f-4a9c-8f30-53fe968ec15d/istio-proxy/0.log" Oct 30 15:11:55.684158 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:55.684125 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7lx4t_083c70c2-0d5e-4876-9e88-10c6efbb681c/dns/0.log" Oct 30 15:11:55.810078 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:55.810051 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7lx4t_083c70c2-0d5e-4876-9e88-10c6efbb681c/kube-rbac-proxy/0.log" Oct 30 15:11:56.459200 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:56.459168 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bjswt_e9a035cc-e78b-4bdb-b0c6-ef13bc07249a/dns-node-resolver/0.log" Oct 30 15:11:58.012693 ip-10-0-140-105 kubenswrapper[4982]: E1030 15:11:58.012632 4982 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-87kd9" containerName="extractor" Oct 30 15:11:58.024435 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:58.024408 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-llx79_c06b4b47-6141-4263-94b3-803d9f28bb94/node-ca/0.log" Oct 30 15:11:58.365060 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:58.365017 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-f2kp6_f4dd3cf1-c4f4-45ee-8ef7-294af175cbdc/istio-proxy/0.log" Oct 30 15:11:58.566683 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:58.566653 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-kxfp8_556451f5-3d37-40a5-bf8f-f470d05ef183/istio-proxy/0.log" Oct 30 15:11:58.633361 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:58.633270 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-mqc7s_f784c488-9f50-4924-a54b-811e5be8661f/istio-proxy/0.log" Oct 30 15:11:58.910697 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:58.910595 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-659879fb99-pjvcq_6d73c6c3-044d-4b88-a105-72bd5e4491cd/router/0.log" Oct 30 15:11:59.930033 ip-10-0-140-105 kubenswrapper[4982]: I1030 15:11:59.930000 4982 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-98f7774f-vb9c4_84f3cb27-ca2f-4a9c-8f30-53fe968ec15d/istio-proxy/0.log"